JP2022161527A - Image processing method, image processing apparatus, imaging device, and program - Google Patents

Abstract

To satisfactorily reduce a deterioration component in an input image obtained by pixel addition while reducing an amount of data to be retained with regard to image recovery processing.SOLUTION: An image processing method includes: a step S11 for acquiring an input image obtained by performing weighted addition for each of a plurality of pixels to an image generated by imaging through an imaging optical system; a step S12 for acquiring information regarding imaging conditions in the imaging; and steps S13 to S14 for acquiring an image recovery filter according to the imaging conditions. The input image is an image obtained by performing the weighted addition using asymmetric weight for the plurality of pixels. The image recovery filter acquisition steps acquire the image recovery filter on the basis of deterioration characteristics of the imaging optical system under the imaging conditions and an aperture size equivalent to one pixel according to the weight of the weighted addition.SELECTED DRAWING: Figure 5

Description

The present invention relates to a technique for performing image restoration processing on an input image.

An image captured by an imaging device such as a digital camera is dependent on the aperture value (F number) even when various aberrations such as spherical aberration, coma aberration, curvature of field, and astigmatism of the imaging optical system are corrected. It deteriorates due to diffraction phenomenon.

FIG. 11 shows the diffraction limit for each F-number, with the horizontal axis representing the spatial frequency and the vertical axis representing the MTF (Modulation Transfer Function). As shown in this figure, the cutoff frequency shifts to the low frequency side as the F number increases (darkens). For example, the Nyquist frequency of an imaging device with a pixel size of 4 μm is 125 lines/mm. For this reason, the effect is small for bright F-numbers such as F2.8, but the effect is large for dark F-numbers such as F16 and F32. Diffraction, like aberration, can be described by an optical transfer function (OTF) or a point spread function (PSF), so blurring due to diffraction can be corrected by image restoration processing. As image restoration processing, a method of convoluting an input image with an image restoration filter having the inverse characteristics of OTF is known.

Japanese Patent Application Laid-Open No. 2002-200003 discloses a method of performing image restoration processing in which the amount of data and the amount of calculation of an image restoration filter are reduced by making only diffraction the object of correction. Moreover, Patent Document 1 describes that the shape of the pixel aperture also affects the optical transfer function, and therefore it is preferable to take this effect into consideration. In this case, the image restoration filter is further generated based on the deterioration characteristic (aperture deterioration) due to the pixel aperture.

Japanese Patent No. 5653464

In the method disclosed in Patent Document 1, the amount of data and the amount of calculation are reduced by limiting the correction targets to diffraction and aperture deterioration. I need to keep the filter. In addition, still images are often acquired with high-pixel imaging because high definition is required, and moving images are often acquired with low-pixel imaging in consideration of the frame rate. pixel addition may be performed in the method. Since aperture deterioration changes according to the pixel addition method, it is necessary to hold an image restoration filter for each imaging mode and addition method. Furthermore, when image restoration filters having different restoration gains (restore strengths) are held, the amount of data of the image restoration filters to be held becomes enormous.

SUMMARY OF THE INVENTION The present invention provides an image processing method and the like capable of satisfactorily reducing deterioration components in an input image obtained by performing pixel addition while reducing the amount of data to be held for image restoration processing.

An image processing method as one aspect of the present invention includes a step of acquiring an input image obtained by performing weighted addition for each of a plurality of pixels on an image generated by imaging through an imaging optical system; It has a step of acquiring information about imaging conditions and a step of acquiring an image restoration filter according to the imaging conditions. The input image is an image in which weighted addition is performed using asymmetric weights for a plurality of pixels. In the step of obtaining the image restoration filter, the image restoration filter is obtained based on deterioration characteristics of the imaging optical system under imaging conditions and an aperture size corresponding to one pixel according to the weight of the weighted addition. A program that causes a computer to execute processing according to the above image processing method also constitutes another aspect of the present invention.

Further, an image processing apparatus as another aspect of the present invention provides an image that acquires an input image obtained by performing weighted addition for each of a plurality of pixels on an image generated by imaging through an imaging optical system. It has acquisition means, condition acquisition means for acquiring information about imaging conditions in imaging, and filter acquisition means for acquiring an image restoration filter corresponding to the imaging conditions. The input image is an image in which weighted addition is performed using asymmetric weights for a plurality of pixels. The filter obtaining means is characterized in that it generates an image restoration filter based on the deterioration characteristic of the imaging optical system under the imaging condition and the aperture size corresponding to one pixel according to the weight of the weighted addition. Note that an imaging device having the above image processing device also constitutes another aspect of the present invention.

According to the present invention, it is possible to satisfactorily reduce deterioration components in an input image obtained by performing pixel addition while reducing the amount of data to be held regarding image restoration processing.

4A and 4B are diagrams for explaining an image restoration filter in the embodiment; FIG. FIG. 2 is a cross-sectional view of an image restoration filter in an example; FIG. 4 is a diagram for explaining a point spread function PSF in an example; FIG. 4 is a diagram for explaining an amplitude component MTF and a phase component PTF of an optical transfer function OTF in the example; 4 is a flowchart showing an image processing method according to the first embodiment; 4A and 4B are diagrams showing an example of pixel addition according to the first embodiment; FIG. 4A and 4B are diagrams for explaining weighted addition using asymmetric weights in the first embodiment; FIG. 4A and 4B are diagrams showing an example of pixel addition in a Bayer array according to the first embodiment; FIG. 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment; FIG. FIG. 11 is a block diagram showing the configuration of an image processing system according to a second embodiment; FIG. The figure explaining a diffraction limit.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, definitions of terms and an outline of an image processing method including image restoration processing in the embodiment will be described.
[Captured image]
A captured image is a digital image obtained by receiving light with an imaging device via an imaging optical system, and is degraded by an optical transfer function OTF due to aberration of the imaging optical system including lenses and various optical filters. The imaging optical system may be configured using not only a lens but also a mirror (reflecting surface) having a curvature.

The color components of the captured image have, for example, RGB color component information. As the color components, it is also possible to select and use a generally used color space such as lightness, hue, and saturation represented by LCH, and luminance and color difference signals represented by YCbCr. XYZ, Lab, Yuv, and JCh can be used as other color spaces. Additionally, color temperature may be used.

A captured image may be a still image or a moving image, or may be a frame image forming one frame of a moving image. In addition, the captured image may be obtained by performing pixel addition. In moving images, pixel addition may be performed by a method according to the imaging mode at the time of imaging.

Captured images and output images after image restoration processing contain information on imaging conditions such as the focal length of the imaging optical system, aperture value, and imaging distance, information on the addition method for pixel addition, and various corrections for correcting each image. Correction information can be attached. When an image to be corrected is passed from an imaging device to another image processing device and subjected to correction processing, it is preferable to attach information about imaging conditions, information about an addition method, and correction information to the captured image. An image and each information may be transferred directly from the imaging device to the image processing device, or may be transferred indirectly via a memory.
[Image restoration processing]
The image deterioration component (bokeh) due to aberration and diffraction means that the luminous flux emitted from a single point on the subject, if there were no aberration and no effect of diffraction, would have spread out as it should have gathered again at a single point on the image sensor. , a point spread function PSF.

The optical transfer function OTF obtained by Fourier transforming the point spread function PSF is information on the frequency component of the aberration and is represented by a complex number. The absolute value of the optical transfer function OTF, that is, the amplitude component is called MTF, and the phase component is called PTF (Phase Transfer Function). The amplitude component MTF and phase component PTF are the frequency characteristics of the amplitude component and phase component of image degradation due to aberration, respectively, and are expressed by the following equations using the phase component as a phase angle.

PTF = tan-1 (Im(OTF)/Re(OTF))
where Re(OTF) and Im(OTF) represent the real and imaginary parts of the optical transfer function OTF, respectively. The above optical transfer function OTF and point spread function PSF are expressed as deterioration characteristics of the imaging optical system. As a method of correcting deterioration of the amplitude component MTF and phase component PTF, there is known a method of correcting using information of the optical transfer function OTF of the imaging optical system. This method is called image restoration or image restoration, and in the following description, processing for correcting deterioration of a captured image using information on the optical transfer function (OTF) of the imaging optical system is referred to as image restoration processing. . In this embodiment, as the image restoration processing, a method of convolving an input image with an image restoration filter having the inverse characteristic of the optical transfer function OTF is used.

In order to perform image restoration processing effectively, it is necessary to obtain a more accurate OTF of the imaging optical system. As for the method of obtaining the OTF, for example, if there is design value information of the imaging optical system, it is possible to obtain the OTF by calculation from that information. It can also be obtained by capturing an image of a point light source and performing a Fourier transform on its intensity distribution. Furthermore, in diffraction, it can also be obtained from a theoretically derived calculation formula.

When the captured image (degraded image) is g (x, y), the original image is f (x, y), and the point spread function PSF, which is the Fourier pair of the optical transfer function OTF, is h (x, y), The following formula (1) holds.

g(x,y)=h(x,y)*f(x,y) (1)
Here, * is convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image.

Further, when Equation (1) is Fourier-transformed into a display form in terms of frequency, Equation (2) is obtained which is represented by the product of each frequency.

G(u,v)=H(u,v)·F(u,v) (2)
Here, H is an optical transfer function OTF obtained by Fourier transforming the point spread function PSF(h), and G and F are obtained by Fourier transforming the degraded image g and the original image f, respectively. is a defined function. (u, v) are the coordinates on the two-dimensional frequency plane, that is, the frequency.

In order to obtain the original image f from the degraded image g obtained by imaging, both sides of the equation (2) should be divided by the optical transfer function H as in the following equation (3).

G(u,v)/H(u,v)=F(u,v) (3)
Then, F(u, v), that is, G(u, v)/H(u, v), is inverse Fourier transformed and restored to the real plane to obtain a restored image corresponding to the original image f(x, y). (output image) is obtained.

Assuming that the inverse Fourier transform of H ⁻¹ is R, the original image f (x, y) is similarly obtained by performing convolution processing on the image on the real plane as in the following equation (4) be able to.

g(x,y)*R(x,y)=f(x,y) (4)
Here, R(x,y) is called an image restoration filter. When the image is a two-dimensional image, the image restoration filter R is also generally a two-dimensional filter having taps (cells) corresponding to each pixel of the image. Also, in general, the greater the number of taps (the number of cells) of the image restoration filter R, the higher the restoration accuracy. Therefore, the number of taps that can be realized is set according to the required image quality, image processing capability, aberration characteristics, and the like.

Since the image restoration filter R must reflect at least the characteristics of aberration, it is different from a conventional edge enhancement filter (high-pass filter) or the like having about three horizontal and vertical taps. Since the image restoration filter R is set based on the optical transfer function OTF, it is possible to correct both the deterioration of the amplitude component and the phase component with high accuracy.

Since the actual image contains noise components, using the image restoration filter R created by taking the complete reciprocal of the optical transfer function OTF as described above greatly amplifies the noise components as the degraded image is restored. be done. This is to raise the MTF (amplitude component) of the imaging optical system so that it returns to 1 over all frequencies in a state where the amplitude of the noise component is added to the amplitude component of the image. The MTF, which is amplitude deterioration due to the imaging optical system, returns to 1, but at the same time, the power spectrum of the noise component is raised, and as a result, the noise component is amplified according to the degree of raising the MTF (recovery gain).

Therefore, if the degraded image contains noise components, the restored image will not be a good viewing image. This is represented by the following formulas (5-1) and (5-2).

G (u, v) = H (u, v) F (u, v) + N (u, v) (5-1)
G(u,v)/H(u,v)=F(u,v)+N(u,v)/H(u,v) (5-2)
where N is the noise component.

As for an image containing a noise component, there is a method of controlling the degree of restoration according to the intensity ratio SNR between the image signal and the noise signal, such as the Wiener filter represented by Equation (6) below.

Here, M(u, v) is the frequency characteristic of the Wiener filter, and |H(u, v)| is the absolute value (MTF) of the optical transfer function OTF. In this method, for each frequency, the smaller the MTF, the smaller the recovery gain (recovery degree), and the larger the MTF, the larger the recovery gain. In general, the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, so this method substantially reduces the recovery gain on the high frequency side of the image.

Next, the image restoration filter will be described in more detail with reference to FIGS. 1(a) and 2(a) and (b). The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter illustrated in FIG. 1(a) is a two-dimensional filter with 11×11 taps. Although the values in each tap are omitted in FIG. 1(a), the distribution of the values in the taps (coefficients: hereinafter referred to as tap values) in one section of this image restoration filter is shown in FIG. 2(a). show. The distribution of the tap values of the image restoration filter ideally has the function of returning the signal value (PSF) spatially spread due to aberration to the original point. Here, when the target is diffraction by an aperture that can be approximated to be rotationally symmetrical, the PSF due to diffraction is rotationally symmetrical. symmetrical distribution.

Each tap value of the image restoration filter is convolved with each corresponding pixel of the degraded image. In the convolution process, a predetermined pixel in the degraded image is aligned with the center of the image restoration filter in order to improve the signal value of the pixel. Then, the signal value of the degraded image and the tap value of the filter are multiplied for each pixel corresponding to the degraded image and the image restoration filter, and the sum total is replaced as the signal value of the central pixel.

Next, characteristics of image restoration processing in real space and frequency space will be described with reference to FIGS. FIG. 3(a) shows the PSF before image restoration processing, and FIG. 3(b) shows the PSF after image restoration processing. Broken line (a) in FIG. 4(M) indicates the amplitude component MTF of the OTF before the image restoration processing, and dashed line (b) indicates the MTF after the image restoration processing. In FIG. 4P, the broken line (a) indicates the phase component PTF of the OTF before the image restoration processing, and the dashed line (b) indicates the PTF after the image restoration processing.

As shown in FIG. 3(a), the PSF before image restoration processing has an asymmetric spread, and due to this asymmetry, the PTF has a non-linear value with respect to frequency. Since the image restoration process is a process of amplifying the MTF and correcting the PTF so that it becomes zero, the PSF after the image restoration process has a symmetrical and sharp shape.

Thus, the image restoration filter can be obtained by inverse Fourier transforming a function designed based on the inverse function of the OTF of the imaging optical system. The image restoration filter used in this embodiment can be changed as appropriate, and for example, the Wiener filter described above can also be used. When using the Wiener filter, it is possible to create a real-space image restoration filter that is actually convoluted with the image by inverse Fourier transforming Equation (6).

Moreover, since the OTF due to aberration changes according to the image height (position on the image) even under the same imaging conditions, it is necessary to change the image restoration filter to be used according to the image height. On the other hand, the OTF due to diffraction, whose influence becomes dominant as the F-number increases, can be treated as a uniform OTF with respect to image height when the influence of vignetting in the imaging optical system is small.

In this embodiment, the degradation component (diffraction blur) due to diffraction is to be corrected. Therefore, the image restoration filter depends only on the aperture value and the wavelength of light, and does not depend on the image height. Therefore, a uniform (identical) image restoration filter can be used for one degraded image. That is, the image restoration filter of this embodiment is generated based on the OTF due to the diffraction blur that occurs according to the aperture value. As for wavelengths, OTFs at a plurality of wavelengths can be calculated, and an OTF for each color component can be generated by weighting each wavelength based on the spectrum of an assumed light source and the light receiving sensitivity of an imaging device. Note that an OTF may be generated at a representative wavelength for each predetermined color component. Then, an image restoration filter can be generated based on the OTF for each color component.

Therefore, in this embodiment, in which only the diffraction blur is to be corrected, a plurality of image restoration filters depending on the aperture value are stored in advance, and a uniform image restoration filter is used according to the aperture value (imaging condition) at the time of imaging. image recovery processing can be performed on the degraded image.

Further, aperture deterioration due to the shape of the pixel aperture and deterioration due to an optical low-pass filter may be taken into consideration. The aperture deterioration PIX can be expressed as a deterioration function given by Equation (7) below.

PIX(u)=sin(πau/2)/(πau/2) (7)
Aperture degradation in Equation (7) changes based on the aperture ratio a (ratio of apertures taken into account when the size of one pixel is 1).

By multiplying the optical transfer function due to diffraction blur by aperture deterioration and deterioration due to the optical low-pass filter, it can be treated as optical transfer of the entire imaging system including the imaging optical system, the optical low-pass filter, and the imaging device. That is, the optical transfer function H(u, v) of the entire imaging system is expressed by the following equation (8): OTF(u, v) of the imaging optical system, aperture deterioration PIX(u, v), optical low-pass It can be expressed using the deterioration characteristic OLPF(u, v) of the filter.
H(u,v)=OTF(u,v)×PIX(u,v)×OLPF(u,v) (8)
By using an image restoration filter based on the optical transfer function of the entire imaging system, it is possible to reduce not only diffraction blur but also aperture deterioration and deterioration due to an optical low-pass filter.

The flowchart of FIG. 5 shows the image recovery processing (image processing method) in this embodiment. The image restoration processing of this embodiment is executed according to a computer program by an image processing unit (CPU, GPU, etc.) as an image processing device built in the imaging device. The image processing unit functions as image acquisition means, condition acquisition means, filter acquisition means, and processing means. In the following description, the subject of processing is assumed to be an image processing unit in the image capturing apparatus, but an image processing apparatus (computer such as a PC) different from the image capturing apparatus may be the subject.

In step S11, the image processing unit acquires, as an input image, a captured image generated by imaging by the imaging device. The captured image here is a RAW image before development processing. Further, the captured image here is an image generated by performing pixel addition when reading image data from an imaging device of an imaging device. Particularly in moving image capturing, the amount of data read out by pixel addition can be reduced, so the frame rate can be made higher than in all-pixel readout without pixel addition. In still image pickup and moving image pickup, pixel addition is performed according to the imaging mode at the time of imaging.

Next, in step S12, the image processing unit acquires information about imaging conditions. The information about the imaging conditions includes aperture value (F number), focal length, imaging distance, and the like. Further, the information regarding the imaging conditions when using a lens device (interchangeable lens) that is detachable from the imaging device includes an ID for identifying them. Information about the imaging conditions may be obtained directly from the imaging device, or may be obtained from information attached to the captured image. The information about the imaging conditions does not have to be information that directly indicates the aperture value, focal length, imaging distance, and the like, and may be information that can be converted into them.

Next, in step S13, the image processing unit acquires information about the addition method in pixel addition when the captured image is generated. The information about the addition method includes the pixel range in which pixel addition is performed (addition range), the number of pixels added to form one pixel (number of added pixels), and the weight for each added pixel in weighted addition. . The information about the addition method may not be information directly indicating the addition range, the number of pixels to be added, the weight, etc., but may be information that can be converted to them. Furthermore, the information about the addition method may be information indicating the imaging mode at the time of imaging when the addition method is determined for each imaging mode.

The image restoration filters are stored in association with the addition method (addition range, number of added pixels, weight, imaging mode, etc.), and the image restoration filter corresponding to the addition method obtained in this step is used.

If there is only one addition method for the imaging device, the information on the addition method may not be obtained in step S13, and a specific image restoration filter may be obtained in the next step S14.

In step S14, the image processing unit acquires an image restoration filter suitable for the addition method and imaging conditions. When correcting diffraction blur as in the present embodiment, the image restoration filter does not depend on the focal length and imaging distance among the imaging conditions, but depends on the aperture value and addition method among the imaging conditions. Therefore, the image processing unit acquires an image restoration filter corresponding to the aperture value and addition method from the image restoration filters held.

However, since the effective F-number changes depending on the imaging distance, the image restoration filter may be selected in consideration of the imaging distance.

Selectable image restoration filters are generated in advance according to the aperture value and addition method and stored in a memory (not shown). The image processing unit selects and acquires an image restoration filter corresponding to the aperture value and addition method acquired in steps S12 and S13 from the stored image restoration filters.

A method of generating a captured image by pixel addition and a method of generating an image restoration filter according to the aperture value and addition method will be described below with reference to FIGS. 6, 7 and 8. FIG.

FIG. 6 shows generation of a captured image 12 after pixel addition by performing pixel addition on the original image 11 before pixel addition. By adding two hatched pixels in the horizontal direction and two pixels in the vertical direction in the original image 11 , one hatched pixel in the captured image 12 is generated.

In the upper diagram of FIG. 7A, two pixels within the addition range in the horizontal direction in the captured image are weighted and added with a weight of 2:1 (weighted pixel addition), and two pixels within the addition range in the vertical direction are obtained. is added with weighting of 1:1. Furthermore, in the upper diagram of FIG. 7B, the three pixels within the addition range in the horizontal direction in the captured image are weighted and added with a weighting of 3:1:1, and the three pixels within the addition range in the vertical direction are weighted and added to 1:1. It shows an addition method in which pixels are added with a weight of 1:1. The lower diagrams of FIGS. 7A and 7B respectively show the cross sections of the upper diagrams, and the magnitude of the weight is shown as the height. As shown in these figures, the weights for a plurality of added pixels for which weighted addition is performed in the pixel addition direction (here, horizontal direction) are not symmetrical with respect to the center (weight center) of these weighted pixels. In this embodiment, the weights in weighted addition are said to be asymmetric. If the weights of the weighted sum are asymmetric, then of course the weights for all of the summed pixels will not be the same. On the other hand, in the other direction of pixel addition, the vertical direction, the weights for the pixels to be added are symmetrical.

When image recovery processing is performed on a captured image that has undergone pixel addition, the sampling interval is widened by the addition range with respect to the sampling interval of the image sensor. That is, the pixel addition changes the frequency range of the optical transfer function to be considered. If two pixels within the addition range in the horizontal direction are added at a weighted ratio of 2:1 and two pixels within the addition range in the vertical direction are added at a ratio of 1:1, the sampling interval in the horizontal and vertical directions is is doubled before pixel addition, and the frequency range to consider is halved before pixel addition. In the image restoration process, an optical transfer function in the frequency range up to the Nyquist frequency in the addition range is acquired to generate an image restoration filter.

Considering the pixel aperture, when two horizontal pixels are weighted and added at 2:1, the pixel aperture becomes asymmetric in the horizontal direction. An image restoration filter for correcting aperture deterioration caused by such an asymmetric pixel aperture is a filter with asymmetric tap values in the horizontal direction as shown in FIG. 2(a).

Here, the two-dimensional image restoration filter includes tap value data for the number of taps. Since the diffraction blur changes according to the aperture value, it is necessary to hold an image restoration filter for each aperture value. Furthermore, especially in the case of moving images, pixel addition of various addition methods is performed depending on the imaging mode at the time of imaging. Since the effect of the pixel aperture changes depending on the addition method and imaging mode, it is necessary to hold an image restoration filter for each addition method and imaging mode. Furthermore, it may be necessary to hold image restoration filters with different restoration gains (restore strengths). For these reasons, the data amount of the image restoration filter to be held becomes enormous.

As a method of reducing the data amount of the image restoration filter to be held, there is a method of holding only part of the two-dimensional image restoration filter. For example, as shown in FIG. 1B, by holding the tap values of the hatched 1/4 (one quadrant) portion of the two-dimensional image restoration filter, all the tap values of the image restoration filter are held. The amount of data can be reduced compared to the case. Further, the data amount may be reduced by holding data of a plurality of one-dimensional tap values that are components of a two-dimensional image restoration filter. For example, the amount of data can be reduced by keeping one-dimensional data that is half the horizontal and vertical data, including the center of the two-dimensional image restoration filter.

On the other hand, if the tap values of the image restoration filter are symmetrical, the amount of data can be reduced by holding the data obtained by decomposing the image restoration filter using fitting or the singular value decomposition theorem. In other words, by utilizing the symmetry of the two-dimensional image restoration filter, data amount can be reduced by holding a part of the data or the data of the constituent element whose data amount is smaller than that of the image restoration filter.

However, as described above, when performing weighted addition with an asymmetrical pixel aperture as pixel addition, the image restoration filter becomes asymmetrical, so the method for reducing the amount of data for a symmetrical image restoration filter cannot be used.
Therefore, in this embodiment, it is assumed that the size of the pixel aperture (aperture size) corresponding to one pixel after the weighted addition increases according to the weight of the weighted addition, and the pixel aperture is assumed to be symmetrical, thereby suppressing aperture deterioration. to generate an image restoration filter that can also correct This allows the use of data volume reduction methods for symmetrical image restoration filters.

That is, using the following formula (9), the aperture size ApertureSize when the pixel n (n=1, . The aperture deterioration PIX is determined by the above equation (7).

Expression (9) is an expression when the aperture size is 100% (aperture ratio=1) with respect to the pixel size of the image sensor. If the aperture ratio of the imaging device is not 1 or is significantly smaller than 1, the aperture ratio may be taken into consideration.

For example, when two horizontal pixels are weighted and added at 2:1, the aperture size can be calculated to be 1.5 times the pixel size of the image sensor when the aperture ratio of the image sensor is 1. Therefore, the aperture deterioration PIX can be calculated by setting the aperture ratio a=1.5 in Equation (7). Since two pixels are added in the vertical direction, the aperture size is twice the pixel size of the image sensor, and the aperture deterioration can be calculated with the aperture ratio a=2 in Equation (7).

By calculating the aperture deterioration at the time of weighted addition as described above and multiplying it by the optical transfer function of the diffraction blur, it can be handled as the optical transfer function of the entire imaging system. Furthermore, by generating and using an image restoration filter based on the optical transfer function, it is possible to satisfactorily correct diffraction blur and aperture deterioration.

In addition, the same can be applied to a case where an image is captured using a Bayer array image sensor as shown in FIG. In the Bayer array as well, two pixels within the addition range in the horizontal direction are weighted and added at 2:1 for pixels of each color, and two pixels within the addition range in the vertical direction are added. , the sampling interval is doubled before pixel addition, and the frequency range to be considered is halved. As for the aperture size, the aperture size at the time of weighted addition may be obtained using equation (9), and the aperture deterioration may be determined using equation (7). The pixel apertures here are not pixel apertures separated by one pixel due to the addition of pixels for each color of the Bayer array, but rather the pixel apertures are considered to be enlarged according to the weight of the weighted addition.

An image restoration filter is generated by the above method and stored in advance, and an image restoration filter suitable for the addition method and imaging conditions (aperture value) is obtained in step S14.

Subsequently, in step S15, the image processing unit performs image restoration processing by convolving the captured image (input image) after pixel addition with an image restoration filter to generate a restored image. At this time, the image processing section performs various processes related to development as well as the image recovery process. In convolution, the same image restoration filter is used for the entire captured image (of which image restoration processing is performed) without switching the image restoration filter according to the image height. Further, as the image restoration filter, a two-dimensional image restoration filter obtained by expanding or converting the image restoration filter with the reduced data amount acquired in step S14 may be used, or the image restoration filter with the reduced data amount may be used. The image restoration filter may be used as it is.

Then, in step S16, the image processing section outputs the restored image as an output image.

Thus, in this embodiment, the amount of data to be held in advance regarding the image restoration filter is smaller than the original amount of data of the two-dimensional image restoration filter. As a result, even when using an image restoration filter with a different restoration gain for each aperture value and each addition method, an increase in the amount of data to be held can be suppressed.

FIG. 9 shows the configuration of the imaging device 100 in this embodiment. Light from a subject (not shown) forms an image on an imaging device 102 by an imaging optical system 101 including an aperture 101a and a focus lens 101b. The aperture value is determined by the aperture diameter of the aperture 101a. The focus lens 101b is moved so as to bring the subject into focus according to autofocus (AF) control by the system controller 110 according to the subject distance or manual operation by the user.

The imaging device 102 photoelectrically converts the subject image formed by the imaging optical system 101 . An analog imaging signal output from the imaging element 102 is converted into a digital imaging signal by the A/D converter 103 , and an original image formed by the digital imaging signal is input to the image processing unit 104 .

The image processing unit 104 performs various types of image processing including pixel addition on the input original image to generate a captured image, and further performs the above-described image restoration processing on the captured image. Note that pixel addition may be performed on the analog image signal from the image sensor 102 .

In the image restoration process, the image processing unit 104 acquires information about the imaging conditions of the imaging apparatus 100 and information about the addition method in pixel addition from the state detection unit 107 . The information about the imaging conditions includes aperture value, shooting distance, focal length of the zoom lens, and the like. The state detection unit 107 may acquire information about the imaging conditions and information about the addition method directly from the system controller 110 that controls the entire imaging apparatus 100 , or may use the information about the imaging conditions to control driving of the imaging optical system 101 . It may be acquired from the imaging optical system control unit 106 that does.

The storage unit 108 stores and holds data regarding a plurality of image restoration filters that differ for each aperture value and addition method. The data relating to the image restoration filter held in the storage unit 108 is reduced in data amount with respect to the two-dimensional image restoration filter having symmetrical tap values as described above.

The image processing unit 104 acquires data related to the image restoration filters according to the aperture value and addition method from the data related to the plurality of image restoration filters held in the storage unit 108 . Then, an image restoration filter generated using the acquired data is convoluted over the entire captured image to generate a restored image. A restored image is recorded in a predetermined format on the image recording medium 109 . The display unit 105 displays a display image obtained by performing display processing on the restored image.

Note that the imaging optical system 101 may include optical elements such as an optical low-pass filter and an infrared cut filter. If the optical low-pass filter affects the optical transfer function of the imaging system, it is preferable to generate the image restoration filter based on the optical transfer function including the optical element. Since the infrared cut filter affects the PSF of the R channel, it is preferable to generate the image restoration filter in consideration of this influence.

Further, although the imaging apparatus 100 integrally including the imaging optical system 101 has been described in this embodiment, the imaging optical system 101 may be detachable from the imaging apparatus.

FIG. 9 shows the configuration of an image processing system 200 that is Embodiment 2 of the present invention. The imaging device 202 is a camera, a microscope, an endoscope, a scanner, or the like, and similarly to the imaging device 100 of the first embodiment, the imaging device 202 performs pixel addition on an original image obtained by imaging a subject to obtain a captured image. Generate. A storage medium 203 is a semiconductor memory, a hard disk, a server on a network, or the like, and can store captured images.

The image processing apparatus 201 is a computer such as a PC, and has image processing software 205 for executing the image restoration processing described in the first embodiment (FIG. 5). The image processing apparatus 201 acquires captured image data, imaging conditions, and addition method information from the imaging apparatus 202 or the storage medium 203, and generates a restored image by executing image restoration processing on the captured image. At this time, the image processing apparatus 201 may acquire the data of the captured image, the imaging conditions, and the addition method via wired or wireless communication with the imaging apparatus 202 or via a detachable storage medium (not shown). . Further, the information about the imaging conditions and the addition method may be attached to the captured image data.

The image processing device 201 outputs the restored image to the imaging device 202 and the storage medium 203 . A display device 204 that is a monitor is connected to the image processing apparatus 201 . Through the display device 204, the user can work on the image processing and evaluate the restored image.

According to the embodiments described above, it is possible to use a two-dimensional image restoration filter with a symmetrical distribution of tap values in image restoration processing. Specifically, it is assumed that the size of the pixel aperture (aperture size) corresponding to one pixel after weighted addition increases in accordance with the weight of the added pixels when generating the input image by pixel addition (weighted addition). We treat the pixel aperture as being symmetrical. This makes it possible to reduce the amount of data to be held since it is sufficient to hold data relating to the image restoration filter with a smaller amount of data than the two-dimensional image restoration filter. That is, according to each embodiment, while reducing the amount of data to be held regarding image restoration processing, image restoration processing can be performed on an input image subjected to pixel addition to satisfactorily reduce diffraction blur and aperture deterioration. can be done.
(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Each embodiment described above is merely a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

Reference Signs List 100, 200 imaging device 101 imaging optical system 102 imaging device 104 image processing unit 201 image processing device

Claims (15)

a step of acquiring an input image obtained by performing weighted addition for each of a plurality of pixels on an image generated by imaging through an imaging optical system;
a step of acquiring information about imaging conditions in the imaging;
obtaining an image restoration filter according to the imaging conditions;
the input image is an image in which the weighted addition is performed using asymmetric weights for the plurality of pixels;
In the step of obtaining the image restoration filter, the image restoration filter is obtained based on deterioration characteristics of the imaging optical system under the imaging conditions and an aperture size corresponding to one pixel according to the weight of the weighted addition. image processing method. 2. The method of claim 1, wherein said weights for all of said plurality of pixels are not the same. In the step of obtaining the image restoration filter,
Acquiring information about an addition method in the weighted addition;
3. The image processing method according to claim 1, wherein the image restoration filter is acquired based on the addition method and the imaging conditions. 4. The image processing method according to claim 3, wherein the information about the addition method includes at least one of an addition range, the number of pixels to be added, and a weight in the weighted addition, or information on an imaging mode at the time of the imaging. the image restoration filter is a two-dimensional filter having multiple taps,
5. The image processing method according to claim 1, wherein coefficients of said plurality of taps are symmetrical. 6. The image processing method according to claim 5, wherein data constituting part of said image restoration filter is held. Let PixelSize be the pixel size of the image sensor used for the imaging, and let wn be the weight for each of the plurality of pixels n for which the weighted addition is performed,
The aperture size ApertureSize,

7. The image processing method according to any one of claims 1 to 6, wherein the image is obtained by: 8. The image processing method according to any one of claims 1 to 7, wherein the image restoration filter is obtained using an optical transfer function in a frequency range up to the Nyquist frequency of the addition range in the weighted addition. 9. The image processing method according to any one of claims 1 to 8, wherein said deterioration characteristic includes a deterioration characteristic due to diffraction according to an aperture value of said imaging optical system. 10. The image processing method according to claim 1, further comprising the step of performing image restoration processing using said image restoration filter on said input image. 10. The image processing method according to claim 9, wherein in the step of performing the image restoration processing, the same image restoration filter is used for the entire area of the input image to be subjected to the image restoration processing. A program that causes a computer to execute processing according to the image processing method according to any one of claims 1 to 11. an image acquisition means for acquiring an input image obtained by performing weighted addition for each of a plurality of pixels on an image generated by imaging through an imaging optical system;
a condition acquiring means for acquiring information about imaging conditions in the imaging;
a filter obtaining means for obtaining an image restoration filter corresponding to the imaging condition;
the input image is an image in which the weighted addition is performed using asymmetric weights for the plurality of pixels;
The image processing, wherein the filter obtaining means generates the image restoration filter based on deterioration characteristics of the imaging optical system under the imaging conditions and an aperture size corresponding to one pixel according to the weight of the weighted addition. Device. 14. The image processing apparatus according to claim 13, further comprising processing means for performing image restoration processing using said image restoration filter on said input image. an imaging device that performs imaging through an imaging optical system;
An imaging apparatus comprising the image processing apparatus according to claim 13 or 14.

Images