JP2018107541A - Image processing apparatus, imaging device, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform filter processing to the vicinity of the ends of an image without adding false images to the ends of an input image.SOLUTION: An image processing apparatus 204 acquires an input image created through imaging using an optical system, and performs image processing on the input image. The image processing apparatus performs first sharpening processing on a first image area of the input image by using a first filter created on the basis of information on an optical transfer function or a point spread function of the optical system. The image processing apparatus performs second sharpening processing on a second image area closer to an end of the image than the first image area by using a second filter having a smaller number of taps than that of the first filter and created not on the basis of the information on the optical transfer function or the point spread function of the optical system.SELECTED DRAWING: Figure 17

Description

  The present invention relates to an image processing technique for performing a sharpening process on an input image generated by imaging.

  An image obtained by imaging a subject with an imaging device such as a digital camera has image degradation caused by spherical aberration, coma aberration, field curvature, astigmatism, etc. of an imaging optical system (hereinafter simply referred to as an optical system). A blur component is included as a component. Such a blur component is generated when a light beam emitted from one point of a subject to be collected again at one point on the imaging surface when an aberration is not caused and there is no influence of diffraction is formed by forming an image with a certain spread.

  The blur component here is optically represented by a point spread function (PSF), and is different from blur due to focus shift. In addition, color bleeding in a color image can also be said to be a difference in blurring for each wavelength of light with respect to axial chromatic aberration, chromatic spherical aberration, and chromatic coma aberration of the optical system. Further, the lateral color misregistration can be said to be a positional misalignment or a phase misalignment due to a difference in imaging magnification for each wavelength of light when the lateral chromatic aberration is caused by the optical system.

An optical transfer function (OTF) obtained by Fourier transform of a point spread function (PSF) is frequency component information of aberration and is represented by a complex number. The absolute value of the optical transfer function (OTF), that is, the amplitude component is called MTF (Modulation Transfer Function), and the phase component is called PTF (Phase Transfer Function). MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively. Here, the phase component is expressed as the phase angle by the following formula. Re (OTF) and Im (OTF) represent the real part and the imaginary part of the OTF, respectively.
PTF = tan ⁻¹ (Im (OTF) / Re (OTF))
As described above, since the optical transfer function (OTF) of the optical system deteriorates the amplitude component and the phase component of the image, each point of the subject becomes asymmetrically blurred like coma aberration in the deteriorated image. .

  As a method for correcting the deterioration of the amplitude component (MTF) and the deterioration of the phase component (PTF) in the deteriorated image (input image), a method using information on the optical transfer function of the optical system is known. This method is also called image restoration or image restoration. Hereinafter, processing for correcting (reducing) a deteriorated image using information on the optical transfer function of the optical system is called image restoration processing. As will be described in detail later, as one of image restoration processing methods, a convolution method is known in which a real space image restoration filter having an inverse characteristic of an optical transfer function is convoluted with an input image.

  When performing image restoration processing (filter processing) using an image restoration filter, in order to convolve an image restoration filter with a pixel to be processed (hereinafter referred to as a processing target pixel) in an input image, the processing target pixel The same number of pixels as the size of the image restoration filter (the number of taps) are required around. For this reason, the image restoration filter cannot be applied to the image area near the edge of the input image.

  A case of performing filter processing for convolving a filter on the input image shown in FIG. 2 will be described. The coordinates of each pixel are given as shown in the figure with the upper left coordinate of the input image as (0, 0). When the filter process using the 11 × 11 tap image restoration filter shown in FIG. 3A is performed on this input image, if there is ± 5 pixel data from the processing target pixel, the filter process can be performed. is there. However, as shown in FIG. 3B and FIG. 3C, it is not possible to perform the filtering process in the peripheral image area near the image edge where there is no pixel data of ± 5 pixels from the processing target pixel (non-filterable area). . For this reason, as shown in FIG. 3D, the image area in which the image restoration process can be performed (the area where the restoration process is possible) is excluded from the peripheral image area of 5 pixels (the area where the restoration process is impossible) from the edge of the image. It becomes an image area.

  In such a case, in order to perform the filtering process on the peripheral image region as well, as disclosed in Patent Document 1 and Patent Document 2, the pixels of the pixels necessary for the filtering process are provided outside the image edge of the input image. There is a method of adding data (that is, a pseudo image). Specifically, a single-color image having the same luminance is added, or a partial image near the edge of the input image is copied and added.

  However, in the method of performing filter processing by adding such a pseudo image, since the filter processing is performed using pixel data different from the pixel data indicating the actual subject image, correct image restoration processing cannot be performed. In Patent Document 1, in order to improve image restoration accuracy when performing a filtering process by adding a partial image near the edge of an image to the outside of the image edge of the input image, a filter is added according to the number of added partial pixels. The weight of is changed. Further, in Patent Document 2, when the number of pixels necessary for performing filter processing in the peripheral image region is insufficient, the size of the filter is reduced by the amount of the insufficient pixels.

JP 2008-129808 A JP 2001-0667468 A

  As a filter that performs image processing different from image restoration processing, there is an edge enhancement filter that enhances an edge portion of an image. Since the image restoration filter created based on the OTF is a filter corresponding to the optical characteristics of the imaging optical system with respect to the edge enhancement filter, more accurate image restoration processing is possible. As the edge emphasis filter, sharpening is performed using pixel data of four pixels on the top, bottom, left, and right of the central processing target pixel as shown in FIG. 4A, or the processing target as shown in FIG. 4B. Some perform sharpening using pixel data of eight pixels around the pixel.

  However, an image restoration filter created based on an OTF of a general imaging optical system having a certain degree of aberration is larger in size than these edge enhancement filters. For this reason, a peripheral image area without pixel data for performing filter processing in the input image becomes large.

  When a pseudo image is added to the outside of the image end in order to perform the filtering process as disclosed in Patent Document 1, it is necessary to secure a storage area for holding the pseudo image data. Further, if the size of the image restoration filter created based on the OTF is reduced in the peripheral image region as in the method disclosed in Patent Document 2, the image restoration filter that reflects the actual aberration of the imaging optical system cannot be obtained. The image restoration process cannot be performed sufficiently.

  In addition, there is a method in which filter processing is not performed in an area where filter processing cannot be performed, and only the area where filter processing is possible is cut out and filter processing is performed. The angle) decreases.

  The present invention provides an image processing apparatus capable of performing filter processing up to the vicinity of the image end without adding a pseudo image to the image end of the input image, and generating a filtered image having good image quality. I will provide a.

  An image processing apparatus according to one aspect of the present invention includes an image acquisition unit that acquires an input image generated by imaging using an optical system, and a processing unit that performs image processing on the input image. The processing means performs a first sharpening process on the first image region of the input image using a first filter generated based on information related to the optical transfer function or point spread function of the optical system. And the second image area closer to the image end than the first image area has a smaller number of taps than the first filter, and is based on information on the optical transfer function or point spread function of the optical system. The second sharpening process is performed using the second filter generated without any other feature.

  Note that an imaging apparatus having an imaging element that captures a subject image formed by an optical system and the image processing apparatus also constitutes another aspect of the present invention.

  An image processing method according to another aspect of the present invention includes a step of acquiring an input image generated by imaging using an optical system, and a processing step of performing image processing on the input image. In the processing step, a first sharpening process is performed on the first image region of the input image using a first filter generated based on information related to the optical transfer function or point spread function of the optical system. And the second image area closer to the image end than the first image area has a smaller number of taps than the first filter, and is based on information on the optical transfer function or point spread function of the optical system. The second sharpening process is performed using the second filter generated without any other feature.

  An image processing program that causes a computer to execute image processing according to the image processing method also constitutes another aspect of the present invention.

  In the present invention, in the first image region, a first sharpening process is performed using a first filter generated based on information related to the optical transfer function or point spread function of the optical system. In the second image region closer to the image end than the first image region, the number of taps is smaller than that of the first filter, and the second image region is generated without being based on information on the optical transfer function or point spread function of the optical system. A second sharpening process is performed using the second filter. Accordingly, it is possible to perform filter processing using the second filter up to the vicinity of the image end without adding a pseudo image to the image end of the input image, and it is possible to generate a filtered image having good image quality. .

The figure which shows the image restoration filter in the Example of this invention. The figure which shows an input image. The figure explaining the conventional image recovery filter, filter process impossible area, recoverable area, and recovery process impossible area. The figure which shows a 2nd filter. The flowchart which shows the process in Example 1 of this invention. FIG. 3 is a diagram illustrating an image restoration filter according to the first embodiment. FIG. 6 is a diagram illustrating a tap value of an image restoration filter in the first embodiment. FIG. 6 is a diagram illustrating correction of a deteriorated point image in the first embodiment. FIG. 3 is a diagram illustrating an amplitude and a phase in the first embodiment. FIG. 6 is a diagram illustrating another example of the image restoration filter according to the first embodiment. The figure which shows the image restoration process using the image restoration filter shown in FIG. FIG. 10 is a diagram illustrating another example of image restoration processing according to the first embodiment. The figure explaining the weighting of the image restoration filter used for the image restoration process of FIG. 12, and a 2nd filter. FIG. 3 is a diagram illustrating a configuration of an image processing system that is Embodiment 2. FIG. 10 is a diagram illustrating coefficient calculation processing according to the second embodiment. FIG. 6 is a diagram for explaining a reconfiguration OTF in Embodiment 2. FIG. 6 is a diagram illustrating a configuration of an imaging apparatus that is Embodiment 3. FIG. 10 is a diagram illustrating a configuration of an image processing system that is Embodiment 4. Illustration of unsharp mask processing

Embodiments of the present invention will be described below with reference to the drawings. First, definitions of terms and image restoration processing used in embodiments of the present invention described later will be described.
"Input image"
The input image is a digital image generated by using an output from an image sensor obtained by photoelectrically converting a subject image formed by an imaging optical system in the imaging apparatus. For example, a RAW image (RAW data) having RGB color component information is used. It is. The input image is an image degraded by an optical transfer function (hereinafter referred to as OTF) including aberration of an imaging optical system including an optical element such as a lens or an optical filter.

  The imaging optical system may include a mirror (reflection surface) having a curvature. Further, the imaging optical system may be detachable (exchangeable) with respect to the imaging device. In the imaging apparatus, an imaging system is configured by an imaging device and a signal processing unit that generates an input image using an output of the imaging device. The imaging element is configured by a photoelectric conversion element such as a CMOS sensor or a CCD sensor.

  The input image and output image may be accompanied by imaging parameters such as the focal length and aperture value of the imaging optical system, the imaging distance (subject distance), and various correction information for correcting the input image.

[Image recovery processing]
The input image (degraded image) generated by the imaging device is g (x, y), the original image (undegraded image) is f (x, y), and the optical transfer function (hereinafter referred to as OTF) A point spread function (hereinafter referred to as PSF) that is a Fourier pair is represented by h (x, y). In this case, the following formula (1) is established. Note that * indicates convolution (convolution integration or sum of products), and (x, y) indicates coordinates (position) on the input image.
g (x, y) = h (x, y) * f (x, y) (1)
When this equation (1) is Fourier transformed into a display format on the frequency plane, it becomes a product format for each frequency as in the following equation (2). H (u, v) is an OTF obtained by Fourier transform of h (x, y), which is a PSF. G (u, v) and F (u, v) are functions obtained by Fourier transforming g (x, y) and f (x, y), respectively. (U, v) indicates the coordinates on the two-dimensional frequency plane, that is, the frequency.
G (u, v) = H (u, v) · F (u, v) (2)
In order to obtain the original image from the degraded image, both sides of the above equation (2) may be divided by H (u, v) as in the following equation (3).
G (u, v) / H (u, v) = F (u, v) (3)
F (u, v), that is, G (u, v) / H (u, v) is subjected to inverse Fourier transform to return to the actual surface, thereby obtaining a restored image corresponding to the original image f (x, y). It is done.

Here, assuming that R ⁻¹ (u, v) is an inverse Fourier transform, R is the same as the original image by performing convolution processing on the actual image as in the following equation (4). A recovered image that is f (x, y) can be obtained.
g (x, y) * R (x, y) = f (x, y) (4)
R (x, y) in the equation (4) is an image restoration filter. When the input image is a two-dimensional image, generally the image restoration filter is also a two-dimensional filter having taps (cells) corresponding to the respective pixels of the two-dimensional image. In general, as the number of taps (cells) of the image restoration filter increases, the image restoration accuracy improves. Therefore, it can be realized according to the required image quality as an output image, the image processing capability as an image processing device, the spread width of the PSF, etc. The number of taps is set.

  Since the image restoration filter (first filter) is generated based on the OTF, both the amplitude component and the phase component degradation in the degraded image (input image) can be corrected with high accuracy. For this reason, the image restoration filter is a conventional sharpening filter (edge enhancement filter) of about 3 taps in the horizontal and vertical directions as a second filter generated without being based on information on the optical transfer function or point spread function of the optical system. ) Etc. are completely different.

  Since an actual image contains a noise component, using an image restoration filter created by taking the perfect inverse of OTF as described above not only recovers a degraded image but also greatly amplifies the noise component. End up. This is because the MTF is raised so that the MTF (amplitude component) of the imaging optical system is returned to 1 over the entire frequency in a state where the amplitude of noise is added to the amplitude component of the input image. The MTF, which is amplitude degradation due to the imaging optical system, returns to 1, but the power spectrum of the noise component also rises. As a result, the noise is amplified according to the degree to which the MTF is raised, that is, the recovery gain. Therefore, a good restored image cannot be obtained as an image for viewing from an input image containing a noise component.

  As a method for image restoration of an image including a noise component, for example, a method of controlling the degree of restoration according to the intensity ratio (SNR) of an image signal and a noise signal like a Wiener filter shown in Expression (6) It has been known.

M (u, v) represents the frequency characteristic of the Wiener filter, and | H (u, v) | represents the absolute value (MTF) of the OTF. In an example described later, M (u, v) in Expression (6) indicates the frequency characteristic of the image restoration filter. In this method, for each frequency, the recovery gain is suppressed as the MTF is small, and the recovery gain is increased as the MTF is large. In general, since the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, this is a method of substantially suppressing the recovery gain on the high frequency side of the image signal. A specific example of the image restoration filter will be described later.

  The image restoration filter will be described with reference to FIGS. The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required image restoration accuracy. In FIG. 6, an 11 × 11 tap two-dimensional image restoration filter is shown as an example. In FIG. 6, values (coefficient values) in each tap are omitted, but one cross section of this image restoration filter is shown in FIG. The distribution of the values in each tap of the image restoration filter serves to return the signal value (PSF) spatially spread by the aberration to an original one point ideally.

  In the image restoration process, the value of each tap of the image restoration filter is convolved (also referred to as convolution integration or product sum) with respect to each pixel corresponding to each tap in the input image. In the convolution process, in order to improve the signal value of a certain pixel, the pixel is matched with the center of the image restoration filter. Then, the product of the signal value of the input image and the tap value (coefficient value) of the image restoration filter is taken for each corresponding pixel of the input image and the image restoration filter, and the sum is replaced with the signal value of the central pixel.

  The characteristics of the image restoration processing in the real space and the frequency space will be described with reference to FIGS. 8 (a), 8 (b) and FIG. FIG. 8A shows the PSF before image recovery, and FIG. 8B shows the PSF after image recovery. Further, (a) in (M) of FIG. 9 shows the MTF before image recovery, and (b) in (M) shows the MTF after image recovery. Further, (a) in (P) of FIG. 9 shows the PTF before image recovery, and (b) in (P) shows the PTF after image recovery. The PSF before image restoration has an asymmetric spread, and due to this asymmetry, the PTF has a non-linear value with respect to the frequency. Since the image restoration process amplifies the MTF and corrects the PTF to zero, the PSF after the image restoration is symmetric and sharp.

  The method for creating the image restoration filter can be obtained by performing inverse Fourier transform on a function designed based on the inverse function of the optical transfer function (OTF) of the imaging optical system. For example, when a Wiener filter is used, a real space image restoration filter that is actually convolved with the input image can be created by performing inverse Fourier transform on Equation (1).

  Also, since the optical transfer function (OTF) changes according to the image height (position on the image) of the imaging optical system even under the same imaging conditions, the image restoration filter is changed and used according to the image height. Is done.

  Hereinafter, specific examples of the present invention will be described.

  The image processing (image processing method) that is Embodiment 1 of the present invention will be described with reference to FIG. This processing is executed by an image processing apparatus as a computer according to an image processing program that is a computer program.

  In step S11, the image processing apparatus acquires a captured image generated by the imaging system as an input image. The input image can be acquired from the imaging device via wired or wireless communication, or can be acquired via a recording medium on which the input image is recorded.

  Next, in step S12, the image processing apparatus acquires the imaging condition of the imaging system when the input image is acquired. The imaging conditions are a focal length, an aperture value, an imaging distance (subject distance), and the like of the imaging optical system. When the imaging optical system can be exchanged in the imaging apparatus, the camera ID that is identification information of the imaging apparatus and the lens ID that is identification information of the imaging optical system (exchange lens) are included. The imaging condition information may be acquired directly from the imaging apparatus or may be acquired from information attached to the input image.

  In step S13, the image processing apparatus acquires an image restoration filter corresponding to the imaging condition acquired this time in step S12. Specifically, the image restoration filter corresponding to the imaging condition acquired this time is selected from the image restoration filter data corresponding to various imaging conditions prepared (saved) in advance. At this time, coefficient data for generating an image restoration filter corresponding to each imaging condition is stored, and an image restoration filter corresponding to the imaging condition is selected by selecting a coefficient corresponding to the imaging condition acquired this time. May be generated. In addition, when the image restoration filter (or coefficient) corresponding to the imaging condition acquired this time is not stored, the current acquisition is performed by interpolation processing using the image restoration filter corresponding to a plurality of imaging conditions close to the imaging condition acquired this time. An image restoration filter corresponding to the imaging condition thus obtained may be generated. As a method of the interpolation process, bilinear interpolation (linear interpolation), bicubic interpolation, or the like can be used.

  Next, in step S14, the image processing apparatus starts filter processing for the input image. In step S15, the image processing apparatus (determination unit) determines whether or not an image restoration process can be performed on the input image. For this determination, the number of taps (that is, the number of necessary pixels) in the specific direction of the image restoration filter and the image edge in the specific direction from the processing target pixel that performs the image restoration process by applying the image restoration filter to the input image. The number of pixels up to this pixel (the number of remaining pixels up to the end of the image) is used. Then, it is determined whether or not the pixels necessary for convolution (filter processing) of the image restoration filter are present in the input image, that is, whether or not the number of remaining pixels up to the end of the image is greater than or equal to the required number of pixels.

  If the image restoration processing for the processing target pixel is possible, the image processing apparatus proceeds to step S16, and performs filtering processing (first sharpening processing) using the image restoration filter (first filter) for the processing target pixel. ) Image recovery processing is performed. When the image restoration process for the processing target pixel is impossible, the image processing apparatus proceeds to step S17. In step S <b> 17, a sharpening process as a filter process using a sharpening filter (second filter) generated on the processing target pixel without being based on information on the optical transfer function or point spread function of the optical system. (Second sharpening process) is performed.

  Through the processing in steps S15 to S17, as shown in FIG. 1, the processing target pixel included in the first image region to which the image restoration filter can be applied is recovered from the input image by the image restoration processing. On the other hand, the pixel to be processed included in the second image area that is outside the first image area and cannot be applied with the image restoration filter in the input image is sharpened by the sharpening process.

  As the sharpening filter, for example, as shown in FIG. 4A, there is a filter that applies an effective tap value to pixel data of four pixels on the upper, lower, left, and right of a central processing target pixel. Further, as shown in FIG. 4B, a filter that applies an effective tap value to pixel data of eight pixels around the processing target pixel may be used.

  The sharpening filter may be a 3 × 3 tap filter as shown in FIGS. 4A and 4B, or may be a 5 × 5 tap, 7 × 7 tap, or other image restoration filter. Also, any filter with a small number of taps may be used. In particular, it is preferable to use a sharpening filter having a tap number less than half that of the image restoration filter. Since the sharpening filter also has 3 × 3 taps at the minimum, the pixel to be processed at the edge of the image cannot be filtered because there are not enough pixels to convolve the sharpening filter. However, since the sharpening filter has a smaller number of taps than the image restoration filter, the image area where the filtering process using the sharpening filter cannot be performed becomes smaller. For this reason, it is possible to generate a high-resolution filtered image up to an image region closer to the image end.

  By the way, even when the image restoration filter has 11 × 11 taps, not all 11 × 11 taps have a valid tap value other than 0. In general, an image restoration filter having a larger number of taps is used in consideration of aberration variations due to the focal length of the imaging optical system, the subject distance, and the like. For this reason, as shown in FIG. 10, a tap near the end of the image restoration filter may not have a valid tap value.

  As shown in FIG. 15, when the image restoration filter shown in FIG. 14 is convolved with the processing target pixel of the input image, the pixel corresponding to the tap at the top of the filter is not in the input image. However, since the tap value of the tap is 0, the presence or absence of the corresponding pixel is not related to the convolution. For this reason, it is possible to perform the filtering process (image restoration process) of the processing target pixel by returning 0 for such a pixel. Therefore, when determining the presence / absence of pixels necessary for convolution, it is possible to determine the presence / absence of a pixel corresponding to a tap having a valid tap value in the image restoration filter, so that the processing target pixel closer to the image end is imaged. Recovery processing can be performed.

  Further, when the first image area to which the image restoration filter is applied and the second image area to which the sharpening filter is applied as in the present embodiment, the boundary between the image areas is different because the characteristics of these filters are different. However, an unnatural filtered image may be obtained. Therefore, in order to eliminate such unnaturalness, as shown in FIG. 12, the effect (recovery amount) of the image restoration process becomes smaller in the first image area as it approaches the second image area. It is desirable to perform image restoration processing. Further, in the second image area, it is desirable to perform the sharpening process so that the effect (sharpening amount) of the sharpening process becomes smaller as it approaches the first image area.

  Specifically, as shown in FIG. 13, the weight on the recovery amount by the image recovery filter is decreased from 1 to the second image region, and the boundary where the image recovery filter cannot be applied to the second image region. The pixel is set to 0. Further, the weighting for the sharpening amount by the sharpening filter is decreased from 1 toward the first image area so that it becomes 0 at the boundary pixel to which the image restoration filter can be applied in the first image area. Thereby, it is possible to obtain a filtered image in which unnaturalness due to switching between the first and second image regions (that is, the filter to be applied) is reduced.

  When such processing is performed, there are pixels that are subjected to both image restoration processing and sharpening processing (hereinafter referred to as both processing target pixels). When the image restoration process is performed after the sharpening process, the blur due to the aberration is sharpened by the sharpening process, and thus the image restoration process cannot be performed correctly. For this reason, it is desirable to perform the sharpening process after performing the image restoration process for both the processing target pixels.

  Finally, in step S18, the image processing apparatus checks whether or not the filter processing (image restoration processing or sharpening processing) for all the pixels that can be filtered in the input image is completed. If completed, the image processing apparatus proceeds to step S19 to obtain a filtered image as an output image. On the other hand, if there is a pixel for which the filter processing has not been completed, the image processing apparatus returns to step S15 and performs the processing after step S15 on the pixel for which the filter processing has not been completed.

  FIG. 14 shows the configuration of an image processing system that is Embodiment 2 of the present invention. The image processing system includes a coefficient calculation device 100, a camera 110 as an imaging device, and an image processing device 120.

  The coefficient calculation apparatus 100 calculates a coefficient for reconfiguring the OTF. The coefficient calculation apparatus 100 first performs a process of calculating the OTF from the design value or measurement value of the imaging lens 112 including the imaging optical system. The coefficient calculation apparatus 100 performs a process of converting the OTF into a coefficient, and performs a process of determining the order of the coefficient used later for OTF reconstruction according to the required accuracy. The coefficient calculation apparatus 100 performs a process of determining the number of taps for each image height required when the OTF is reproduced later from the size of the spatial distribution of the PSF. The coefficient calculation apparatus 100 calculates and outputs coefficient data and tap value information up to the required order for combinations of various imaging lenses (interchangeable lenses) and the image sensor 111 provided in the camera 110.

  The camera 100 includes an image sensor 111 and an imaging lens 112. The camera 110 can express information about the lens ID of the imaging lens 112, imaging conditions (aperture, focal length, imaging distance, etc.), and the imaging element 111 in a captured image acquired by imaging using the imaging lens 112, and a spatial frequency. Output with the Nyquist frequency.

  The image processing apparatus 120 holds data and information output from the coefficient calculation apparatus 100 and the camera 110, and performs filtering processing on an input image deteriorated by the imaging lens 112 using the data and information.

  The image restoration processing device 120 includes an image restoration information holding unit 121, an OTF reconstruction unit 122, and a filter processing unit 123. The image restoration information holding unit 121 records and holds the coefficient, the number of taps, the lens ID, the imaging condition, and the Nyquist frequency of the imaging element for each of various combinations of imaging lenses and imaging elements calculated by the coefficient calculation apparatus 100. To do. The OTF reconstruction unit 122 acquires the Nyquist frequency of the image sensor 111, the lens ID of the imaging lens 112, and the imaging conditions from the camera 110. Then, the OTF reconstruction unit 122 searches for the coefficient and the number of taps stored in the image recovery information holding unit 121 from the lens ID and the imaging conditions when the camera 110 performs imaging. The OTF reconstruction unit 122 reconstructs the OTF used by the filter processing unit 123 using the searched coefficient and the number of taps in the spatial frequency region up to the Nyquist frequency of the image sensor. In the following description, the OTF created by the OTF reconstruction unit 122 is referred to as a reconstruction OTF.

  The filter processing unit 123 creates an image restoration filter for recovering the degradation of the captured image using the reconstruction OTF created by the OTF reconstruction unit 122, and performs image restoration processing using the image restoration filter. By holding the coefficient and the number of taps calculated in advance by the coefficient calculation apparatus 100 in the image restoration information holding unit 121, it is not necessary to provide the coefficient calculation apparatus 100 to the user (imager). The user can also download and use information necessary for image restoration processing such as coefficient data through a network or various storage media.

  Next, a coefficient calculation method in the coefficient calculation processing apparatus 100 will be described. In this embodiment, the coefficient is created by approximating the OTF of the imaging optical system by fitting to a function. Here, the function used for fitting is described as a Legendre polynomial, but other than this, a Chebushev polynomial or the like may be used. The Legendre polynomial is given by equation (7). Here, [x] is a maximum integer not exceeding x.

Since OTF is expressed in the form of z = f (x, y), the coefficient a _ij in equation (8) is calculated.

Expression (8) is an orthogonal function, and the value of a _ij is determined regardless of the order at the time of fitting. By utilizing the properties of the orthogonal function, if the OTF fitting can be performed with sufficiently high accuracy even at a low order, the calculation can be terminated at a low order, and the amount of coefficient information to be held in the apparatus is minimized. It becomes possible to reduce the amount.

  FIG. 15 shows a specific method for fitting the OTF using the equations (7) and (8). Fum and fvm shown in FIG. 15 are Nyquist frequencies in the meridional direction and sagittal direction of the OTF, respectively. Nx and Ny are odd tap numbers in the meridional direction and sagittal direction of the OTF, respectively. In the coefficient calculation apparatus 100, a process of calculating a coefficient by the above fitting is performed on each of the real part and the imaginary part of the OTF.

  The real part of the OTF is characterized by being symmetric in the meridional direction and the sagittal direction. The imaginary part of the OTF is symmetric in the meridional direction, but is symmetric in the sagittal direction.

  Due to these symmetries, it is necessary and sufficient if there is information on a quarter of the entire definition area as the OTF data to be fitted. For the above reason, in the present embodiment, in order to fit the OTF with high accuracy, fitting is performed by cutting out a quarter of the entire domain from the OTF so that the DC component is included in both the real part and the imaginary part.

  In the present embodiment, an example in which the OTF data is Nx (row) × Ny (column) tap is shown, and from the OTF data, 1 to [Nx / 2] +1 rows and 1 to [Ny / 2] +1 columns are shown. Extracting data. However, the fitting method is not limited to this.

  FIG. 16 shows the relationship between the Nyquist frequency of the reconstructed OTF and the number of taps in more detail. FIG. 16 shows a cross section of the reconstructed OTF. As described above, the Nyquist frequency is a parameter determined from the spatial resolution of the image sensor 111, and the number of taps is a parameter depending on the PSF of the imaging lens. A desired reconstructed OTF is created from these two parameters and coefficients.

  In FIG. 16, the Nyquist frequency is f_nyq1> f_nyq2, and the number of taps is N> M1> M2. As shown, the Nyquist frequency and the number of taps can be controlled to desired values.

  As described above, the combination of the imaging element and the imaging lens and the OTF corresponding to the imaging case are converted into coefficient data and stored in the image processing device 120, thereby enabling image restoration processing according to the imaging conditions. .

  FIG. 17 shows a basic configuration of an image pickup apparatus that is Embodiment 3 of the present invention. The imaging optical system 201 forms a subject image on light from a subject (not shown). The image sensor 202 converts the subject image into an electrical signal, and the output is converted into a digital signal (imaging signal) by the A / D converter 203 and input to the image processing unit 204. The imaging optical system 201 and the imaging element 202 constitute an imaging system.

  The image processing unit 204 performs predetermined processing on the imaging signal and performs image restoration processing. The state detection unit 207 supplies imaging condition information to the image processing unit 204. As described in the previous embodiment, the imaging conditions include an aperture value, a focal length, an imaging distance, and the like. The state detection unit 207 may obtain information on the imaging conditions directly from the system controller 210, and the imaging conditions related to the imaging optical system 21 may be obtained from the optical system control unit 206. The image processing unit 204 performs the same processing as that described in the first embodiment with reference to FIG. The coefficient data for generating the reconstructed OTF is held in the storage unit 208.

  Then, the output image as the filtered image generated by the image processing unit 204 is stored in the image recording medium 209 in a predetermined format. The display unit 205 displays a filtered image.

  A series of controls in the imaging apparatus is performed by the system controller 210, and mechanical driving of the imaging optical system 206 is performed by the optical system control unit 206 according to an instruction from the system controller 210. The aperture 201a variably sets the aperture value (F number). The position of the focus lens 201b is controlled by an auto focus (AF) mechanism or a manual focus mechanism (not shown) in order to adjust the focus according to the subject distance. The imaging optical system 201 may include an optical element such as a low-pass filter or an infrared cut filter, but an image restoration filter is created when an element that affects the characteristics of an optical transfer function (OTF) such as a low-pass filter is used. It may be necessary to consider at the point of time. The infrared cut filter also affects each PSF of the RGB channel, particularly the PSF of the R channel, which is the integral value of the point spread function (PSF) of the spectral wavelength, and therefore needs to be considered when creating the image restoration filter. It may become. In this case, a rotationally asymmetric transfer function is added after reconfiguring the OTF.

  Further, the imaging optical system 201 may be configured as a part of the imaging device, or may be removable (replaceable) from the imaging device main body having the imaging element 202.

  An image processing system that is Embodiment 4 of the present invention will be described with reference to FIG. The image processing flow is not described because the flow described in the first embodiment can be used as appropriate.

  The image processing apparatus 301 is a computer device equipped with image processing software 306 including the image processing program described in the first embodiment. The imaging device 302 is an imaging device such as a camera, a microscope, an endoscope, or a scanner. The storage medium 303 is a medium that stores captured images, such as a semiconductor memory, a hard disk, and a server on a network.

  The image processing apparatus 301 acquires captured image data from the imaging device 302 or the storage medium 303, and outputs the image data subjected to predetermined image processing to at least one of the output device 305, the imaging device 302, and the storage medium 303. To do. Further, the output destination can be a storage unit built in the image processing apparatus 301. An example of the output device 305 is a printer. A display device 304 that is a monitor is connected to the image processing apparatus 301, and the user can perform image processing work through the display device 304 and observe (evaluate) the output image subjected to the image processing. The image processing software 306 has a function of causing the image processing apparatus 301 to execute the image processing described in the first embodiment, and a function of executing development and other image processing as necessary.

  In the above embodiment, the case where the first sharpening process using the first filter generated based on the information related to the optical transfer function of the optical system is the image restoration process has been described. Is not limited to image restoration processing. For example, unsharp mask processing using a first filter generated based on information about the point spread function of the optical system may be used. That is, the first sharpening process is an unsharp mask that sharpens an image by adding or subtracting the difference between the original image and the image blurred by applying an unsharp mask to the original image. Includes processing. Hereinafter, Example 5 using unsharp mask processing will be described.

[Unsharp mask processing]
FIG. 19 is a schematic diagram of sharpening by unsharp mask processing (sharpening processing). The solid line in FIG. 19A is an input image, the broken line is an image obtained by blurring the input image with an unsharp mask, and the dotted line is sharpened. The subsequent image is represented, and the solid line in FIG. 19B represents the correction component. In FIG. 19, the horizontal axis represents coordinates, and the vertical axis represents pixel values or luminance values. FIG. 19 corresponds to a cross section of the image in a predetermined direction (for example, the X direction).

If the original image is g (x, y) and the correction component is c (x, y), the sharpened image f (x, y) can be expressed by the following equation (9).
f (x, y) = g (x, y) + m × c (x, y) (9)
In Expression (9), the correction signal c (x, y) is multiplied by a constant and added to g (x, y) as an input image. In Expression (9), m is a constant. The amount of correction can be adjusted by changing the value of m. Note that m may be a constant regardless of the position of the input image, or may be adjusted according to the position of the input image by using an adjustment coefficient m (x, y) that varies depending on the position of the input image. The correction amount can also be adjusted. Further, the constant m and the adjustment coefficient m (x, y) can be varied according to the imaging conditions such as the focal length, aperture value, and subject distance of the optical system. The adjustment factor m (x, y) can be used instead of the constant m in the following description.

The correction component c (x, y) can be expressed as the following equation (10), where the unsharp mask is USM. USM (x, y) is, for example, a tap value at a certain coordinate (x, y) of USM.
c (x, y) = g (x, y) -g (x, y) * USM (x, y) (10)
* Is convolution (convolution integral, sum of products), and δ is a delta function (ideal point image). The delta function is data in which the number of taps is equal to USM (x, y) and the center value is 1 and the rest are filled with 0.

When the right side of Expression (10) is modified, it can be expressed by the following Expression (11).
c (x, y) = g (x, y) * (δ (x, y) −USM (x, y)) (11)
Since Expression (11) can be expressed by modifying Expression (10), Expression (10) and Expression (11) are equivalent. Therefore, the generation of the correction component will be described below using Equation (10).

  In Expression (10), the difference between the captured image g (x, y) and the image obtained by blurring the captured image g (x, y) with the unsharp mask USM is calculated, and the correction component c (x , Y). In general unsharp mask processing, a smoothing filter such as a Gaussian filter, a median filter, or a moving average filter is used for the unsharp mask USM.

  For example, when a Gaussian filter is used as the unsharp mask USM for the captured image g (x, y) shown by the solid line in FIG. 19A, an image obtained by blurring the captured image g (x, y) is shown in FIG. As indicated by the broken line in FIG. The correction component c (x, y) is a difference between the captured image g (x, y) and an image obtained by blurring the image, as shown in the equation (10). Therefore, the correction component c (x, y) is changed from the solid line in FIG. By subtracting the broken line in A), the component is represented by the solid line in FIG. By using the correction component calculated in this way and performing the calculation of Expression (9), the captured image f (x, y) shown by the solid line in FIG. 19A is represented by the dotted line in FIG. Can be sharpened.

Next, a case where an image is sharpened by applying an unsharp mask process to an image deteriorated by an imaging optical system that forms an optical image of a subject will be described. A captured image g (x, y) obtained through the imaging optical system is an image (image of a subject) before imaging (I (x, y)), and a PSF that is a function representing a response to the point light source of the imaging optical system. When psf (x, y) is given, it can be expressed as the following equation (12).
g (x, y) = I (x, y) * psf (x, y) (12)
If the imaging optical system is a rotationally symmetric coaxial optical system, the PSF corresponding to the center of the image is rotationally symmetric. Therefore, it is possible to sharpen the captured image g (x, y) closer to the original image I (x, y) by applying a rotationally symmetric USM to the center of the image. Since the correction amount is a difference value between the captured image and an image obtained by blurring the image with an unsharp mask, the unsharp mask USM does not use a simple smoothing filter to correct with high accuracy. It is better to use a mask having a shape close to x, y). For example, when a captured image deteriorates due to the influence of spherical aberration, the spherical symmetry affects rotational symmetry, but a smoothing filter such as a Gaussian filter has a distribution shape different from that of PSF due to the influence of spherical aberration. . For this reason, even when the effect of blurring in rotational symmetry is reduced, it is possible to correct with higher accuracy by using the point spread function (PSF) of the imaging optical system.

  Note that the PSF of the imaging optical system differs depending on the imaging conditions including the image height of the image formed through the optical system, the focal length of the optical system, the F value, and the subject distance, and therefore a different PSF is used for each imaging condition. It is better to execute unsharp mask processing.

Unsharp mask processing can be performed using the following equation (13) derived from equations (9) and (10).
f (x, y) = g (x, y) + m × {g (x, y) −g (x, y) * USM (x, y)}
(13)
Further, the unsharp mask process can be executed using the equations (14), (15), and (16) derived by transforming the equation (13).

f (x, y) = g (x, y) + m × g (x, y) * {δ (x, y) −USM (x, y)}
(14)
f (x, y) = g (x, y) * {δ (x, y) + m × (δ (x, y) −USM (x, y))}
... (15)
f (x, y) = g (x, y) * {(1 + m) × δ (x, y) −m × USM (x, y)}
... (16)
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

DESCRIPTION OF SYMBOLS 201 Image pick-up optical system 202 Image pick-up element 204 Image processing part 301 Image processing apparatus

Claims (12)

Image acquisition means for acquiring an input image generated by imaging using an optical system;
Processing means for performing image processing on the input image;
The processing means includes the input image,
A first sharpening process is performed on the first image area using a first filter generated based on information related to the optical transfer function or point spread function of the optical system,
The second image area closer to the image end than the first image area has a smaller number of taps than the first filter, and is based on information related to the optical transfer function or point spread function of the optical system. An image processing apparatus that performs a second sharpening process using a second filter that is generated without any problem.   The image processing apparatus according to claim 1, wherein the first sharpening process is an image restoration process using a first filter generated based on information related to an optical transfer function of the optical system. .   3. The image according to claim 2, wherein the first sharpening process is an unsharp mask process using a first filter generated based on information relating to a point spread function of the optical system. Processing equipment.   The said 2nd image area | region is an area | region which does not contain the number of pixels required in order to apply a said 1st filter around the process target pixel. An image processing apparatus according to 1.   Using the number of taps of the first filter and the number of pixels from the processing target pixel to the image end in the input image, the first filter or the second filter is applied to the processing target pixel The image processing apparatus according to claim 4, further comprising a determination unit configured to determine whether to perform the process.   The image processing apparatus according to claim 5, wherein the number of taps of the first filter is the number of taps having a tap value that is not zero.   The processing means performs the first sharpening process in the first image area so that the effect of the first sharpening process becomes smaller as it approaches the second image area. The image processing apparatus according to claim 1.   The processing means performs the second sharpening process in the second image area so that the effect of the second sharpening process becomes smaller as it approaches the first image area. The image processing apparatus according to claim 1. The processing means includes
Providing a third image area for performing both the first sharpening process and the second sharpening process between the first image area and the second image area;
The image processing apparatus according to claim 1, wherein the second sharpening process is performed after the first sharpening process is performed on the third image region. An image sensor for imaging a subject image formed by an optical system;
An image pickup apparatus comprising: the image processing apparatus according to claim 1. Obtaining an input image generated by imaging using an optical system;
Processing steps for performing image processing on the input image,
In the processing step, of the input images,
A first sharpening process is performed on the first image area using a first filter generated based on information related to the optical transfer function or point spread function of the optical system,
The second image area closer to the image end than the first image area has a smaller number of taps than the first filter, and is based on information related to the optical transfer function or point spread function of the optical system. A second sharpening process is performed using a second filter generated without any other process. A computer program for causing a computer to execute image processing on an input image generated by imaging using an optical system,
Of the input images to the computer,
First sharpening processing is performed on the first image region using a first filter generated based on information related to the optical transfer function or point spread function of the optical system,
The second image area closer to the image end than the first image area has a smaller number of taps than the first filter, and is based on information related to the optical transfer function or point spread function of the optical system. An image processing program that causes a second sharpening process to be performed using a second filter that is generated without any problem.