JP2019087851A - Image processing device, imaging apparatus, image processing method, image processing program, and recording medium - Google Patents

Abstract

To provide an image processing device capable of reducing noise amplification due to sharpening while reducing a processing load.SOLUTION: An image processing unit 104 includes: first acquisition means 104a configured to acquire noise information relevant to noise characteristics in an input image which is generated by picking up using an optical system 101; second acquisition means 104b configured to acquire a sharpening filter in a real space on the basis of optical characteristics of the optical system 101; and third acquisition means 104c configured to acquire gain information of the sharpening filter on the basis of components of the sharpening filter. The image processing unit 104 performs sharpening processing on the input image on the basis of the noise information and the gain information.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an image processing apparatus for improving the quality of a captured image.

  In imaging using an optical system, light generated from one point of a subject reaches the image plane with a minute spread under the influence of diffraction or aberration generated in the optical system. Therefore, blurring occurs due to diffraction or aberration of the optical system in the photographed image.

  It is known that such image blurring can be sharpened by image processing using a point spread function (PSF), a point spread function (OTF), an optical transfer function (OTF) or the like. However, when trying to sharpen an image, there is a problem that the noise of the image is amplified.

  Patent Document 1 describes that information on noise characteristics (ISO sensitivity) is used to adjust a correction effect in order to reduce noise amplification when sharpening an image.

JP 2017-41763 A

  In the method of Patent Document 1, the image correction effect is adjusted by controlling the gain value of the two-dimensional image restoration filter in the frequency space. Therefore, it is necessary to perform two-dimensional inverse Fourier transform in image processing. Therefore, the method of Patent Document 1 has a problem that the load required for image processing is increased.

  An object of the present invention is to provide an image processing apparatus capable of reducing amplification of noise accompanying sharpening while reducing processing load.

  An image processing apparatus according to the present invention comprises: a first acquisition unit for acquiring noise information on noise characteristics in an input image generated by photographing using an optical system; and sharpening in real space based on the optical characteristics of the optical system A second acquisition unit that acquires a filter; and a third acquisition unit that acquires gain information of the sharpening filter based on a sum of squares of components of the sharpening filter; Sharpening processing is performed based on the noise information and the gain information.

  In addition, another image processing apparatus according to the present invention includes a first acquisition unit for acquiring noise information on noise characteristics in an input image generated by photographing using an optical system, and an actual based on optical characteristics of the optical system. A second acquisition unit that acquires a sharpening filter in a space; and a third acquisition unit that acquires gain information of the sharpening filter at a specific frequency based on a component of the sharpening filter; Sharpening processing is performed on the input image based on the noise information and the gain information.

  According to the present invention, it is possible to realize an image processing apparatus capable of reducing the amplification of noise accompanying sharpening while reducing the processing load.

It is a figure explaining sharpening by unsharp mask processing. It is a schematic diagram which shows PSF of an optical system. It is a figure explaining the sharpening process using rotation-symmetrical unsharp mask. It is a figure explaining the sharpening processing using rotation non-symmetrical unsharp mask. It is a schematic diagram which shows the gain of a sharpening filter. It is a schematic diagram of MTF before and behind sharpening processing. It is a block diagram of an imaging device. 5 is a flowchart of image processing in Embodiment 1. It is a figure which shows the relationship between gain information and a weighted addition rate. FIG. 7 is a diagram for explaining generation of a sharpening filter in Embodiment 2. 7 is a flowchart of image processing in Embodiment 2.

  Hereinafter, an embodiment of an imaging device having an image processing unit as an image processing device of the present invention will be described based on the attached drawings.

  The image processing unit according to the present embodiment performs unsharp mask processing on an input image generated by photographing using an optical system, as a sharpening process using a sharpening filter based on optical characteristics of the optical system. First, prior to the description of the configuration of the imaging apparatus of the present embodiment, the sharpening processing of the present embodiment will be described.

  FIG. 1 is a schematic view of sharpening by unsharp mask processing. The solid line in FIG. 1A represents an input image, the broken line represents an image obtained by blurring the input image with an unsharp mask, and the dotted line represents an image after sharpening. The solid line in FIG. 1 (b) represents the correction component. The horizontal axis of each drawing of FIG. 1 is a coordinate, and the vertical axis is a pixel value or a luminance value. FIG. 1 corresponds to a cross section in a predetermined direction (for example, the x direction) of FIG. 2 described later.

Assuming that the input image is f (x, y) and the correction component is h (x, y), the image g (x, y) after sharpening can be expressed by the following equation.
g (x, y) = f (x, y) + m x h (x, y) (1)

  In Equation (1), m is an adjustment coefficient for changing the strength of the correction, and the amount of correction can be adjusted by changing the value of the adjustment coefficient m. Note that m may be a constant constant regardless of the position of the input image, or the correction amount may be adjusted according to the position of the input image by making it different according to the position of the input image. Further, the adjustment coefficient m (x, y) can be made different according to the imaging conditions such as the focal length of the optical system, the aperture value, and the object distance (imaging distance).

Assuming that the unsharp mask is USM (x, y), the correction component h (x, y) can be expressed by the following equation. USM (x, y) is, for example, a tap value at coordinates (x, y).
h (x, y) = f (x, y)-f (x, y) * USM (x, y) (2)

The right side of equation (2) can be modified and expressed by the following equation.
h (x, y) = f (x, y) * (δ (x, y)-USM (x, y)) (3)

  Here, “*” is convolution (convolution integral, product-sum), and “δ (x, y)” is a delta function. The delta function is data in which the number of taps is equal to that of USM (x, y) and the value of the center is 1 and other values are 0.

  Because equation (3) can be expressed by modifying equation (2), equation (2) and equation (3) are equivalent. Therefore, the generation of the correction component will be described below using Equation (2).

  In equation (2), the difference between the input image f (x, y) and the image obtained by blurring the input image f (x, y) with the unsharp mask is calculated, and the correction component h (x, y) is calculated based on this difference information. Is generated. In general unsharp mask processing, smoothing filters such as Gaussian filter, median filter, moving average filter, etc. are used for the unsharp mask.

  For example, when a Gaussian filter is used as an unsharp mask for an input image f (x, y) shown by a solid line in FIG. 1A, an image obtained by blurring the input image f (x, y) is shown in FIG. It becomes as shown by the broken line of a). The correction component h (x, y) is a component shown by the solid line in FIG. 1 (b) obtained by subtracting the broken line in FIG. 1 (a) from the solid line in FIG. 1 (a). By performing the operation of equation (1) using the correction component calculated in this manner, the input image f (x, y) shown by the solid line in FIG. 1A is shown by the dotted line in FIG. Can be sharpened.

Next, the case where an image is sharpened by applying an unsharp mask process to an image degraded by the influence of diffraction or aberration generated in an optical system will be described. An input image f (x, y) generated by photographing using an optical system is an image (hereinafter referred to as an original image) when degradation due to the optical system does not occur is I (x, y), the optical system Let psf (x, y) be the PSF of f, and can be expressed as in the following equation.
f (x, y) = I (x, y) * psf (x, y) (4)

  First, the case where the input image f (x, y) is blurred in rotational symmetry with respect to the original image I (x, y) will be described. If the optical system is a rotationally symmetric coaxial optical system, the PSF corresponding to the center of the image is rotationally symmetric. Therefore, by applying a rotationally symmetric unsharp mask to the central portion of the image, it is possible to sharpen the input image f (x, y) closer to the original image I (x, y). The correction component is the difference value between the input image and the image obtained by blurring the input image with the unsharp mask. Therefore, to correct with high precision, it is not a simple smoothing filter as the unsharp mask, but more psf (x, y) It is preferable to use a shape close to.

  For example, consider the case where the input image is degraded due to the influence of spherical aberration. If it is a spherical aberration, it affects the rotational symmetry with respect to the center of the screen, but the smoothing filter such as the Gaussian filter has a distribution shape different from that of PSF reflecting the influence of the spherical aberration. Therefore, by using the PSF of the optical system for the unsharp mask, the input image can be corrected (sharpened) with high accuracy.

  Therefore, in the present embodiment, the PSF of the optical system is used as the unsharp mask USM (x, y). The input image f (x, y) shown in FIG. 1A has a line-symmetrical shape for simplification, but the image may not have an axis of symmetry. Even if the shape of the original image I (x, y) is asymmetric, if the degradation function (corresponding to psf (x, y)) to be convoluted to the original image I (x, y) is rotationally symmetric Sharpening can be performed using a rotationally symmetric unsharp mask.

  Next, the case where the input image f (x, y) is blurred rotationally asymmetrically with respect to the original image I (x, y) will be described. Even if the optical system is a rotationally symmetric coaxial optical system, PSFs at positions other than the center of the image usually have an asymmetrical shape. FIG. 2 is a schematic view of the PSF of the optical system in the xy plane, FIG. 2 (a) shows an on-axis PSF, and FIG. 2 (b) shows an off-axis PSF.

  For example, assuming that the original image is composed of a set of ideal point images, from Equation (4), the input image f (x, y) becomes the PSF of the optical system. If there is an ideal point image at the angle of view corresponding to FIG. 2B and the original image is deteriorated by the influence of the PSF of the optical system, the input image is blurred as shown in the shape of FIG. Image. A description will be given of the case of performing sharpening by unsharp mask processing on an image that is asymmetrically blurred as described above.

  FIG. 3 and FIG. 4 are diagrams for explaining the case where the unsharp mask processing is performed on the image which is blurred asymmetrically. FIG. 3 shows the case where processing is performed using a rotationally symmetric unsharp mask, and FIG. 4 shows the case where processing is performed using a rotationally asymmetric unsharp mask. The vertical axis of each of FIGS. 3 and 4 is a pixel value or a luminance value. Moreover, the horizontal axis of each figure of FIG. 3, FIG. 4 is corresponded to the y direction of FIG.2 (b).

  The solid lines in FIGS. 3A and 4A represent the cross section in the y-axis direction in FIG. 2B, and the dotted lines represent the input image blurred by the unsharp mask. A Gaussian filter is applied to the rotationally symmetric unsharp mask in FIG. 3, and a PSF of an optical system is applied to the rotationally asymmetric unsharp mask in FIG.

  FIGS. 3 (b) and 4 (b) are plots of the difference values between the input image and the input image blurred by each unsharp mask, and represent correction components. In FIGS. 3A and 4A, the side where the base of the input image is wider is taken as the plus side of the Y axis.

  In FIG. 3A, the difference between the input image and the original image is small on the plus side with respect to the peak position of the solid line, and the difference between the input image and the original image is large on the minus side. Therefore, the extremum on the minus side is smaller than the extremum on the plus side with respect to the peak position of the center of the correction component in FIG. 3B.

  As can be understood by comparing the curves in FIG. 3A and FIG. 3B, the correction amount on the plus side is small, and the correction amount on the minus side is large. For this reason, it is not possible to correct an asymmetric blur even if sharpening is performed according to equation (4). FIG. 3 (c) shows the result after sharpening when m = 1, and although the sharpening is achieved with respect to the solid line in FIG. 3 (a), the negative side is larger than the positive side. It can be seen that the asymmetrical blur can not be corrected.

  Here, a case is considered in which the correction amount is adjusted by changing the adjustment coefficient m of Expression (4) without changing the unsharp mask. If the value of the adjustment coefficient m is increased to sufficiently correct the positive side of the image, the negative side of the image becomes overcorrected (undershoot), and the adjustment coefficient m is adjusted so that the amount of correction on the negative side of the image becomes appropriate. If the value of is set, the positive side of the image is undercorrected.

  As described above, it is difficult to sharpen the asymmetrically degraded input image even if the asymmetrically blurred image is subjected to the unsharp mask processing using the rotationally symmetric unsharp mask. Such a problem similarly occurs even if a rotationally symmetric filter other than the Gaussian filter is used as the rotationally symmetric unsharp mask.

  On the other hand, in FIG. 4A, the difference value between the input image and the original image is large on the plus side with respect to the solid peak position, and the difference value between the input image and the original image is small on the minus side. That is, the opposite tendency to FIG. 3A appears. Therefore, the extremum of the correction component in FIG. 4B is smaller on the plus side than on the minus side with respect to the central peak position.

  By applying the correction component shown in FIG. 4 (b) to the input image represented by the solid line in FIG. 4 (a), the correction amount is added on the plus side (in the larger blur) to the peak position. The correction amount can be made smaller on the larger side and the negative side (the smaller the blur).

  In the case of such an asymmetric unsharp mask, since the balance tendency of the blurring of the input image and the balance of the correction amount of the correction component coincide with each other, excess or deficiency of correction which becomes a problem when applying a rotationally symmetric unsharp mask is also It becomes difficult to get up. FIG. 4 (c) shows the result after sharpening when m = 1, which can be sharpened with respect to the solid line in FIG. 4 (a), and occurs in FIG. 3 (c). The sink on the negative side has been improved.

  Furthermore, since the correction is less likely to be overcorrected as compared with the rotationally symmetric unsharp mask, the value of the adjustment coefficient m in equation (4) can also be made relatively large, and sharpening is achieved while reducing asymmetry. can do.

  Further, in order to further improve the correction accuracy, it is necessary that the portion blurred more largely by the PSF of the optical system be largely blurred by the unsharp mask. Therefore, it is preferable to use the PSF of the optical system as the unsharp mask even when the input image f (x, y) is blurred rotationally asymmetrically with respect to the original image I (x, y). In this case, the sharpening filter is a two-dimensional filter in which components are distributed rotationally asymmetrically. The sharpening filter is a coefficient matrix that is convoluted with the input image in the sharpening process, and the component of the sharpening filter is a component (tap value) of the coefficient matrix.

  The PSF of the optical system is different for each imaging condition including the focal length of the optical system, the F value of the optical system, and the subject distance (imaging distance). Also, the PSF differs depending on the position in the input image. Therefore, it is preferable to change the sharpening filter for each imaging condition with respect to each position in the input image.

  Next, amplification of noise components included in the input image will be described.

  In general, an image generated by imaging using an optical system includes a component deteriorated by noise in addition to a component deteriorated by PSF of the optical system.

  Since the sharpening process is a process that brings the MTF (Modulation Transfer Function), which is the amplitude component of the optical transfer function of the optical system, closer to 1, the noise component is also amplified when the noise component is included in the input image. There is a risk of That is, when the above-described sharpening processing is performed on the input image, the input image is sharpened and the noise component included in the input image is easily noticeable.

  This can be explained as follows using equations (1) and (3).

From equations (1) and (3), the image g (x, y) after the sharpening process can be expressed as the following equation.
g (x, y) = f (x, y) + m × f (x, y) * {δ (x, y) −USM (x, y)} (5)

If the right side of equation (5) is summarized for the input image f (x, y), equation (6) is obtained.
g (x, y) = f (x, y) * {δ (x, y) + m × (δ (x, y) −USM (x, y))} (6)

In equation (6), the term in the braces on the right side corresponds to the sharpening filter. By Fourier transforming Equation (6), the following Equation (7) can be obtained.
G (u, v) = F (u, v) × {1 + m × (1-U (u, v))} (7)

  G (u, v) in equation (7) is the Fourier transform of g (x, y), F (u, v) is the Fourier transform of f (x, y), U (u, v) is USM (x, y) y) Fourier transform.

  FIG. 5 shows the absolute values in parentheses {} of equation (7). The horizontal axis in FIG. 5 is the spatial frequency, and the vertical axis is the gain. Assuming that the real part of the Fourier transform U (u, v) of the unsharp mask USM (x, y) is Re (U (u, v)) and the imaginary part is Im (U (u, v)), the unsharp mask The gain Ga in the process is expressed by the following equation (8).

  If the unsharp mask USM (x, y) is a rotationally symmetric Gaussian distribution, then the function U (u, v) also has a Gaussian distribution. Therefore, the imaginary part Im (U (u, v)) is 0, the real part Re (U (u, v)) is 0 ≦ Re (U (v, v) ≦ 1 and the gain Ga is 1 ≦ Ga ≦ It becomes (1 + m). In addition, the real part Re (U (u, v)) approaches zero as the Gaussian distribution approaches zero as it leaves the center of the distribution. Therefore, as the frequency becomes higher, the gain Ga gradually approaches (1 + m), and a curve as shown in FIG. 5 is obtained. Note that FIG. 5 shows the case where the adjustment coefficient m is 1, and the gain Ga approaches 2 at the high frequency side.

  When the PSF of the optical system is used as the unsharp mask USM (x, y), the Fourier transform U (u, v) of the unsharp mask USM (x, y) is an optical transfer function (OTF) which is a Fourier transform of PSF ). The absolute value of OTF is the amplitude component MTF.

  FIG. 6 is a diagram showing changes in the amplitude component MTF before and after sharpening by unsharp mask processing. The solid line in FIG. 6 represents the amplitude component MTF before the sharpening process, and the broken line represents the amplitude component MTF after the sharpening process.

  FIG. 6 shows the frequency characteristic of the amplitude component of image degradation due to aberration or the like. As shown in FIG. 6, the low frequency side is high and the high frequency side is low. As the amplitude component MTF approaches zero, the real and imaginary parts of OTF approach zero, and beyond the frequency that can be resolved by the optical system, the real and imaginary parts of OTF become zero. At this time, since the gain Ga (u, v) is (1 + m) according to equation (8), even when PSF of the optical system is used as the unsharp mask USM (x, y), the gain Ga is a dotted line in FIG. It becomes a curve which increases on the high frequency side shown by a broken line. The gain Ga is constant in any direction (cross section) when the PSF is rotationally symmetric, but varies depending on the direction when the PSF is rotationally asymmetric. Therefore, the gain Ga is also rotationally asymmetric when the PSF is rotationally asymmetric. Further, since the real part and imaginary part of OTF may be negative, gain Ga may be Ga> (1 + m) at a predetermined frequency.

  As described above, even when using a rotationally symmetric filter such as a Gaussian distribution as the unsharp mask USM (x, y) or using the PSF of the optical system, the gain Ga is It becomes a curve which increases from the low frequency side to the high frequency side.

  When the amplitude components MTF before and after the sharpening process are compared, the difference is the largest at the frequency fr. This is because the amplitude component MTF before the sharpening process shown by the solid line in FIG. 6 is multiplied by the gain Ga shown in FIG. 5 to become the amplitude component MTF after the sharpening process. When the amplitude component MTF before the sharpening process is low, even if the gain Ga is large, the change in the amplitude component MTF due to the sharpening becomes small.

Here, it is assumed that the input image contains a noise component. Assuming that the noise component contained in the input image is n (x, y), equation (6) can be expressed as equation (9) below.
g (x, y) = (f (x, y) + n (x, y)) * {δ (x, y) + m × (δ (x, y) −USM (x, y))} (9)

Similarly, when the Fourier transform of n (x, y) is N (u, v), the equation (7) can be expressed as the following equation (10).
G (u, v) = (F (u, v) + N (u, v)) * {1 + m * (1-U (u, v))} (10)

  According to equation (10), when the unshark mask process is performed on the input image including the noise component, the Fourier transform F (u, v) of f (x, y) which is a component other than noise in the input image and the noise Parts of parentheses {} are applied to both components N (u, v). Further, since F (u, v) is obtained by Fourier transforming Equation (4), it is the product of the Fourier transform of the original image I (x, y) and the OTF of the optical system. On the high frequency side where the amplitude component MTF approaches zero, as described above, the real part and imaginary part of OTF approach zero, so the Fourier transform F (u, v) of the input image f (x, y) is also the solid line in FIG. It also approaches zero as indicated by. On the other hand, considering a noise component such as white noise as the noise component N (u, v), a uniform distribution is obtained without depending on the frequency. Therefore, at the high frequency side, the ratio of N (u, v) to F (u, v) is large. Therefore, if a large gain is applied to the high frequency side with respect to the input image including the noise component, the noise component Is amplified.

  Therefore, in order to correct the input image well, it is necessary to estimate the gain in the sharpening process and adjust the degree of sharpening by the sharpening process based on the gain and the noise characteristic of the input image.

  The gain of the sharpening filter can be obtained using equation (8), but the real-space sharpening filter needs to be two-dimensionally Fourier transformed. Therefore, when the gain is calculated using the equation (8), the amount of calculation increases and the processing load required for the sharpening process increases.

  On the other hand, when the gain calculated beforehand is stored, although the amount of calculation required to obtain the gain is reduced, it is necessary to allocate a storage capacity to store the gain. Although it is necessary to change the sharpening filter for each shooting condition in order to correct the input image properly, if you try to store the gain for each sharpening filter that is different for each shooting condition, a huge storage capacity will be obtained. Needs to be allocated for the storage of

  Therefore, in the present embodiment, as described later, the gain is estimated from the sharpening filter in the real space, and the degree of sharpening by the sharpening processing is adjusted based on the gain and the noise characteristic of the input image. As a result, while reducing the processing load required for the sharpening process, it is possible to suppress the amplification of the noise accompanying the sharpening.

  The sharpening processing performed in the image processing unit of the present embodiment has been described above.

  Next, the configuration of the imaging device of the present embodiment will be described.

  An imaging device 100 according to the present embodiment will be described with reference to FIG. FIG. 7 is a block diagram showing the configuration of the imaging device 100. As shown in FIG. An image processing program for performing the sharpening process of the present embodiment is installed in the imaging apparatus 100, and the sharpening process is executed by an image processing unit (image processing apparatus) 104 in the imaging apparatus 100. .

  The imaging device 100 includes an optical system 101 and an imaging device main body (camera main body).

  The optical system 101 includes an aperture 101a and a focus lens 101b, and is integrally configured with the camera body. However, the present invention is not limited to this, and is also applicable to an imaging apparatus in which the optical system 101 is detachably mounted to the camera body. The optical system 101 may be configured to include an optical element having a diffractive surface and an optical element having a reflective surface, in addition to an optical element having a refractive surface such as a lens.

  The imaging device 102 is formed of a CCD sensor or a CMOS sensor, and photoelectrically converts an object image (an optical image formed by the optical system 101) obtained by the optical system 101 to generate a photographed image. That is, the subject image is converted into an analog signal (electric signal) by photoelectric conversion by the imaging device 102. Then, this analog signal is converted into a digital signal by the A / D converter 103. This digital signal is input to the image processing unit 104.

  The image processing unit 104 performs predetermined processing on the digital signal and performs the sharpening processing of the present embodiment. As shown in FIG. 7, the image processing unit 104 includes a photographing condition acquisition unit (first acquisition unit) 104a, a sharpening filter acquisition unit (second acquisition unit) 104b, and a gain information acquisition unit (third acquisition). Means) 104c and a processing unit (processing means) 104d. The imaging condition acquisition unit 104 a acquires imaging conditions of the imaging device 100 from the state detection unit 107. The imaging conditions include the aperture value of the optical system 101, the imaging distance, and the focal length. The state detection unit 107 may acquire the imaging conditions directly from the system controller 110 or may acquire it from the optical system control unit 106.

  Data necessary for generating PSF or PSF is held in the storage unit (storage means) 108. The storage unit 108 is configured by, for example, a ROM. The output image processed by the image processing unit 104 is stored in the image recording medium 109 in a predetermined format. On the display unit 105 configured by a liquid crystal monitor or an organic EL display, an image obtained by performing a predetermined process for display on the image subjected to the sharpening process is displayed. However, the image displayed on the display unit 105 is not limited to this, and the image subjected to the simplified processing for high-speed display may be displayed on the display unit 105.

  The system controller 110 controls the imaging device 100. Mechanical driving of the optical system 101 is performed by the optical system control unit 106 based on an instruction of the system controller 110. The optical system control unit 106 controls the aperture diameter of the diaphragm 101a so as to obtain a predetermined F number. Further, the optical system control unit 106 controls the position of the focus lens 101b by an auto focus (AF) mechanism (not shown) or a manual manual focus mechanism (not shown) in order to perform focus adjustment according to the subject distance. Note that functions such as aperture diameter control of the aperture stop 101 a and manual focusing may not be performed according to the specifications of the imaging device 100.

  Although an optical element such as a low pass filter or an infrared cut filter may be disposed between the optical system 101 and the imaging element 102, a sharpening filter is formed when using an element such as a low pass filter that affects optical characteristics. It may be necessary to consider at the time of The infrared cut filter also needs to be considered at the time of creating the sharpening filter because it affects the PSF of the RGB channel, which is an integral value of the PSF of the spectral wavelength, particularly the PSF of the R channel. Therefore, the sharpening filter may be changed according to the presence or absence of the low pass filter or the infrared cut filter.

  The image processing unit 104 is configured by an ASIC, and the optical system control unit 106, the state detection unit 107, and the system controller 110 are each configured by a CPU or an MPU. Further, one or more of the image processing unit 104, the optical system control unit 106, the state detection unit 107, and the system controller 110 may be configured to be shared by the same CPU or MPU.

  Next, an embodiment of the sharpening process performed in the image processing unit 104 will be described.

Example 1
FIG. 8 is a flowchart of the sharpening process of the first embodiment. The flowchart in FIG. 8 is executed based on an instruction from the image processing unit 104. In FIG. 8, "S" represents a step. The flowchart in FIG. 8 can also be embodied as a program for causing a computer to execute each step to exhibit the same function as the image processing unit 104. The same applies to the flowchart shown in FIG.

  In S101, the image processing unit 104 acquires an image captured by the imaging device 100 as an input image. The input image is stored in the storage unit 108. Further, an image stored in the image recording medium 109 may be acquired as an input image.

  In S102, the imaging condition acquisition unit 104a acquires imaging conditions at the time of imaging of the input image. The shooting conditions include the focal length of the optical system 101, the aperture value, the shooting distance, and the like. In the case of an imaging apparatus in which the optical system 101 is exchangeably mounted on the camera body, the imaging conditions further include a lens ID and a camera ID. The imaging conditions may be acquired directly from the imaging device or may be acquired from information attached to the input image (for example, EXIF information).

  Further, in S102, the imaging condition acquisition unit 104a acquires noise information on noise characteristics of the input image. The noise information may be any information that can be associated with the magnitude of noise included in the input image. The noise information includes, for example, the sensitivity of the image sensor 102 and the temperature. In addition, if the image processing is performed on the input image prior to the sharpening processing, noise characteristics of the input image may be affected. Therefore, the noise information may include information on image processing performed on the input image prior to the sharpening processing. In addition, in a general imaging apparatus, ISO sensitivity is recorded as a sensitivity of the imaging device together with the above-described imaging condition together with the photographed image, so it is preferable to use the ISO sensitivity as the noise information. In the following, an example in which ISO sensitivity is used as noise information will be described.

  In S103, the sharpening filter acquisition unit 104b acquires the optical characteristics of the optical system 101 from the storage unit 108. Here, PSF is acquired as an optical characteristic. However, as an optical characteristic, one corresponding to PSF such as OTF, or coefficient data of a function that approximately represents PSF or OTF may be acquired.

  In S104, the sharpening filter acquisition unit 104b acquires a sharpening filter corresponding to the imaging condition acquired in S102 based on the optical characteristic acquired in S103. In the present embodiment, in the sharpening filter in the equation (6), the sharpening filter is acquired using the PSF acquired in S103 as USM (x, y).

  In addition, when the PSF corresponding to the imaging condition acquired in S103 is not stored, among the PSFs stored in the storage unit 108, the PSF corresponding to the imaging condition closest to the imaging condition acquired in S103 is used. A sharpening filter may be acquired. In addition, a sharpening filter based on PSF obtained by interpolating PSF of a shooting condition close to the shooting condition obtained in S103 is obtained, or a sharpening filter based on a PSF near the shooting condition obtained in S103 is interpolated Alternatively, the sharpening filter obtained may be acquired.

  Further, although the sharpening filter is obtained based on the optical characteristics in the present embodiment, the present invention is not limited to this. The sharpening filter generated based on the optical characteristics of the optical system 101 may be stored in advance in the storage unit 108, and the sharpening filter acquisition unit 104b may directly acquire the sharpening filter from the storage unit 108. In this case, S103 is skipped.

  Note that in S104, the sharpening filter acquisition unit 104b acquires a plurality of sharpening filters to be applied to each of a plurality of positions of the input image. Since the optical characteristics (PSF) at each position of the input image are different, each sharpening filter applied to each position of the input image is different.

  Also, the sharpening filter acquired in S104 is a filter in real space, not in frequency space. In the case of acquiring a filter in frequency space as a sharpening filter, it is necessary to perform an inverse Fourier transform when sharpening an input image, but if a filter in real space is acquired, it is possible to apply it directly to the image . Therefore, the processing load required for the sharpening process can be reduced. This makes it possible to improve the calculation speed of the sharpening process. Further, since the processing load required for the sharpening process is reduced, the sharpening process can be realized with a relatively inexpensive chip. This can reduce the product cost.

  In S105, the gain information acquisition unit 104c acquires the gain information of the sharpening filter acquired in S104. The gain information is information on data that can be handled in the same manner as the gain of the sharpening filter. The gain information may be information on the gain at a specific frequency of the sharpening filter, or information on a value obtained by averaging the gains of the sharpening filter with respect to the frequency.

  As described above, in the present embodiment, gain information is acquired from the components of the sharpening filter in the real space. As a method for acquiring gain information from a real space sharpening filter, there is a method using Perceval's equation.

  Sharpening filter in real space k (x, y), Fourier transform of k (x, y) K (fx, fy), tap number of k (x, y) in x direction Nx, tap in y direction Assuming that the number is Ny, the following Parseval equation holds.

  That is, by calculating the sum of squares of the components of the sharpening filter in the real space, it is possible to calculate the average of the sum of squares for the frequency components of the gain without performing Fourier transform. That is, an average value for the frequency of the gain of the sharpening filter can be calculated.

  Also, as another method for acquiring gain information from the component of the sharpening filter in the real space, as shown in the following equation (12), the product of the component of the real space sharpening filter and a specific frequency component There is a way to take the sum of In this case, it is possible to calculate only the value of the gain at a specific frequency of the sharpening filter.

  The gain that can be obtained by equation (12) is the gain for a specific frequency in the gain characteristics shown in FIG. By calculating only the gain for a specific frequency, the processing load can be suppressed and gain information can be acquired rather than performing two-dimensional Fourier transform. The value of the “exp” portion in the equation (12) corresponding to the specific frequency component may be stored in the storage unit 108 in advance.

  By doing this, it is possible to obtain information on the gain in the frequency space without performing two-dimensional Fourier transform. The gain may be obtained by calculation using another equation that can obtain the same result as equation (11) or equation (12).

  In addition, when using values in shooting conditions different from the shooting conditions of the input image as gain information in the shooting conditions of the input image, or interpolating from values in the shooting conditions different from the shooting conditions of the input image, sufficient accuracy is obtained. There are cases where gain information can not be obtained. On the other hand, in the present embodiment, since gain information is obtained directly from the sharpening filter, it is not necessary to interpolate the gain information with respect to the imaging conditions. Therefore, gain information can be acquired with high accuracy.

  In the present embodiment, the gain information acquisition unit 104c acquires the gain information of each sharpening filter acquired in S104 based on the components of each sharpening filter, but the present invention is not limited to this. That is, for all the sharpening filters applied to each screen position of the input image, it is not necessary to acquire gain information from the components of each sharpening filter. For example, using gain information (first gain information) acquired using a component of the sharpening filter (first sharpening filter) for a specific position (first position), another position (second Sharpening processing for the position of

  Specifically, gain information is acquired based on the component of the sharpening filter only for the sharpening filter applied to the partial screen position of the input image, and the gain information at the other screen positions is the peripheral screen position. You may acquire by interpolating from gain information. Thereby, the processing load at the time of acquiring gain information can be reduced. In this case, although interpolation with respect to the screen position in the input image is performed, interpolation with respect to the imaging conditions is not performed, so it is possible to obtain sufficiently accurate gain information while reducing the processing load.

  In S106, the weighted addition rate (proportion of the input image in the output image) in the sharpening process is determined from the noise information acquired in S102 and the gain information acquired in S105. An example of the method of determining the weighted addition rate will be described with reference to FIG.

  FIG. 9 is a view showing a weighted addition rate to gain information at a predetermined ISO sensitivity. In FIG. 9, when the gain information is smaller than a, the weighted addition rate is 0%. That is, when the gain information is smaller than a, the image after correction using the sharpening filter (g (x, y) in Expression (6)) becomes the output image as it is.

  Further, in FIG. 9, when the gain information is larger than b, the weighted addition rate is 80%. That is, an image obtained by weighted averaging at a ratio of 80% for the input image and 20% for the image after correction is the output image. This is equivalent to adding 20% of the difference between the image after correction and the image before correction to the image before correction, and such calculation may be performed. That is, the value of the adjustment coefficient m in equation (6) may be adjusted.

  As described above, by performing the sharpening process by determining the weighted addition rate based on the gain information and the noise information, the signal of the input image before the correction can be increased when the gain becomes large. In other words, the degree of sharpening by the sharpening process can be weakened based on the gain information and the noise information when the influence of noise amplification is considered to be large. As a result, amplification of noise by the sharpening process can be reduced by a simple process.

  The relationship between the gain information and the weighted addition rate as shown in FIG. 9 may be stored in the storage unit 108 in association with the noise information. For example, when the noise information has ISO sensitivity, the relationship between the gain information and the weighted addition rate may be stored as table data for each value that can be taken as the ISO sensitivity. In addition, the values of a in typical ISO values and the inclination between a and b are stored, and the relationship between gain information and weight addition rate in unstored ISO sensitivity is calculated from the stored relationship. You may

  In step S107, the image processing unit 104 executes the sharpening processing of the input image using the sharpening filter acquired in step S104. That is, the input image is sharpened by convolving the sharpening filter with the input image.

  In the present embodiment, the weighted addition rate (or the adjustment coefficient m) is determined based on the gain information and the noise information to output the output image, but the present invention is not limited to this. The processing in S106 and S107 may be processing which reduces amplification of noise by the sharpening processing based on gain information and noise information.

Example 2
Next, the sharpening process of the second embodiment will be described.

  In the first embodiment, gain information of the acquired sharpening filter is acquired, and the sharpening process is performed using the same sharpening filter. Thereby, gain information can be accurately acquired for the filter that performs the sharpening process. On the other hand, in this embodiment, the gain information (first gain information) acquired using the component of the sharpening filter (first sharpening filter) for a specific position (first position) is used to Acquire a sharpening filter (second sharpening filter) for the position (second position) of Then, the second sharpening processing is performed using the second sharpening filter. As a result, the gain information can be estimated more easily, and the processing load of the sharpening process can be further reduced.

  Prior to the description of the method of acquiring gain information in the present embodiment, acquisition of the sharpening filter in the present embodiment will be described with reference to FIG.

  FIG. 10A shows the position on the input image at which the sharpening filter is generated. The white circles in FIG. 10 (a) represent the position on the input image at which the sharpening filter is generated. In this embodiment, a sharpening filter is generated based on the optical characteristics of the optical system 101 for discrete 81 positions on the input image. By performing linear interpolation or the like on these generated sharpening filters, it is possible to generate sharpening filters at arbitrary positions on the input image. This makes it possible to apply an appropriate sharpening filter to any position on the input image while reducing the processing load for generating the sharpening filter.

  In FIG. 10A, the position to generate the sharpening filter is 9 × 9 = 81. However, the position may be further reduced to further reduce the processing load, and to further improve the correction accuracy. Position may be increased.

  The optical characteristics can be obtained directly for each point of the white circle in FIG. 10A to generate a sharpening filter, but these optical characteristics or the sharpening filter may also be generated by interpolation. The example is shown in FIG. 10 (b), which shows the case where the optical characteristic at each position is generated by interpolation.

  First, the optical characteristic at the position of the black circle in FIG. 10B is acquired. In general, since the PSF of the optical system is rotationally symmetric with respect to the optical axis, the optical characteristics are also rotationally symmetric with respect to the optical axis. By using this feature, in FIG. 10 (b), optical characteristics at a plurality of positions (10 places indicated by black circles) having different image heights (distance from the center of the input image) are obtained, Optical characteristics at positions corresponding to each white circle are generated by interpolation while being rotated with respect to the center. Then, based on the optical characteristics at each position, a sharpening filter at each position is acquired. As a result, it is not necessary to acquire optical characteristics for each position indicated by a white circle, so that processing load and data amount can be reduced.

  Here, although 81 points of optical characteristics are obtained by interpolation from optical characteristics of 10 places, a sharpening filter is first obtained from 10 points of optical characteristics to interpolate the sharpening filter 81 A sharpening filter for points may be acquired.

  Next, a method of acquiring gain information in the present embodiment will be described. In FIG. 10 (b), in order to obtain gain information accurately for each position, it is necessary to calculate the gain information based on the sharpening filter at each position. On the other hand, gain information at each image height can be obtained by obtaining a sharpening filter corresponding to the optical characteristic of the black circle in FIG. 10B and calculating gain information from the sharpening filter. That is, it is possible to acquire the change in gain information with respect to the image height.

  By using the gain information of each information calculated in this manner, it is possible to estimate the gain information of the sharpening filter generated at the position corresponding to each white circle in FIG. In this embodiment, the gain information of each image height is used to adjust the gain of each sharpening filter generated at the position corresponding to each white circle in FIG. Sharpening processing is performed with a certain degree. This suppresses the amplification of noise accompanying sharpening while reducing the processing load.

  That is, although the gain information is calculated for each sharpening filter generated in the first embodiment, the gain information for each image height is acquired in the present embodiment, and the input image is obtained using the gain information for each image height. The second embodiment differs from the first embodiment in that a sharpening filter for each position is acquired.

  When gain information is to be acquired for each sharpening filter generated at the position corresponding to each white circle in FIG. 10B, the gain information for 81 points is given by equation (11), equation (12), etc. It is necessary to calculate based on On the other hand, according to the present embodiment, it is sufficient to calculate the gain information based on the equation (11) or the equation (12) for only the ten places indicated by the black circles in FIG. The processing load to calculate can be further reduced.

  FIG. 11 is a flowchart of the sharpening process in the second embodiment. The flowchart in FIG. 11 is executed based on an instruction from the image processing unit 104.

  Since S201 to S203 are the same as S101 to S103, the description will be omitted.

  In S204, the sharpening filter acquisition unit 104b acquires a sharpening filter (first sharpening filter) at each black circle position based on the optical characteristic (PSF) at the position indicated by the black circle in FIG. 10B. Do.

  In step S205, as in the first embodiment, gain information of each image height is acquired by acquiring gain information from each first sharpening filter.

  In S206, the sharpening filter acquisition unit 104b performs the sharpening filter (second filter) at the position of each white circle based on the gain information of each image height, the PSF at the position indicated by the black circle in FIG. Acquire the sharpening filter). At this time, the sharpening filter acquisition unit 104b generates a second sharpening filter based on the gain information and noise information of each image height so that the gain of the second sharpening filter becomes a desired value. Specifically, the gain of each second sharpening filter is adjusted by adjusting the adjustment coefficient m in Equation (6).

  The gain of the second sharpening filter acquired in S206 may be adjusted based on the gain information and noise information at each image height. For this reason, the second sharpening filter can be obtained by various methods. For example, a PSF at a position indicated by a black circle may be obtained from PSF at a position indicated by a black circle, and a second sharpening filter may be generated based on the PSF at a position indicated by a white circle and gain information and noise information for each image height. . In this case, the adjustment factor m is determined for each of the second sharpening filters generated for each white circle position.

  Also, based on the PSF at the position indicated by the black circle, and the gain information and noise information for each image height, the adjustment coefficient m is determined for each position indicated by the black circle, and the gain is determined for each position indicated by the black circle. Produces a sharpening filter with adjusted. The sharpening filter whose gain is adjusted may be rotated with respect to the center of the input image and interpolated to generate a second sharpening filter at each white circle position. In this case, it is only necessary to determine the adjustment coefficient m for each black circle position.

  Further, based on the PSF at the position indicated by the black circle and the gain information and the noise information at each image height, the adjustment coefficient m is determined for each position indicated by the black circle. Using the PSF at each position of m shown by each image height and the black circle determined in this way, m and PSF at the position shown by each black circle (corresponding to USM (x, y) in equation (6) Get the product of). The product of m and PSF at the position indicated by each black circle is rotated with respect to the center of the input image, and interpolation is performed to obtain m and PSF at each white circle position. The second sharpening filter may be generated using the product of m and PSF acquired in this manner. Also in this case, the adjustment coefficient m may only be determined for each black circle position. In step S207, the processing unit 104d executes a sharpening process. The sharpening process in step S207 is performed by convolving the second filter acquired in step S206 with the input image.

  As described above, it is possible to obtain sufficiently accurate gain information by simpler processing compared to the first embodiment. By this, it is possible to suppress amplification of noise accompanying sharpening while further reducing processing load.

  In each of the above embodiments, the unsharp mask processing using PSF is described as the sharpening processing, but the present invention is not limited to this. The sharpening process in the present embodiment may be a process using a sharpening filter based on the optical characteristics of the optical system 101. For example, unsharp mask processing or edge enhancement processing without using PSF may be used. Further, it may be an image restoration process represented by a Wiener filter, or an iterative process type image restoration process represented by an RL method.

  Note that each processing in the sharpening processing in each embodiment does not have to be performed by a single image processing apparatus. For example, a program for realizing part or all of the processing in each embodiment may be supplied to a system consisting of one or more devices via a network or a recording medium, and the system or device may execute the program. good.

  Although the preferred embodiments and examples of the present invention have been described above, the present invention is not limited to these embodiments and examples, and various combinations, modifications, and changes are possible within the scope of the present invention.

104 Image processing unit (image processing apparatus)
104a Shooting condition acquisition unit (first acquisition unit)
104 b Sharpening filter acquisition unit (second acquisition unit)
104c Gain Information Acquisition Unit (Third Acquisition Means)

Claims (19)

First acquisition means for acquiring noise information on noise characteristics in an input image generated by photographing using an optical system;
A second acquisition unit for acquiring a sharpening filter in real space based on the optical characteristics of the optical system;
And third acquisition means for acquiring gain information on a gain of the sharpening filter based on a square sum of components of the sharpening filter,
An image processing apparatus characterized by performing sharpening processing on the input image based on the noise information and the gain information. First acquisition means for acquiring noise information on noise characteristics in an input image generated by photographing using an optical system;
A second acquisition unit for acquiring a sharpening filter in real space based on the optical characteristics of the optical system;
And third acquisition means for acquiring gain information on the gain of the sharpening filter at a specific frequency based on the component of the sharpening filter,
An image processing apparatus characterized by performing sharpening processing on the input image based on the noise information and the gain information.   The image processing apparatus according to claim 1, wherein the optical characteristic is an optical characteristic under a photographing condition at the time of photographing the input image.   The image processing apparatus according to any one of claims 1 to 3, wherein the optical characteristic is a point image intensity distribution of the optical system.   The image processing apparatus according to any one of claims 1 to 4, wherein the noise information is an ISO sensitivity.   The image processing apparatus according to any one of claims 1 to 5, wherein the sharpening process is an unsharp mask process. The sharpening filter is a two-dimensional filter,
The image processing apparatus according to any one of claims 1 to 6, wherein the components of the sharpening filter are distributed rotationally asymmetrically.   The image processing apparatus according to any one of claims 1 to 7, wherein the second acquisition unit acquires different sharpening filters for each of a plurality of positions in the input image. .   The third acquisition means is characterized in that gain information is acquired for each of the sharpening filters acquired by the second acquisition means based on the components of the respective sharpening filters. 8. The image processing apparatus according to 8.   The sharpening process is performed by adding the input image and an image obtained by correcting the input image using the sharpening filter at a ratio determined based on the noise information and the gain information. The image processing apparatus according to claim 9, wherein The third acquisition unit acquires first gain information using a component of a sharpening filter for a first position in the input image;
9. The image processing apparatus according to claim 8, wherein a sharpening process is performed on a second position different from the first position using the first gain information and the noise information. The second acquisition unit acquires a sharpening filter for the second position based on the first gain information and the noise characteristic,
12. The image processing apparatus according to claim 11, wherein the sharpening processing for the second position is performed using a sharpening filter for the second position. The third acquisition means acquires gain information at each image height of the input image using components of a first sharpening filter for each of a plurality of positions of different image heights in the input image;
9. The image processing apparatus according to claim 8, wherein the sharpening process is performed on the input image using gain information and the noise information at each of the image heights. The second acquisition unit acquires a second sharpening filter for each of a plurality of positions in the input image based on the gain information at each of the image heights and the noise characteristic.
The image processing apparatus according to claim 13, wherein the sharpening process is performed using the second sharpening filter.   An image pickup apparatus comprising: an image pickup element for photoelectrically converting an image formed by an optical system; and the image processing apparatus according to any one of claims 1 to 14. A first step of acquiring noise information on noise characteristics in an input image generated by photographing using an optical system;
A second step of acquiring a sharpening filter in real space based on the optical characteristics of the optical system;
A third step of acquiring gain information of the sharpening filter based on a sum of squares of components of the sharpening filter;
A fourth step of performing a sharpening process on the input image based on the noise information and the gain information;
An image processing method comprising: A first step of acquiring noise information on noise characteristics in an input image generated by photographing using an optical system;
A second step of acquiring a sharpening filter in real space based on the optical characteristics of the optical system;
A third step of acquiring gain information of the sharpening filter at a specific frequency based on a component of the sharpening filter;
A fourth step of performing a sharpening process on the input image based on the noise information and the gain information;
An image processing method comprising:   An image processing program for causing a computer to execute the image processing method according to claim 16 or 17.   A computer-readable recording medium on which the image processing program according to claim 18 is recorded.