JP2021027565A - Image processing method, imaging apparatus employing the same, image processing device, and image processing program - Google Patents

Abstract

To provide an image processing method capable of executing highly accurate sharpening processing while reducing the quantity of information required for the sharpening processing.SOLUTION: An image processing method includes: an acquisition step of acquiring a captured image which is generated by imaging through an optical system, and information on a point image intensity distribution function corresponding to an imaging condition of the optical system; and a processing step of performing unsharp mask processing on the captured image using a filter which is generated based on the information on the point image intensity distribution function. An accessory optical system which changes optical characteristics is attachable to and detachable from the optical system, and the filter is generated from information on the accessory optical system and the information on the point image intensity distribution function at the time when the accessory optical system is not mounted.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image sharpening process.

Conventionally, an unsharp mask process for sharpening an image by adding or subtracting a difference between an image blurred by applying an unsharp mask to the original image and the original image to the original image has been known. The larger the difference between the blurred image and the input image, the sharper the image. Further, in Patent Document 1, the influence of the point image intensity distribution function (PSF: Point Spread Function) of the optical system is reduced by applying an asymmetric one-dimensional filter to the pixel signal sequences arranged in the image height direction. The method of doing so is disclosed.

JP-A-2010-81263

However, the conventional unsharp mask processing uses a rotationally symmetric filter for the unsharp mask, and sharpens the deteriorated image under the influence of PSF having a complicated shape such as asymmetric aberration and sagittal halo. It is difficult. That is, when trying to correct the aberration in the Azimus direction having a large aberration, an undershoot occurs in the Azimus direction having a small aberration. On the contrary, if the undershoot is suppressed, the aberration cannot be sufficiently corrected.

Further, the above-mentioned method of Patent Document 1 considers only the asymmetry in the image height direction, and the correction filter is also one-dimensional, so that the asymmetry in the direction other than the image height direction should be improved. I can't. The image height direction is the meridional direction of Azimus. Also, regarding the filter, the asymmetry of the filter is adjusted by the number of minus tap coefficients, and the correction in the image height direction is also different from the blurring of the PSF of the optical system, so the conventional method should be sufficiently sharpened. I can't.

Also, conversion lenses used to change the focal length of the lens are known. Since the PSF of the imaging optical system changes by attaching / detaching or inserting / removing the conversion lens, it is necessary to apply an appropriate unsharp mask depending on the presence / absence of the conversion lens in order to perform sharpening processing with higher accuracy. ..

Therefore, an object of the present invention is to provide an image processing device, an imaging device, an image processing method, and an image processing program capable of performing high-precision sharpening processing while reducing the amount of information required for sharpening processing. And.

In order to achieve the above object, the image processing method according to the present invention is
It is generated based on the captured image generated by imaging through the optical system, the acquisition step of acquiring the information of the point image intensity distribution function corresponding to the imaging conditions of the optical system, and the information of the point image intensity distribution function. It has a processing step of performing unsharp mask processing on the captured image using the filter, and the optical system has an accessory optical system that changes the optical characteristics, and the accessory optical system can be attached and detached. It is characterized in that the filter is generated from the information of the point image intensity distribution function when the accessory optical system is not attached.

According to the present invention, there is provided an image processing device, an imaging device, an image processing method, and an image processing program capable of performing high-precision sharpening processing while reducing the amount of information required for sharpening processing. Can be realized.

Flow chart showing the image processing method in Examples 1, 2 and 3. Block diagram of the image pickup apparatus in Examples 1 and 4. Schematic diagram of sharpening by unsharp mask processing in each example Schematic diagram of PSF of the imaging optical system on the xy plane in each embodiment Schematic diagram of sharpening treatment by rotationally symmetric unsharp mask in each example Schematic diagram of sharpening treatment by rotationally asymmetrical unsharp mask in each example Schematic and schematic cross-sectional views of the unsharp mask in each embodiment Schematic diagram of Bayer array in each example Block diagram of the image processing unit in the first embodiment Contour map of approximate PSF Flow chart showing the calculation method of the coefficient in each embodiment The figure which shows the approximate PSF and the design value in each Example Diagram for explaining the rotation processing of the point image intensity distribution function Sectional view of the reconstructed point image intensity distribution function Block diagram of the imaging device according to the second embodiment Block diagram of the imaging device according to the third embodiment Flow chart showing the image processing method in the fourth embodiment

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[Input image]
The input image is a digital image (captured image) generated by using the output from the image sensor obtained by photoelectrically converting the subject image (optical image) formed through the image pickup optical system (hereinafter, simply referred to as an optical system) in the image pickup device. ). This digital image is an image deteriorated by an optical transfer function (OTF) including aberrations of an optical system including an optical element such as a lens or an optical filter. The image sensor is composed of a photoelectric conversion element such as a CMOS sensor or a CCD sensor. The imaging optical system may include a mirror (reflecting surface) having a curvature. Further, the optical system may be detachable (replaceable) with respect to the image pickup apparatus. In the image pickup apparatus, the image pickup system is configured by a signal processing circuit that generates a digital image (input image) using the optical system, the image pickup element, and the output of the image pickup element.

The color component of the input image has, for example, information on the RGB color component. In addition to this, as the treatment of color components, a commonly used color space such as brightness, hue and saturation expressed by LCH, brightness and color difference signal expressed by YCbCr, etc. can be selected and used. .. As other color spaces, for example, XYZ, Lab, Yuv, and JCh can be used, and further, the color temperature can be used.

The input image and output image include information on shooting conditions such as focal length, aperture value (F value), imaging distance, and image height of the optical system in the imaging device when the input image is generated (imaging) (hereinafter, shooting condition information). ) Can be attached. Further, various correction information for correcting the input image can be attached to the input image and the output image. When an input image is output from the image pickup device to an image processing device provided separately from the image pickup device and image processing is performed by this image processing device, it is preferable to attach shooting condition information and correction information to the input image. .. In addition to being incidental to the input image, the shooting condition information and the correction information can be directly or indirectly passed from the image pickup apparatus to the image processing apparatus by communication.

[Image sharpening process]
FIG. 3 is a schematic view of sharpening by the unsharp mask processing (image sharpening processing) of the present embodiment. In FIG. 3A, the solid line shows the input image, the broken line shows the image obtained by blurring the input image with an unsharp mask, and the dotted line shows the sharpened image. The solid line in FIG. 3B shows the correction component. In each of FIGS. 3A and 3B, the horizontal axis is the coordinates and the vertical axis is the pixel value or the luminance value. FIG. 3 corresponds to a cross section in a predetermined direction (for example, the X direction) of FIG. 4 described later.

Assuming that the original image is f (x, y) and the correction component is h (x, y), the sharpened image g (x, y) can be represented by the following equation (1).
g (x, y) = f (x, y) + m × h (x, y) ... (1)
In the formula (1), m is an adjustment coefficient for changing the strength of the correction, and the correction amount can be adjusted by changing the value of the adjustment coefficient m. The adjustment coefficient m may be a constant constant regardless of the position of the input image, or the correction amount can be adjusted according to the position of the input image by making it different according to the position of the input image. Further, the adjustment coefficient m (x, y) can be made different depending on the shooting conditions such as the focal length, the aperture value, and the shooting distance of the optical system.

The correction component h (x, y) can be expressed by the following equation (2), where the unsharp mask is USM (x, y). USM (x, y) is, for example, a tap value at coordinates (x, y).
h (x, y) = f (x, y) -f (x, y) * USM (x, y) ... (2)
By transforming the right side of equation (2), the following equation (3) can be obtained.
h (x, y) = f (x, y) * (δ (x, y) -USM (x, y)) ... (3)
In equations (2) and (3), * is a convolution (convolution integral, sum of products), and δ is a delta function (ideal point image). The "delta function" is data in which the number of taps is the same as that of USM (x, y), the center value is 1, and the other values are filled with 0.

Since the equation (3) can be expressed by transforming the equation (2), the equation (2) and the equation (3) are equivalent to each other. Therefore, the generation of the correction component will be described below using the equation (2).

In the formula (2), the difference between the captured image f (x, y) and the image obtained by blurring the captured image f (x, y) with an unsharp mask is taken, and the correction component h (x, y) is taken based on this difference information. ) Is generated. In general unsharp mask processing, a smoothing filter such as a Gaussian filter, a median filter, or a moving average filter is used for the unsharp mask.

For example, when a Gaussian filter is used as an unsharp mask for the captured image f (x, y) shown by the solid line in FIG. 3 (A), the captured image f (x, y) is blurred. 3 (A) will be indicated by the broken line. Since the correction component h (x, y) is the difference between the captured image f (x, y) and the blurred image as shown in the equation (2), the solid line in FIG. 3 (A) to FIG. 3 (A) ) Is subtracted to obtain the component represented by the solid line in FIG. 3 (B). By performing the calculation of the equation (1) using the correction component calculated in this way, the captured image f (x, y) shown by the solid line in FIG. 3 (A) is converted to the dotted line in FIG. 3 (A). It can be sharpened like this.

Next, a case will be described in which the image is sharpened by applying the unsharp mask processing to the image deteriorated by the imaging optical system that forms the optical image of the subject.

The captured image f (x, y) obtained through the imaging optical system is a point image which is a function representing the response of the imaging optical system to a point light source, with the image before imaging (subject image) being I (x, y). Assuming that the intensity distribution function PSF is psf (x, y), it can be expressed by the following equation (4).
f (x, y) = I (x, y) * psf (x, y) ... (4)
Here, if the imaging optical system is a rotationally symmetric co-axis optical system, the PSF corresponding to the central portion of the image is rotationally symmetric. Therefore, by applying a rotationally symmetric USM to the central portion of the image, it is possible to sharpen the captured image f (x, y) so that it approaches the original image I (x, y). Since the correction amount is the difference between the captured image and the captured image blurred by the unsharp mask, in order to correct accurately, instead of using a simple smoothing filter as the unsharp mask, psf (x, It is preferable to use a mask having a shape close to y). For example, when the captured image is deteriorated due to the influence of spherical aberration, the spherical aberration affects the rotational symmetry, but the smoothing filter such as the Gaussian filter has a different distribution shape from the PSF due to the influence of spherical aberration. Therefore, even when the influence of rotationally symmetric blurring is reduced, it is possible to correct the correction more accurately by using the PSF of the imaging optical system.

In this embodiment, PSF is used as the unsharp mask USM (x, y). The captured image f (x, y) shown in FIG. 3A has a symmetrical shape for simplification, but the shape of the image does not have to be symmetrical. Even if the shape of the original image I (x, y) is asymmetric, if the deterioration function of the original image I (x, y) corresponding to psf (x, y) is rotationally symmetric, the rotationally symmetric Ann It can be sharpened using a sharp mask.

On the other hand, the PSF usually has an asymmetric shape at a position other than the center of the image even if the imaging optical system is a rotationally symmetric co-axis optical system. 4A and 4B are schematic views of the PSF of the imaging optical system in the xy plane, FIG. 4A shows the PSF on the axis, and FIG. 4B shows the PSF off the axis.

For example, assuming that the original image (subject) is an ideal point image, the captured image f (x, y) is the PSF of the imaging optical system from the equation (4). Assuming that there is an ideal point image at the angle of view corresponding to FIG. 4B and the original image (subject) is deteriorated due to the influence of the PSF of the imaging optical system, the image obtained as the input image is shown in FIG. 4B. The image becomes blurred like the shape of). A case where sharpening the asymmetrically blurred image by unsharp mask processing will be described.

5 and 6 are schematic views of unsharpening processing for an asymmetrically deteriorated image, FIG. 5 shows a rotation-symmetrical unsharpening mask, and FIG. 6 shows processing using a rotation-asymmetrical unsharpening mask. Each case is shown. The vertical axis and the horizontal axis are the same as in FIG. 3, respectively.

The solid lines in FIGS. 5 (A) and 6 (A) represent the cross section in the y-axis direction of FIG. 4 (B), and the dotted lines represent the photographed images blurred by the unsharp mask. A Gaussian filter is applied to the rotationally symmetric unsharp mask of FIG. 5, and the PSF of the imaging device is applied to the rotationally asymmetrical unsharp mask of FIG.

5 (B) and 6 (B) are plots of the difference values between the photographed image blurred by each unsharp mask and the original photographed image, and represent the correction components. For convenience, in FIGS. 5 (A) and 6 (A), the side in which the captured image is blurred by the PSF and has a wider base is defined as the plus side of the Y axis.

In FIG. 5A, the difference value between the blurred image on the plus side and the original image is small with respect to the peak position of the solid line, and the difference value between the blurred image on the minus side and the original image is large. Therefore, the extremum of the correction component in FIG. 5B is smaller on the left side (minus side) than on the right side (plus side) with respect to the central peak position.

As can be seen by comparing the curves of FIGS. 5 (A) and 5 (B), the correction amount of the correction component is small on the positive side of the captured image, and the correction amount is large on the negative side with a narrow base. ) Cannot correct the asymmetrical blur even if it is sharpened.

FIG. 5C shows the result after sharpening when m = 1, and although sharpening is achieved with respect to the solid line in FIG. 5A, the negative side with respect to the positive side is It can be seen that the asymmetrical blur has not been corrected due to the large dent. For example, consider a case where the correction amount is adjusted by changing the adjustment coefficient m of the equation (4) without changing the unsharp mask. If the value of the adjustment coefficient m is increased in order to sufficiently correct the positive side of the image, the negative side of the image becomes overcorrected (undershoot). On the other hand, if the value of the adjustment coefficient m is set so that the correction amount on the minus side of the image is appropriate, the correction on the plus side of the image becomes insufficient.

As described above, even if the unsharp mask processing is performed using the rotationally symmetric unsharp mask on the asymmetrically blurred image, it is difficult to improve the asymmetry and sharpen the image. Such a problem also occurs when a rotationally symmetric filter other than the Gaussian filter is used as the rotationally symmetric unsharp mask.

On the other hand, in FIG. 6A, the difference value between the image with the positive side blurred and the original image is large with respect to the peak position of the solid line, and the difference value between the image with the negative side blurred and the original image is small. Is the opposite of FIG. 5 (A). Therefore, the extremum of the correction component in FIG. 6B is smaller on the right side (plus side) than on the left side (minus side) with respect to the central peak position. When such a correction component is applied to the captured image represented by the solid line in FIG. 6 (A), the correction amount is larger on the side where the positive side blur is larger than the peak position, and the negative side blur is large. The smaller the value, the smaller the correction amount. In the case of such an asymmetrical unsharp mask, the tendency of the balance of the blur of the input image and the balance of the correction amount of the correction component match, so that the overcorrection that becomes a problem when applying the rotationally symmetric unsharp mask Shortages are less likely to occur.

FIG. 6 (C) shows the result after sharpening when m = 1, and the solid line in FIG. 6 (A) was also sharpened and was conspicuous in FIG. 5 (C). The difference in balance between the dents on the minus side and the dent on the plus side has improved. Further, since the correction is less likely to be excessive as compared with the case of the rotationally symmetric unsharp mask, the value of the adjustment coefficient m in the equation (4) can be made relatively large, and the asymmetry is reduced and sharpened. can do. Further, in order to perform correction with higher accuracy, the balance of the correction amount of the correction component is the difference between the blurred image and the original image, so that the portion largely blurred by the PSF of the imaging optical system is covered by the unsharp mask. It needs to be more blurred than the other parts. As described above, in order to correct the correction more accurately, it is ideal to use the PSF of the imaging optical system as an unsharp mask.

The example of the unsharp mask using PSF as the image sharpening process has been shown above, but PSF is used for the image restoration process represented by the Wiener filter and the iterative processing type image restoration process represented by the RL method. You may. It can also be used to generate DL (deep learning) learning images, which have been studied in recent years.

[Coefficient data]
The coefficients used to generate the unsharp mask USM will be described.

As described above, in each embodiment, the PSF of the imaging optical system is used as an unsharp mask. At this time, since the PSF of the imaging optical system changes depending on the imaging conditions such as the image height, focal length, F value, and imaging distance, it is necessary to generate an unsharp mask according to the imaging conditions when sharpening. .. In order to deform the unsharp mask according to the PSF that changes depending on the imaging condition, it is conceivable to calculate all the combinations and select the PSF corresponding to the imaging condition. However, such a method is not preferable from the viewpoint of the processing speed when applying the unsharp mask and the recording capacity for holding data.

Therefore, in each embodiment, the coefficient data for approximating the PSF of the imaging optical system is retained, and the PSF is reconstructed (approximate) using the coefficient data when creating the unsharp mask. This makes it possible to maximize the effect of sharpening while reducing the amount of data to be retained. In each embodiment, as a method of approximating the PSF to create an unsharp mask, data of a continuous function and its coefficient are used as described below.

First, a continuous function used for approximating the PSF of the imaging optical system will be described. As described above, when fitting a photometric celestial body in the field of astrophysics, a function P (x, y) shown in the following equation (5) called a Moffat function is often used.

Here, α and β in the equation (5) are coefficients, and particularly when β = 1, it is called a Lorentz function. For example, when the PSF is modeled using the equation (5), these coefficients α and β are obtained by fitting the distribution of the PSF calculated by measurement or calculation with the equation (5). Then, the PSF can be modeled by using the calculated coefficients α and β and the equation (5). As described above, although the approximate PSF can be created by the equation (5), the equation (5) is a function capable of expressing only the rotationally symmetric distribution with respect to the coordinates x and y. Therefore, it is not possible to create a rotation asymmetric distribution or a distribution having N rotation symmetry (N is an integer) using the equation (5).

As a modified function of the equation (5), there is a function represented by the equations (6) (and (6a)) called an Elliptical Moffat function capable of expressing an elliptical shape (distribution having two-fold rotational symmetry).

Α, β, and γ in equation (6) are coefficients. Equation (6a) is a rotation matrix with an angle θ. The equation (6) and the equation (6a) can be summarized as the following equation (7).

In equation (7), a, b, c, σ, and β are coefficients. In addition, in order to maintain the elliptical shape when the equation (7) is used, it is necessary to satisfy the relationship of ^{b 2-ac <0 with respect to the coefficients a, b, and c.}

By using these equations (7) (or equations (6)), it is possible to reproduce the elliptical distribution that cannot be expressed by the function of equation (5), so it is corrected compared to the case of using the function of equation (5). The accuracy of is improved. However, even if the function of the equation (7) is used for fitting the PSF of the imaging optical system, it is not possible to reproduce a complicated shape such as asymmetric aberration or sagittal halo. That is, in the equation (7), although a distribution having rotational symmetry or N-fold rotational symmetry can be expressed, a rotationally asymmetric distribution cannot be expressed.

Therefore, in each embodiment, the functions shown in the following equations (8) ((8a) to (8c)) are used as functions that can reproduce PSF having a complicated shape such as asymmetric aberration of the imaging optical system and sagittal halo.
At x ≧ 0 and y ≧ 0,

When x ≧ 0 and y <0,

In equation (8), a, b, c, d, e, σ, and β are coefficients. As in the case of the equation (7), the equation (8) also needs to satisfy the relationship of ^{b 2-ac <0 with respect to the coefficients a, b, and c.}

10 (A) to 10 (F) show an example of a distribution shape that can be expressed by a function based on the equation (8). FIG. 10A shows a rotationally symmetric distribution in XY coordinates, and can be expressed by any of the functions of equations (5) to (8). If the imaging optical system is a co-axis system and the image point is on the optical axis, the PSF is also rotationally symmetric. Therefore, any function of equations (5) to (8) can be used in FIG. 10 (A). The distribution shape can be expressed.

10 (B) and 10 (C) show an elliptical distribution (hereinafter referred to as an elliptical distribution) in which the major axis and the minor axis of the ellipse overlap on the X-axis and the Y-axis. These elliptical distributions cannot be expressed by the function of equation (5), and the approximation accuracy is improved by using any of the functions of equations (6) to (8). FIG. 10 (D) shows an elliptical distribution when the major axis and the minor axis of the ellipse do not overlap the X-axis and the Y-axis, and the functions of equations (5) and (6) express this elliptical distribution. You can't. This elliptical distribution can be approximated with good accuracy by using any of the functions of Eqs. (7) and (8).

FIGS. 10 (E) and 10 (F) show a distribution shape that is symmetric with respect to the Y axis (that is, in the X direction) and asymmetric with respect to the X axis (that is, in the Y direction). The distribution shape of FIG. 10 (E) corresponds to a distribution in which the portion of the elliptical distribution shown in FIG. 10 (D) on the + X side of the Y axis is mirrored on the −X side with respect to the Y axis. Further, in the distribution shape of FIG. 10 (F), the portion of the elliptical distribution in which the long axis overlaps the Y axis and the short axis does not overlap the X axis is mirrored with respect to the Y axis and is more than the X axis. It corresponds to a distribution in which the lower part has a concentric semicircular shape. The functions of equations (5) to (7) cannot express a line-symmetrical distribution shape as shown in FIGS. 10 (E) and 10 (F). On the other hand, the function of the equation (8) used in this embodiment is a function capable of expressing a rotationally asymmetric distribution, and the distribution shapes shown in FIGS. 10 (E) and 10 (F) can be accurately approximated.

As described above, the PSF at the image point on the optical axis of the imaging optical system has a rotationally symmetric distribution shape, but the PSF rotates at an image point other than the optical axis in the plane (image plane) orthogonal to the optical axis. It does not always have a symmetrical distribution shape. However, when the imaging optical system is a co-axis optical system, even if the image point is not on the optical axis, the straight line connecting the image point and the optical axis extends in the image plane (meridional direction). In the orthogonal direction (sagittal direction), the PSF on the image point has a symmetrical distribution shape. As described above, the PSF of the imaging optical system does not always have a rotationally symmetric distribution shape, but has symmetry in the sagittal direction. Therefore, the x direction (X direction in FIGS. 10 (E) and 10 (F)) of the equation (8) is the sagittal direction, and the y direction (Y direction in FIGS. 10 (E) and (F)) is the meridional direction. It is possible to deal with complicated asymmetric aberrations by matching with.

Next, each coefficient in the equation (8) will be described in detail.

Of the coefficients of the equation (8), the coefficients a, b, and c are coefficients for generating an elliptical distribution in which the major axis and the minor axis do not overlap on the X axis and the Y axis as shown in FIG. 10 (D). .. Then, by controlling these coefficients a, b, and c, the asymmetry of the elliptical distribution in the X direction and the Y direction can be controlled. Further, as shown in FIGS. 10 (E) and 10 (F), like a sagittal halo in which an elliptical distribution in which at least one of a long axis and a short axis is set in addition to the X axis and the Y axis is symmetric only with respect to the Y axis. Aberrations that are difficult to express with other functions can also be expressed.

The coefficient d is a coefficient for asymmetricalizing the elliptical distribution in the Y direction (specific direction), and by controlling this coefficient d, it is possible to deal with an aberration having an asymmetrical shape in the meridional direction. For example, for coma aberration, the approximation accuracy can be further improved by controlling this coefficient d.

The coefficients e, σ, and β are coefficients for controlling the spread of the elliptical distribution. The approximation accuracy can be improved by increasing the coefficient σ when the spread of the elliptical distribution to be approximated is large, and increasing the coefficient β when the shape of the elliptical distribution to be approximated changes rapidly near the peak. The coefficient e is a coefficient for limiting the spread of the elliptical distribution. When the coefficient e = 0, the elliptical distribution gradually approaches P (x, y) = 0 on the peripheral side from the equation (8). Therefore, when the spread of the elliptical distribution is small, the approximation accuracy can be improved by setting the coefficient e to e> 0.

The elliptical distribution needs to be P (x, y) ≥ 0 in order to approximate the PSF of the imaging optical system. Therefore, when e> 0, P (x, y) <0 in the peripheral portion, but in that case, it may be clipped and P (x, y) = 0.

Next, the method of calculating the coefficient data will be described with reference to the flowchart of FIG.

In FIG. 11, S indicates a step. This also applies to other flowcharts described later. In each embodiment, the design value of the imaging optical system 101 is used to calculate the coefficient used for generating the unsharp mask.

First, in step S1, the information required for calculating the coefficient is acquired. Specifically, the imaging conditions at the time of imaging for generating the captured image for which the coefficient is calculated and the target value of the coefficient in the approximation of PSF are acquired.

Next, in step S2, the PSF of the imaging optical system 101 (hereinafter referred to as the design PSF) is calculated from the data of the design values of the imaging optical system 101 corresponding to the imaging conditions acquired in step S1. FIG. 12 shows a cross section of the design PSF calculated in step S2 and the approximate PSF generated in step S4 described later. As shown in FIG. 12, the design PSF calculated in step S2 is discretized with the number of divisions (number of taps) being N and the interval Do. The size (kernel size) of the design PSF can be expressed by the product of the interval Do and the number of taps N, as is clear from FIG. That is, in the discretized design PSF, if any two of the interval Do, the number of taps N, and the kernel size are known, the remaining one is also uniquely determined. For example, if the interval Do = 2.0 μm and the number of taps N = 11, the kernel size is Do × (N-1) = 20 μm. Further, Do × N may be called a kernel size, and in this case, it is 22 μm. Further, since the interval Do is the pitch at the time of fitting, it is preferable that the interval Do is smaller than the pixel pitch of the actual image sensor. By fitting with a small pixel pitch in advance, it is possible to support image sensors with various pixel pitches.

Although the data of the design value of the imaging optical system 101 is used for the fitting, the data obtained by estimating the PSF of the imaging optical system 101 from the captured image by imaging a chart or the like may be used.

Next, in step S3, the initial values of the coefficients a, b, c, d, e, σ, and β used when approximating the PSF are set. Since each coefficient is updated in the subsequent processing, a tentative value is set as an initial value in this step S3.

Next, in step S4, an approximate PSF is created by substituting the coefficient into the equation (8) and approximating the PSF. In this process, the approximate PSF is discretized in order to derive the optimum coefficient by fitting the design value. The number of divisions and the interval in discretization are adjusted to the design PSF calculated in step S2.

Next, in step S5, the deviation between the design PSF calculated in step S2 and the approximate PSF created in step S4 is evaluated. As an index for evaluating the deviation between the design PSF and the approximate PSF, for example, the root mean square of the difference between the design PSF and the approximate PSF is calculated and used as the evaluation value E. The smaller the evaluation value E, the closer the approximate PSF is to the design PSF.

Subsequently, in step S6, a determination is made using the evaluation value E calculated in step S5 and the target value acquired in step S1. The evaluation value E may include not only information on the deviation between the design PSF and the approximate PSF, but also information on the coefficients of the approximate PSF. In this embodiment, the equation (8) is used as the function (model) for approximating the PSF, but as described above, the coefficients a, b, and c must satisfy the relationship of ^{b 2-ac <0.} Therefore, if the coefficients a, b, and c do not satisfy this relationship and b ^2- ac ≧ 0, the desired result cannot be obtained. Therefore, it is more efficient to weight the evaluation value E so as to be large. Fitting can be performed. If there are other restrictions on the range in which each coefficient can be taken, the efficiency and accuracy of fitting can be improved by changing the evaluation value E in the same manner.

In this step, the evaluation value E calculated in this way is compared with the preset target value, and if the evaluation value E is equal to or less than the target value, fitting, that is, generation of the approximate PSF is completed, and the approximate PSF is obtained. Output the obtained coefficient data. If the evaluation value E exceeds the target value, the fitting has not been sufficiently completed, and the process proceeds to step S7.

In step S7, the coefficient is changed (updated). At this time, the number of coefficients to be updated may be only one or a plurality. After updating the coefficient, the process returns to step S4, the approximate PSF is calculated again, and the evaluation value E is calculated in step S5. Then, the processes of steps S4 to S7 are repeated until the evaluation value E converges to the target value or less in step S6.

If the evaluation value E after the update is not smaller than the evaluation value E before the update of the coefficient, the coefficient before the update may be returned and the processing from step S4 may be repeated, or in order to escape from the local solution. You may return to step S3 and set the initial value again.

By the above coefficient calculation process, PSFs for various imaging optical systems (when interchangeable) 101 and for various imaging conditions can be previously coefficient-coded and the data can be stored in the storage unit 120. By calculating the coefficient of the approximate PSF in advance in this way, the approximate PSF according to the imaging optical system 101 and the imaging conditions can be reconstructed simply by acquiring the data of the calculated coefficient at the stage of sharpening processing ( Can be reproduced).

Hereinafter, specific examples will be described.

First, the image pickup apparatus according to the first embodiment of the present invention will be described with reference to FIG.

FIG. 2 is a block diagram of the imaging device (interchangeable lens 118, conversion lens 119, camera body 120) in this embodiment. In the camera body 120, a program for sharpening an input image (captured image) (image processing method) is installed in a storage unit 109 such as a ROM (memory) or a hard disk drive, and the sharpening processing is performed by the image processing unit. It is executed by 105 (image processing device). Further, a storage unit may be provided inside the image processing unit 105 instead of the storage unit 109, and the program of the image processing method of the present embodiment may be installed in the storage unit. Further, a circuit corresponding to the program may be designed and the sharpening process may be executed by operating the circuit.

The interchangeable lens 118 includes an imaging optical system 101, an optical system control unit 107, a storage unit 112, a lens CPU 113, and a contact unit 114. Further, the imaging optical system 101 includes a diaphragm 101a and a focus lens 101b. In this embodiment, the interchangeable lens 118 is configured to be interchangeable with respect to the camera body 120 or the conversion lens 119, but the present invention is not limited to this, and is also applicable to an imaging device integrally configured with the camera body. It is possible. The storage unit 112 is a rewritable non-volatile memory. The data stored in the storage unit 112 is mainly information indicating optical characteristics peculiar to the interchangeable lens 118, and the camera body 120 acquires this information from the interchangeable lens 118 and further captures an image based on this information. Is being corrected.

Information about the PSF such as coefficient data and adjustment coefficient m required for reconstructing the PSF of the imaging optical system 101 is stored in the storage unit 112. This information is transmitted by communication from the interchangeable lens 118 to the camera body 120 via the contact units 114, 115, 116, 117. The camera body 120 generates a filter from the information about the PSF transmitted from the interchangeable lens 118, executes the correction process, and generates a sharpened image. Further, the information about the PSF may be stored in the storage unit 109 on the camera body side. When the interchangeable lens is attached, the coefficient data or the like stored in the storage unit 112 at the time of initial communication can be transferred to the storage unit 109 via the lens CPU 113 and the camera CPU and stored. In this embodiment, the information about the PSF such as the coefficient data and the adjustment coefficient stored in the storage unit 112 or the storage unit 109 is the information corresponding to the imaging optical system 101 of the interchangeable lens 118. By attaching the conversion lens 119, the optical performance of the entire optical system is changed by the accessory optical system 102, but the PSF after the performance change by the accessory optical system 102 is reconstructed from the coefficient data corresponding to the imaging optical system 101. The details of the PSF reconstruction process will be described later.

The lens CPU 113 includes a communication circuit (communication means) for communicating between the interchangeable lens 118 and the camera body 120, reset exception handling, A / D, timer, input / output port, built-in ROM, built-in RAM, and the like. Has a function. The communication circuit communicates between the interchangeable lens 118 and the camera body 120 by a communication method including control information according to the shooting mode (moving image shooting mode, still image shooting mode). The optical system control unit 107 is a lens control means for controlling each component inside the interchangeable lens 1, and is a lens, an aperture, or the like using control information obtained via a communication circuit based on an instruction from the lens CPU 113. Drive control of the optical element of. The contact unit 8 is a connection means that includes a plurality of metal contacts for communicating between the interchangeable lens 118 and the camera body 120, and electrically connects the lens CPU 113 and the camera CPU 110. As shown in FIG. 2, even when the conversion lens 119 is arranged between the interchangeable lens 118 and the camera body 120, the lens CPU 113 and the camera CPU 110 can be electrically connected as in the case where the conversion lens 119 is not provided. Is.

The conversion lens 119 includes an accessory optical system 102 and contact units 115 and 116. By arranging the conversion lens 119 between the interchangeable lens 118 and the camera body 120, the focal length of the imaging optical system 101 can be changed by the magnification of the accessory optical system 102. For example, if the accessory optical system 102 is a double teleconverter, the focal length of the imaging optical system 101 is doubled. In FIG. 2, one conversion lens 119 is arranged between the interchangeable lens 118 and the camera body 120, but a plurality of conversion lenses 119 can be arranged and can be connected and arranged. The contact units 115 and 116 include a plurality of metal contacts for communicating between the interchangeable lens 118 and the camera body 120, and are connection means for electrically connecting the lens CPU 113 and the camera CPU 110. A storage unit may be arranged in the conversion lens 119 to hold the information of the accessory optical system 102, and the information may be transmitted to the lens CPU 113 and the camera CPU 110. In FIG. 2, the conversion lens 101c is arranged on the image sensor side of the image pickup optical system 101, but a conversion lens of a type mounted on the object side (subject side) may be used.

The camera body 120 includes an optical low-pass filter 103, an image sensor 104, an image processing unit 105, a display unit 106, a state detection unit 108, a storage unit 109, an image recording medium 110, a camera CPU 111, and a contact unit 117.

The image pickup device 104 is a two-dimensional image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The image pickup element 104 generates a photographed image by photoelectrically converting a subject image (optical image, imaging light) obtained via the image pickup optical system 101, the accessory optical system 102, and the optical low-pass filter 103. The subject image is photoelectrically converted by the image pickup element 104 and converted into an analog signal (electrical signal), and this analog signal is converted into a digital signal by an A / D converter (not shown), and this digital signal is converted into a digital signal by the image processing unit 105. Is entered in.

The image processing unit 105 is an image processing means that performs predetermined processing on the digital signal and also performs predetermined unsharp mask processing. In this embodiment, the image processing unit of the camera body performs the sharpening process, but a personal computer (PC) or a dedicated device may perform the sharpening process as the image processing device.

The image processing unit 105 acquires the imaging conditions (imaging condition information) of the imaging device from the state detection unit 108. The imaging condition information is information related to the aperture, the shooting distance, the focal length of the zoom lens, and the like. The state detection unit 108 can acquire imaging condition information directly from the camera CPU 111, but is not limited to this. For example, the imaging condition information regarding the imaging optical system 101 can be acquired from the optical system control unit 107 via the lens CPU.

As shown in FIG. 9, the image processing unit 105 has an acquisition unit 1051, a reconstruction processing unit 1052, and a sharpening processing unit 1053, and performs image sharpening processing on the input image.

The output image processed by the image processing unit 105 is stored in the storage unit 109 in a predetermined format. The storage unit 109 also functions as a storage means for storing the relationship between the imaging conditions of the imaging optical system 101 and the PSF of the imaging optical system. When an image processing device that executes unsharp mask processing is provided separately from the image processing unit 105, the camera CPU 111 may store aberration information in association with the captured image.

The display unit 106 can display an image obtained by performing a predetermined process for display after the sharpening process. An image obtained by performing simple processing for high-speed display may be displayed on the display unit 106.

The above series of processes is controlled by the camera CPU 111.

The imaging optical system 101 may be provided with an optical element such as a low-pass filter or an infrared cut filter. When an optical element that affects PSF, such as a low-pass filter, is used, more accurate sharpening processing can be performed by considering the influence of this optical element at the time of creating the unsharp mask. Even if an infrared cut filter is provided, it affects the PSF of each RGB channel (RGB color component), which is the integral value of the PSF of the spectral wavelength, especially the PSF of the R channel. Therefore, at the time of creating the unsharp mask, It is preferable to consider the effect.

Next, the image processing method of this embodiment will be described with reference to FIG.

FIG. 1 is a flowchart showing an image processing method of this embodiment.

The flowchart of FIG. 1 can be embodied as a program (image processing program) for causing a computer to execute the function of each step. This also applies to the flowcharts of other examples. Each step of FIG. 1 is executed by the image processing unit 105 based on the instruction of the camera CPU 111.

First, in step S11, the image processing unit 105 (acquisition unit 1051) acquires a captured image generated via the optical system as an input image. The color component data as the correction target used as the input image is, for example, the image data of the G channel after demoizing. However, the image data of the R channel or the B channel, the image data of all the RGB channels, or the image data before demoizing may be used.
FIG. 8 is a schematic diagram of a Bayer array, which is a discrete regular array. For example, the data of each RGB channel may be simply extracted as it is and used as an input image for each color, or only a specific channel may be used as an input image. Further, as shown in FIG. 8, the G channel may be divided into two, G1 and G2, and treated as four channels. By dividing the G channel into two, the image data extracted from each of R, G1, G2, and B has the same resolution, which facilitates processing and data processing.

Subsequently, in step S12, the image processing unit 105 (acquisition unit 1051) acquires imaging condition information including information regarding the accessory optical system 102 from the interchangeable lens 118. Table 1 shows the correspondence between the value of the lens ID (identification ID) and the state of the lens. Table 1 shows the values of three types of lens IDs depending on the mounting condition of the conversion lens, and each value is an example. In this embodiment, the lens ID is acquired as information of the accessory optical system 102. The lens ID is stored in the storage unit 112, and is transmitted from the lens CPU 113 to the camera CPU 111 when the lens ID is acquired on the camera side. The lens ID stored in the storage unit 112 is updated by mounting the conversion lens. When the conversion lens is attached, the detection means (not shown) detects it, the lens CPU recognizes it, and the lens ID in the storage unit 112 is updated. The state of the conversion lens does not change during shooting, and the lens ID obtained during the initial communication can always correctly represent the information of the conversion lens. Further, the correspondence table in Table 1 is stored in the storage unit 109, and the camera CPU 111 can recognize the state of the lens from the acquired value of the lens ID. In this embodiment, the lens ID is acquired from the interchangeable lens 118, but the information acquired in this step only needs to be able to specify the type and magnification of the conversion lens, and information such as the presence / absence of the conversion lens, the magnification, and the focal length. May be obtained.

In step S13, the image processing unit 105 (acquisition unit 1051) has the coefficients a, b, c of the function (predetermined function) of the function (predetermined function) of the formula (8) used for reconstructing the PSF of the imaging optical system 101 according to the imaging conditions. Acquire the data of d, e, σ, and β. For example, if the conversion lens 119 is a 1.4x teleconverter, the lens ID acquired in step S12 is 02. In this case, the coefficient data acquired in this step is not the information of the PSF corresponding to the optical system including the teleconverter, but the coefficient data corresponding to the PSF of the imaging optical system 101. That is, regardless of which number in Table 1 the lens ID acquired in step S12 is, the coefficient data acquired from the storage unit 112 or the storage unit 109 is the same.

It is not always necessary to acquire the data of all these coefficients in order to generate the approximate PSF corresponding to a certain image point. For example, in the case of PSF on the optical axis, since the shape is rotationally symmetric as described above, a = c, b = 0, and d = 1.

Further, since the coefficient β is a term of power, if the coefficient β can be changed according to the PSF, the processing load increases. Therefore, β = 1 may be fixed. When the coefficient β is fixed in this way, the shape that can be expressed is reduced as compared with the case where the coefficient β is provided, but the amount of coefficient data held in the storage unit 112 or the storage unit 109 can be reduced and the processing load can be reduced. ..

Further, in order to improve the approximation accuracy, a coefficient may be added. For example, for a PSF with a small distribution spread and a high peak, it is difficult to fit with high accuracy using a continuous function, so the peak of the PSF or a value near the peak may be set directly as a coefficient. .. In this way, by directly setting the region where the distribution changes rapidly as a coefficient, the region reproduced by the function can be set as another region, and the approximation accuracy can be improved. Further, in step S13, the image processing unit 105 (acquisition unit 1051) may acquire not only the coefficient data but also the adjustment coefficient m used when executing the sharpening process.

Next, the unsharp mask USM will be described with reference to FIG. 7.

FIG. 7A is a schematic view of the unsharp mask, and FIG. 7B is a schematic cross-sectional view of the unsharp mask.

The number of taps of the unsharp mask USM is determined according to the aberration characteristics of the imaging optical system and the required sharpening accuracy. The unsharp mask USM shown in FIG. 7A is, for example, a two-dimensional mask with 11 × 11 taps. Further, although the values (coefficients) in each tap are omitted in FIG. 7A, one cross section of the unsharp mask USM is shown in FIG. 7B. In FIG. 7B, the solid line is the cross section of the unsharp mask USM, the horizontal axis is the tap value, and the vertical axis is the tap value. Ideally, the distribution of the value (coefficient value) of each tap of the unsharp mask USM is the distribution of the signal value (PSF of the imaging optical system) spread by the aberration.

In this embodiment, the image processing unit 105 generates an approximate PSF using the coefficient data, and generates an unsharp mask USM corresponding to the approximate PSF. Therefore, as compared with the case where the data corresponding to the PSF of the imaging optical system is directly held, the amount of data to be held while maintaining the correction accuracy can be significantly reduced. For example, as shown in FIG. 7, an 11 × 11 tap unsharp mask USM needs to have data of 121 tap values. When the data for RGB is held separately, the data is tripled, so that the data of 363 tap values must be held. On the other hand, when the coefficients are held, the coefficients in the equation (8) are 7, and even when the coefficients for RGB are held separately, the number is 21. In this way, the amount of retained data can be reduced by retaining the coefficient data.

Subsequently, in step S14, the image processing unit 105 (reconstruction processing unit 2052) reconstructs the PSF using the shooting information acquired in step S12 and the coefficient data acquired in step S13 (reconstruction processing). .. The PSF is reconstructed based on the coefficient data and the equation (8) which is a function used when calculating the coefficient data, and in this embodiment, the reconstructed PSF is used with an unsharp mask. FIG. 14 is a cross-sectional view of the reconstructed point image intensity distribution function PSF. When trying to reproduce the portion of the figure region A as an unsharp mask, fitting is performed in a slightly wider region B and the coefficients are May be generated. As a result, when the number of taps or the pitch is changed later by using an interchangeable lens or the like, it is possible to change the direction to increase the area.

In this embodiment, the sampling pitch when discretizing the continuous function as in the equation (8) is determined by the lens ID acquired in step S12.

When the lens ID is 01, the image pickup optical system 101 is the only state, that is, the image pickup device is composed of the interchangeable lens 118 and the camera body 120. The coefficient data acquired in step S13 is information about the imaging optical system 101 and corresponds to the state where the lens ID is 01. Therefore, in step S12, if the acquired lens ID is 01, the reconstruction can be performed as it is. Basically, the size of one tap of the generated unsharp mask needs to match the size of one pixel of the image sensor when the image is acquired, so if it is uniquely determined, Generate to match the pixel size of the sensor. When the unsharp mask is reconstructed from the coefficient as in this process, the correction accuracy at the time of sharpening is improved by increasing the coefficient, so it is possible to generate and reconstruct the coefficient according to the required accuracy. preferable. Next, when the lens ID is 02, the image pickup apparatus is composed of an interchangeable lens 118, a conversion lens 119 (teleconverter 1.4 times), and a camera body 120.

Even when the lens ID is 02, the coefficient data acquired in step S13 is the information corresponding to the imaging optical system 101, so that the change in the optical performance due to the accessory optical system 102 is not taken into consideration. Therefore, when the lens ID is 02, if the reconstruction is performed as it is as in the case where the lens ID is 01, the PSF formed on the image sensor 104 will be an approximate PSF that is significantly different from the PSF formed on the image sensor 104. Therefore, even when the conversion lens 119 is attached, in order to reproduce the PSF formed on the image sensor 104 with high accuracy, adjustment is performed according to the lens ID based on the coefficient data acquired in step S13. .. If the accessory optical system is aberration-free, the aberration generated in the image pickup optical system of the PSF formed on the image pickup element 104 is magnified by the magnification of the conversion lens 119.

If the lens ID is 02, Table 1 identifies that the conversion lens 119 is a 1.4x teleconverter. By identifying the conversion lens 119, it is possible to uniquely obtain information on the magnification of the accessory optical system.

Although Table 1 shows the lens ID and the state of the lens, similarly, the correspondence relationship between the lens ID and the lens ID may be stored in the storage unit 112 or the storage unit 109 for the magnification information. Next, an unsharp mask is generated using the magnification information of the accessory optical system obtained and the coefficient data acquired in step S13. When the lens ID is 01, the reconstruction is performed so as to match the pixel size, but when the lens ID is 02, discretization is performed while expanding the PSF by the magnification of the accessory optical system 102. That is, instead of discretizing at the pixel size interval, the pixel size is discretized at an interval reduced by 1 / 1.4 times. By doing so, the reconstructed PSF is magnified by the magnification of the accessory optical system 102, and an unsharp mask can be generated in consideration of the change in the optical characteristics due to the conversion lens 119.

Further, in the case of a rear converter type conversion lens arranged between the interchangeable lens 118 and the camera body 120 as shown in FIG. 2, the F value of the imaging optical system 101 also changes. A teleconverter with a magnification of 1.4 times changes to about 1 step, and a teleconverter with a magnification of 2.0 times changes to darker by 2 steps. Therefore, the coefficient data corresponding to the F value after the change by the conversion lens 119 in step S13 may be acquired. When the imaging conditions of the imaging optical system 101 are acquired in step S12, if the obtained F value is F2.8, it becomes F4.0 when the aperture is stopped down by one step. At this time, if the coefficient data acquired in step S13 is not F2.8 but data corresponding to the imaging optical system 101 of F4.0, it also corresponds to the change of the F value by the rear converter as described above. Can be done. When the lens ID acquired in step S12 is 03, the reconstruction can be performed in almost the same flow as in the case where the lens ID is 02, so detailed description thereof will be omitted.

FIG. 13A shows the relationship between the position of the generated unsharp mask and the input image. The white circles indicate the positions of the unsharp masks to be created, and as shown in FIG. 13A, the input image is divided to generate 81 unsharp masks. Then, by performing linear interpolation or the like on these unsharp masks, it is possible to generate an unsharp mask at an arbitrary position in the input image, and it is possible to respond to changes in the image height of the PSF. Here, the number of divisions is 9 × 9 in FIG. 13 (A), but it may be reduced for weight reduction or increased with more emphasis on accuracy. Further, the PSF may not be directly acquired for each point of the white circle in FIG. 13 (A), but may be generated by interpolation.

FIG. 13B shows an example thereof, and shows a case where an unsharp mask at each position is generated by interpolation. The black dots in FIG. 13B indicate the positions of the unsharp masks created in step S14. In general, the PSF of the imaging optical system is rotationally symmetric with respect to the optical axis, so that the unsharp mask is also rotationally symmetric. Utilizing this feature, in the example of FIG. 13B, 10 unsharp masks are generated downward from the center of the image, and these are rotated with respect to the center of the image to form each white circle. An unsharp mask at the corresponding position is generated by interpolation. This eliminates the need to create unsharp masks at each position of the input image one by one, so that the processing load can be reduced. That is all for the correspondence to the image height.

Subsequently, in step S15, the image processing unit 105 (sharpening processing unit 1053) executes the sharpening process of the captured image using the unsharp mask USM generated in step S14. In this embodiment, since the PSF of the imaging optical system is used for the unsharp mask USM, even if the image is deteriorated by the asymmetrical PSF of the imaging optical system as seen in the peripheral portion of the input image, the input image is displayed. It can be corrected and sharpened with high accuracy.

The image g (x, y) after the sharpening process can be expressed as the following equations (9), (10), and (11) from the equations (1) and (3).

g (x, y) = f (x, y) + m × {f (x, y) -f (x, y) * USM (x, y)} ... (9)
g (x, y) = f (x, y) + m × f (x, y) * {δ (x, y) -USM (x, y)} ... (10)
g (x, y) = f (x, y) * {δ (x, y) + m × (δ (x, y) -USM (x, y))} ... (11)
Here, for convenience, the part of the curly braces {} in the equation (11) is called a sharpening filter. The sharpening filter can be generated using the unsharp mask USM and the adjustment factor m. The value of the adjustment coefficient m is determined in consideration of image noise, overcorrection of sharpening, and undercorrection. In step S15, the sharpening processing unit 1054 executes sharpening processing on the input image based on the equation (11) using the unsharp mask USM generated in step S14.

In this embodiment, the unsharp mask USM is discretely held with respect to the input image as shown in FIG. 13 (A). Therefore, in order to perform the sharpening process at a position other than the white circle in FIG. 13A, a corresponding unsharp mask USM or a sharpening filter is required. In this embodiment, the sharpening process can be performed at an arbitrary position by linearly interpolating the discretely generated unsharp USM. Specifically, an unsharp mask USM corresponding to a certain position is generated by linearly interpolating an unsharp mask corresponding to four white circles near the position, and sharpening processing is performed based on the equation (11). .. As a result, sharpening processing can be performed at an arbitrary position in the image, and the sharpening effect in the image also changes continuously, so that a more natural sharpening image can be generated. Further, linear interpolation may be performed by a sharpening filter instead of the unsharp mask USM.

In this embodiment, the sharpening treatment based on the formula (11) has been described, but the sharpening treatment may be performed using the formula (9) or the formula (10), and the same effect can be obtained in either case. Obtainable. Equation (9) or Equation (10) is expressed by adding a correction component to the input image, but this is the case where the adjustment coefficient m is positive, and when the adjustment coefficient m is negative, it is subtraction. .. As described above, since the difference in the sign of the adjustment coefficient m means essentially the same, it does not matter which operation is performed as long as it is changed according to the sign of the adjustment coefficient m.

In this embodiment, the equation (8) corresponding to the PSF of the imaging optical system and the approximate PSF generated from the coefficients are used for the unsharp mask USM. Therefore, it is possible to accurately sharpen the deterioration of the imaging optical system due to the asymmetric PSF, which is seen in the peripheral portion of the input image.

Further, in this embodiment, only the coefficient data corresponding to the imaging optical system 101 is retained, and the approximate PSF including the change in the optical performance due to the accessory optical system is reconstructed from the coefficient data and the information of the accessory optical system 102. As shown in Table 1, when the imaging device takes three states, if it is simply intended to prepare coefficient data corresponding to each state, it is necessary to hold about three times as many data. In this embodiment, since the approximate PSF is reconstructed from the magnification related to the accessory optical system or the lens ID capable of specifying the information, the coefficient data to be stored is 1/3 of the data amount as compared with the case where they are separately prepared. Can be reduced.

By using the image processing method of this embodiment, it is possible to execute high-precision sharpening processing while reducing the amount of information required for sharpening processing.

The image pickup apparatus according to the second embodiment of the present invention will be described.

FIG. 15 is a block diagram of the imaging device according to the second embodiment.

The image pickup apparatus of this embodiment includes an interchangeable lens 218, a conversion lens 219, and a camera body 220. Only the interchangeable lens 218 is different from the first embodiment, and the other configurations are the same as those of the first embodiment.

The interchangeable lens 218 includes an imaging optical system 201, an optical system control unit 207, a storage unit 212, a lens CPU 213, a contact unit 214, and a built-in extender 201c (teleconverter). Further, the imaging optical system 201 is configured to include an aperture 201a and a focus lens 201b. The built-in extender 201c is inserted into and removed from the optical path by the user mechanically operating an operation unit such as a lever provided on the lens barrel of the interchangeable lens 218. By inserting the built-in extender 201c, it is possible to change the focal length of the imaging optical system 201 to 1.4 times, 2.0 times, or the like. When the built-in extender 201c is inserted, it is detected by a detection unit (not shown) or an optical system control unit 207, and the lens ID stored in the storage unit 212 is updated via the lens CPU 213. Since other configurations are the same as those in the first embodiment, the description thereof will be omitted.

Since the flowchart of the first embodiment shown in FIG. 1 and steps S12 and S14 are different from each other, the image processing method of the present embodiment will be described in order.

In step S12, the image processing unit 205 acquires imaging condition information including information regarding the accessory optical system 202 and the built-in extender 201c from the interchangeable lens 218. Table 2 shows the correspondence between the value of the lens ID (identification ID) and the state of the lens in this embodiment. Table 2 shows the values of six types of lens IDs depending on the mounting status of the conversion lens, and each value is an example. The lens ID is stored in the storage unit 212, and is transmitted from the lens CPU 213 to the camera CPU 211 when acquired by the camera side. The correspondence table in Table 2 is stored in the storage unit 209, and the camera CPU 211 can recognize the state of the lens from the acquired value of the lens ID.

In step S14, the image processing unit 205 reconstructs the PSF using the shooting information acquired in step S12 and the coefficient data acquired in step S13. The PSF is reconstructed based on the coefficient data and the equation (8) which is a function used when calculating the coefficient data, and in this embodiment, the reconstructed PSF is used with an unsharp mask. In this embodiment, the sampling pitch when discretizing the continuous function as in the equation (8) is determined by the lens ID acquired in step S12. When the lens ID is other than 01, discretization is performed while expanding the PSF by the magnification of the built-in extender 201c and the accessory optical system 202. For example, if the lens ID is 06 and the magnification of the built-in extender is 2.0 times, the focal length of the imaging optical system 201 is changed to 4.0 times. Therefore, the pixel pitch at the time of discretization may be discretized at intervals obtained by reducing the pixel size by 1/4 times. Further, as described in the first embodiment, when considering the change in the F value of the imaging optical system 201, the F value becomes darker by the product of the magnification of the built-in extender 201c and the accessory optical system 202. The coefficient data may be acquired in consideration of.

In this embodiment, since it is necessary to generate an approximate PSF corresponding to the imaging conditions for the interchangeable lens having the built-in extender, there are many variations in the focal length as shown in Table 2. In particular, in such a case, it is not necessary to generate coefficient data corresponding to each lens state and store it in the storage unit. Therefore, the amount of data can be significantly reduced by using the image processing method of this embodiment. be able to.

The image pickup apparatus according to the third embodiment of the present invention will be described.

FIG. 16 is a block diagram of the imaging device according to the third embodiment.

The image pickup apparatus of this embodiment includes an interchangeable lens 318, a conversion lens 319, and a camera body 320. The conversion lens 319 of the third embodiment is mounted on the object side (subject side) of the interchangeable lens 318, and has an accessory optical system 302 that changes optical characteristics such as a wide converter, a teleconverter, and a close-up lens. When the conversion lens 319 is attached, information is transmitted to the lens CPU 313 via the contact units 314 and 315, and the lens CPU 313 updates the lens ID stored in the storage unit 312. Further, when the conversion lens 319 is attached, the optical characteristics change, so that the aberration generated by the imaging optical system 301 alone changes. In particular, when the imaging optical system 301 is a zoom lens, it is difficult to maintain high optical performance from the center of the screen to the periphery in a wide zoom range. For example, when a wide-angle converter is attached to a zoom lens, spherical aberration is generated from the center of the screen in a zoom range other than the wide-angle end in order to give priority to the performance at the wide-angle end, which deteriorates the optical performance.

Information about the accessory optical system such as the lens ID may be updated by a user input operation. Since other configurations are common to the image pickup apparatus of the first embodiment, the description thereof will be omitted.

Since the flowchart of the first embodiment shown in FIG. 1 and steps S12 and S14 are different from each other, the image processing method of the present embodiment will be described in order.

In step S12, the image processing unit 305 acquires imaging condition information including information regarding the accessory optical system 302 from the interchangeable lens 318. Table 3 shows the correspondence between the value of the lens ID (identification ID) and the state of the lens in this embodiment. Table 3 shows the values of five types of lens IDs depending on the mounting status of the conversion lens, and each value is an example. The lens ID is stored in the storage unit 312, and is transmitted from the lens CPU 313 to the camera CPU 311 when the lens ID is acquired on the camera side. The correspondence table in Table 3 is stored in the storage unit 309, and the camera CPU 311 can recognize the state of the lens from the acquired value of the lens ID.

In step S14, the image processing unit 305 reconstructs the PSF using the shooting information acquired in step S12 and the coefficient data acquired in step S13. The PSF is reconstructed based on the coefficient data and the equation (8) which is a function used when calculating the coefficient data, and in this embodiment, the reconstructed PSF is used with an unsharp mask. In this embodiment, the sampling pitch when discretizing the continuous function as in the equation (8) is determined by the lens ID acquired in step S12. If the lens ID is other than 01, the optical performance of the imaging optical system 301 deteriorates due to the accessory optical system 302. Therefore, the PSF when the conversion lens 319 is attached is distributed more than the PSF of the imaging optical system 301 alone. Spreads. Therefore, in this embodiment, the PSF having deteriorated optical performance is reconstructed by performing discretization at intervals smaller than the pixel size of the image sensor. In this way, since the discretization is performed at intervals smaller than the pixel size, the correction accuracy is improved as compared with the discretization by the pixel size as it is, and the coefficient data to be stored is only the data corresponding to the imaging optical system 301 alone. , The amount of data can be reduced.

Next, the image pickup apparatus of Example 4 will be described.

The image pickup apparatus of this embodiment has the same configuration as the image pickup apparatus of Example 1.

FIG. 17 is a flowchart showing an image processing method of this embodiment.

Regarding the sharpening process in this embodiment, the description thereof will be omitted because step S41, step S42, and step S45 perform the same process as step S11, step S12, and step S15 shown in FIG.

In step S43, the image processing unit 105 acquires PSF data corresponding to the imaging optical system 101. In this embodiment, the approximate PSF corresponding to each lens state is generated by re-discretizing the PSF data that has already been discretized, instead of reconstructing the coefficient data to generate the approximate PSF. Therefore, in this step, PSF data corresponding to the imaging optical system 101 stored in the storage unit 112 or the storage unit 109 is acquired.

Next, in step S44, the approximate PSF is reconstructed using the shooting information acquired in step S42 and the PSF data acquired in step S43. In FIG. 14, the area B is the range of the PSF data acquired in step S43, and the area A is the approximate PSF data after reconstruction. If the acquired lens ID is 01 in step S42, the conversion lens 119 is not attached, so the PSF data acquired in step S43 is used as it is as approximate PSF data. Next, when the lens ID is 02 or 03, adjustment processing is performed according to the magnification of the accessory optical system. When the lens ID is 02, the PSF data is upsampled 1.4 times, and when the lens ID is 03, the PSF data is upsampled twice. By upsampling, the sampling interval is 1 / 1.4 when the lens ID is 02 and halved when the lens ID is 03. In addition, when performing upsampling, interpolation processing is required, so existing interpolation processing such as the bilinear method or bicubic method is applied. Next, the data of a desired number of taps is extracted from the upsampled PSF data. For example, when the number of taps is not changed before and after the reconstruction, in FIG. 14, when the lens ID is 02, the area A is 1 / 1.4 with respect to the area B, and when the lens ID is 03, the area A is halved. The area and the number of taps do not necessarily have to match before and after the reconstruction, and may be changed as necessary. Further, although the case of performing upsampling has been described earlier, the area selection may be performed first. In this case, for example, if the lens ID is 03, half of the area B is set as the area A, and upsampling is performed on the area A. By doing so, the processing load can be reduced because only the necessary area is upsampled.

Although the examples of the present invention have been described above, the present invention can be modified and modified in various ways within the scope of the gist thereof.

105 image processing unit, 151 acquisition unit,
1152 Reconstruction processing unit, 1153 Sharpening processing unit

Claims (12)

It is generated based on the captured image generated by imaging through the optical system, the acquisition step of acquiring the information of the point image intensity distribution function corresponding to the imaging conditions of the optical system, and the information of the point image intensity distribution function. It has a processing step of performing unsharp mask processing on the captured image using the filter, and the optical system has an accessory optical system that changes the optical characteristics, and the accessory optical system can be attached and detached. An image processing method characterized in that the filter is generated from the information of the point image intensity distribution function when the accessory optical system is not attached. The image processing method according to claim 1, wherein the filter is a filter having two-dimensional data. The image processing method according to claim 1 or 2, wherein the filter has a rotationally asymmetric distribution. The first to third claims, wherein a filter is generated based on the point image intensity distribution function reconstructed using the coefficient data used for approximating the point image intensity distribution function in the processing step. The image processing method according to any one of the items. The image processing method according to any one of claims 1 to 4, wherein the information of the accessory optical system is a magnification of the accessory optical system. The image processing method according to any one of claims 1 to 4, wherein the information of the accessory optical system is an identification ID of the accessory optical system. The image processing method according to any one of claims 1 to 6, wherein the accessory optical system is a conversion lens that changes the focal length of the optical system. The image processing method according to any one of claims 1 to 7, wherein the filter is adjusted by an adjustment coefficient according to the position of the captured image. The image processing method according to any one of claims 1 to 9, wherein the shooting conditions include at least one of an image height, a focal length, an F value, and a shooting distance. An image processing device that corrects a captured image taken by an imaging device using a filter generated based on a point image intensity distribution function.
It has a sharpening means for sharpening the photographed image, and has
An image processing apparatus according to claim 1, wherein the image processing method according to any one of claims 1 to 9 is used. An image processing program that corrects a captured image captured by an imaging device using a filter generated based on a point image intensity distribution function.
It has a sharpening process for sharpening the photographed image.
An image processing program characterized by using the image processing method according to any one of claims 1 to 9. It has an image sensor that photoelectrically converts the subject image obtained via the image pickup optical system.
An image pickup device that corrects a captured image acquired from the image pickup device by using a filter generated based on a point image intensity distribution function.
It has an image processing unit that processes a captured image obtained from the image sensor.
An image pickup apparatus according to claim 1, wherein the image processing method according to any one of claims 1 to 9 is used.