JP2017118293A - Image processing apparatus, imaging apparatus, image processing method, image processing program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of effectively reducing color bleeding in a color image even when color bleeding property differs depending on a subject distance like a three-dimensional subject.SOLUTION: The image processing apparatus comprises: an acquisition part (104a) which acquires information relating to a defocus amount of an image formed via an imaging optical system; and a determination part (104b) which determines whether or not to correct the image so as to reduce color bleeding, by using the information relating to the defocus amount and optical information relating to the color bleeding of the imaging optical system.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that reduces color bleeding in a color image.

  Conventionally, a color that does not originally exist around a bright portion of a color image may occur as a color blur due to chromatic aberration of the imaging optical system. In visible light color imaging with an imaging device, color blur tends to occur in areas away from green, which is the central wavelength of the imaging optical system, and blue, red, or purple artifacts mixed with both are blurred. Or purple fringe.

  The chromatic aberration of the imaging optical system can be optically reduced to some extent by combining a plurality of lenses made of materials having different dispersions. However, in recent years, the downsizing of the image pickup apparatus has progressed, and it is required to increase the resolution of the sensor used for the image pickup element and the size of the image pickup optical system, and it is difficult to sufficiently suppress chromatic aberration by the configuration of the image pickup optical system. .

  Chromatic aberration is roughly classified into lateral chromatic aberration (magnification chromatic aberration) and longitudinal chromatic aberration (axial chromatic aberration). Lateral chromatic aberration is a phenomenon in which the image formation position shifts in the direction along the image plane depending on the wavelength. Longitudinal chromatic aberration is a phenomenon in which the imaging position shifts in the direction along the optical axis depending on the wavelength. Patent Document 1 discloses a method of correcting lateral chromatic aberration by performing geometric transformation processing that adds different distortion to each of the R (red), G (green), and B (blue) color planes. On the other hand, the longitudinal chromatic aberration is focused on the subject in the R (red) plane and the B (blue) plane, which are the ends of the visible light region, in an image focused on the G (green) plane that bears the center wavelength of the visible light region, for example. Does not match, resulting in a blurred image (out-of-focus image). For this reason, longitudinal chromatic aberration cannot be corrected using geometric transformation.

  In Patent Document 2, a region in which a G (green) plane is saturated is searched using a characteristic in which color blur is mainly generated in the vicinity of a whiteout (a saturated region of a preset signal), and a region around the region is searched. A method of calculating a correction amount by integrating the pixels is disclosed. Patent Document 3 discloses a method of determining a region where a color plane pixel value in a color image monotonously increases or decreases as a color blur generation region and removing the color blur.

US Pat. No. 6,724,702 JP 2007-133582 A JP 2009-268033 A

  However, in the method disclosed in Patent Document 2, color blur that causes discomfort to the observer also occurs in a region that is not overexposed, and thus correction of color blur is insufficient. Further, according to the method disclosed in Patent Document 3, it is possible to correct the color blur of the region that is not overexposed. However, since color blur occurs due to aberration of the imaging optical system, there are directions where color blur occurs and does not occur with respect to the subject in an imaging optical system having asymmetric aberrations such as coma. When the color of the subject is similar to the color blur, there is a possibility that the subject color is erroneously determined as a color blur even in a direction where the color blur does not occur, and the original color of the subject is removed.

  In a three-dimensional object, the color blur (color blur characteristic) varies depending on the shift amount (defocus amount) from the in-focus distance. This is because the imaging position for each wavelength differs according to the defocus amount. For this reason, the colors to be subjected to color blur correction are different with respect to the in-focus distance (in-focus subject) and the out-of-focus distance (in-focus subject). In this regard, the methods disclosed in Patent Documents 2 and 3 cannot correct the color blurs of the focused subject and the blurred subject.

  Therefore, the present invention provides an image processing device, an imaging device, an image processing method, an image processing program, and an image processing device that can effectively reduce color blur in a color image even when the color blur characteristic varies depending on the subject distance, such as a three-dimensional subject. And a storage medium.

  An image processing apparatus according to an aspect of the present invention relates to an acquisition unit that acquires information about a defocus amount of an image formed via an imaging optical system, and information about the defocus amount and color blurring of the imaging optical system. And a determination unit that determines whether to correct the image so as to reduce the color blur using the optical information.

  An imaging device according to another aspect of the present invention includes an imaging element that photoelectrically converts an optical image formed via an imaging optical system and outputs an image, and an acquisition unit that acquires information about the defocus amount of the image And a determination unit that determines whether to correct the image so as to reduce the color blur using the information regarding the defocus amount and the optical information regarding the color blur of the imaging optical system.

  An image processing method according to another aspect of the present invention relates to a step of acquiring information related to a defocus amount of an image formed through an imaging optical system, and information related to the defocus amount and color blurring of the imaging optical system. Determining whether to correct the image using optical information so as to reduce the color blur.

  An image processing program according to another aspect of the present invention relates to a step of acquiring information relating to a defocus amount of an image formed through an imaging optical system, and information relating to the defocus amount and color blurring of the imaging optical system. Determining whether to correct the image so as to reduce the color blur using optical information, and causing the computer to execute the step.

  A storage medium according to another aspect of the present invention stores the image processing program.

  Other objects and features of the invention are described in the following embodiments.

  According to the present invention, an image processing device, an imaging device, an image processing method, and an image processing program capable of effectively reducing color blur in a color image even when the color blur characteristic varies depending on the subject distance as in a three-dimensional subject. And a storage medium can be provided.

It is a flowchart which shows the image processing method in this embodiment. In this embodiment, it is a figure which shows an example in case the pixel area of the monotonous increase / decrease determination object has a monotonous increase / decrease characteristic. In this embodiment, it is a figure which shows an example in case the pixel area of the monotonous increase / decrease determination object does not have a monotonous increase / decrease characteristic. In this embodiment, it is a figure which shows the pixel area for the monotone increase / decrease determination centering on an attention pixel. In this embodiment, it is a figure which shows the pixel area for the monotone increase / decrease determination which used the attention pixel as the end point. In this embodiment, it is a figure which shows the area | region of 3x3 pixel in an image, and a figure which shows the result of having applied the low-pass filter with respect to each pixel. In this embodiment, it is a figure which shows the example of the change of an input signal at the time of applying a low-pass filter with respect to each pixel. In this embodiment, it is a figure which shows the example of the monotone increase / decrease detection of the imaging optical system which has a symmetrical aberration. In this embodiment, it is a figure which shows the example of the monotone increase / decrease detection of the imaging optical system which has an asymmetrical aberration. It is explanatory drawing of the optical information in this embodiment. In this embodiment, it is a figure which shows the example which determines a color bleeding area | region based on optical information and a monotonous increase / decrease detection result. It is explanatory drawing of the two-dimensional optical information in this embodiment. In this embodiment, it is a figure which shows the color blur determination process using optical information. It is explanatory drawing of the calculation method of the optical information in this embodiment. In this embodiment, it is a figure which shows the typical pixel value change of a blue blur. In this embodiment, it is a characteristic view of the nonlinear conversion with respect to the pixel value of B plane. It is a figure which shows the chromaticity coordinate ab surface in this embodiment. In this embodiment, it is a figure which shows the area | region of 3 * 3 pixel in B plane by a monotone increase / decrease determination result. In the present embodiment, it is an example of color blur correction data corresponding to a defocus amount. 1 is a configuration diagram of an imaging apparatus in Embodiment 1. FIG. 1 is a schematic diagram of an imaging system in Embodiment 1. FIG. 1 is a diagram illustrating an imaging system in Embodiment 1. FIG. 1 is a diagram illustrating an imaging system in Embodiment 1. FIG. 1 is a diagram illustrating an imaging system in Embodiment 1. FIG. It is a figure which shows the conventional image pick-up element. It is a figure which shows the image obtained by the imaging system of FIG. It is a figure which shows the image obtained by the imaging system of FIG. 23 and FIG. 1 is a diagram illustrating an example of an imaging device in Embodiment 1. FIG. 1 is a diagram illustrating an example of an imaging device in Embodiment 1. FIG. FIG. 6 is a configuration diagram of an image processing system in Embodiment 2. It is explanatory drawing of the longitudinal chromatic aberration according to a to-be-photographed object distance. It is explanatory drawing of the longitudinal chromatic aberration according to a to-be-photographed object distance. It is explanatory drawing of the longitudinal chromatic aberration according to a to-be-photographed object distance. It is explanatory drawing of the longitudinal chromatic aberration according to a to-be-photographed object distance. It is explanatory drawing of the defocus map in this embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, an image processing method in this embodiment will be described with reference to FIG. The image processing method described here is executed, for example, by each unit of an image processing apparatus (image processing unit 104, image processing apparatus 201) of each embodiment described later. FIG. 1 is a flowchart of an image processing method (image processing program) in the present embodiment. Each step of FIG. 1 is realized by each unit of the image processing apparatus based on a command of a computer (processor) that executes an image processing program stored in a storage unit (memory), for example. The memory or the processor may be provided either inside or outside the image processing apparatus.

  First, in step S101, the image processing apparatus (acquisition unit) acquires a captured image as an input image. The input image is a digital image (captured image) obtained by receiving light with the image sensor via the imaging optical system, and is deteriorated by the aberration of the imaging optical system including the lens and various optical filters. Note that the imaging optical system can be configured using not only a lens but also a mirror (reflection surface) having a curvature.

  The color component of the input image has information on RGB color components, for example. As the color component, other commonly used color spaces such as brightness, hue, and saturation expressed in LCH, luminance expressed in YCbCr, and color difference signals can be selected and used. As other color spaces, XYZ, Lab, Yuv, and JCh can be used. Color temperature may also be used. The captured image can be acquired by connecting the imaging device and the image processing device by wire or wirelessly. The captured image can also be acquired via a storage medium.

  Subsequently, in step S102, the image processing apparatus (acquisition unit) acquires imaging conditions (imaging condition information) of the imaging optical system. The imaging conditions are the state of the imaging optical system such as the focal length, aperture value, and imaging distance (focus lens position). The shooting condition can be acquired from information added to the image such as Exif information.

  Subsequently, in step S103, the image processing apparatus (acquisition unit) generates a defocus map. A method for generating the defocus map will be described later. The defocus map is data relating to the shift amount of each subject distance within the angle of view from the focusing distance of the imaging optical system. The defocus map may be a subject distance map converted into a distance to each subject.

  In step S <b> 113, the image processing apparatus (acquisition unit) acquires optical information related to color blur corresponding to the shooting condition. The optical information is information related to the optical design value of the imaging optical system, and is correction data for color blur. The optical information is data that is stored in advance in a storage unit (memory) of the imaging device or the image processing device based on design values of the imaging optical system or data measured in the manufacturing process. For example, when the imaging device holds correction data and the image processing device does not hold correction data, it is transmitted from the imaging device to the image processing device by communication, or the correction data is added to the image data. It can be transmitted indirectly.

  Subsequently, in step S104, the image processing apparatus (determination unit) divides the image into a plurality of regions according to the defocus amount (defocus map). For example, a total of two in-focus subjects and the background, three in total on the front side and rear side with respect to the in-focus distance, and more finely divided. In step S105, the image processing apparatus (determination unit) sets (determines) a reference color for each of the plurality of areas divided in step S104 according to the defocus amount (defocus map). For example, one of RGB is used as the reference color. Then, the determination unit determines whether or not to correct the image so as to reduce the color bleeding of a different color (a color other than the reference color) according to the information about the defocus amount (defocus map or the like). The optical information is preferably optical information related to color bleeding of different colors depending on the defocus amount.

  Here, with reference to FIG. 31 to FIG. 34, it will be described that the color blur color varies depending on the defocus amount. 31 to 34 are explanatory diagrams of longitudinal chromatic aberration according to the subject distance.

  FIG. 31 illustrates a situation in which the imaging device 601 captures black bars (subjects 611 to 615) that are arranged obliquely from the near point to the far point. The focus is on the subject 613. FIG. 32 shows an image captured in the state of FIG. FIG. 33 is a cross-sectional view of the arrowed portions of the subjects 611, 613, and 615 in FIG. As shown in FIG. 33, the subject 613 is located at the in-focus distance, the subject 611 is located closer to the in-focus distance, and the subject 615 is located farther than the in-focus distance. The cross-sectional view of the in-focus distance subject 613 in FIG. 33 is focused on the G component, so that the edge blur of the G component is the least. The R component and the B component are substantially the same amount and are blurred compared to the G component.

  FIG. 34 is a schematic optical path diagram of the imaging optical system in the situation of FIG. FIG. 34 shows incident light on the image sensor 102 at each subject distance. In the case of the above-described focusing distance, the G component forms an image on the image sensor, but the R component and the B component have a spread on the image sensor. In the case of a long distance, the R component and the B component are imaged on the image sensor, but the G component has a spread on the image sensor. In the case of a long distance, both the RGB components have a spread, and in particular, the R component and the B component have a larger spread than the G component. FIG. 34 describes axial chromatic aberration, but a similar phenomenon occurs with respect to color variation, although asymmetry occurs in the basic imaging characteristics even at a position with a field angle.

  Next, the defocus map will be described with reference to FIG. FIG. 35 is an explanatory diagram of a defocus map in the present embodiment. FIG. 35 (a) is an image taken while focusing on a person as a subject. FIGS. 35B to 35D show regions divided according to the defocus amount. In each figure, the white area is an area divided as the same distance. In addition, since the color blur characteristic (chromatic aberration) of the imaging optical system changes in accordance with the defocus amount, the most focused color plane at each defocus amount is used as the reference color (reference color plane).

  Next, the monotonous increase / decrease detection process in steps S106 to S110 and S114 will be described. The image processing apparatus (detection unit) detects a region (first region) in which the pixel value (signal level) of any one of the color planes constituting the color image monotonously increases or decreases To do. Then, the image processing apparatus (detection unit) tentatively determines the detected area as a color blur generation area. Several methods are conceivable as a method of determining the color blur generation region in the monotonous increase / decrease detection step. In the present embodiment, the determination is made based on the change characteristic of the pixel value that blurs in a certain pixel section (predetermined section).

  Since color blur occurs when the imaging position shifts in the direction along the optical axis depending on the wavelength, for example, in the case of blue blur, the blue plane (B plane) is out of focus (out of focus). . The color blur due to the defocusing exists over a certain pixel section, and the pixel value change of the color blur at that time has a characteristic of gradually decreasing from the highlight portion to the shadow portion of the image. Therefore, when the pixel value change of color blur in a certain pixel section has a monotonous increase or monotonic decrease characteristic, there is a possibility that color blur has occurred, so that it is temporarily determined as a color blur generation region.

  Actually, when the imaging optical system has an asymmetric (rotationally asymmetric) aberration with respect to the optical axis (center axis) such as coma aberration, the color blur generated by the imaging optical system is the direction in which the subject is generated. There is a direction that does not occur. For this reason, if the color blur generation area is determined only from monotonous increase / decrease, if the subject has a color similar to the color blur, the subject color may be erroneously determined to be color blur. Accordingly, the color blur occurrence region may not be correctly determined only by monotonous increase / decrease detection, and therefore, the provisional determination is made here.

  In the present embodiment, the region for detecting monotonic increase or monotonic decrease is a horizontal, vertical or diagonal pixel section centered on the target pixel of the color image, a horizontal direction with the target pixel as an end point, It includes either a vertical or diagonal pixel section.

  Detection of monotonic increase / decrease is performed by first calculating the pixel value gradient of the color plane. For example, when the input image is composed of three color planes, ie, a G plane, a B plane, and an R plane, the B plane is set as a color blur removal target, and the G plane is used as a reference plane (reference color). The respective luminance gradients Blea and Glea with respect to the B plane and the G plane are calculated as in the following equation (1).

  In Expression (1), G (x + 1, y) and B (x + 1, y) are pixel values on the right side of the target pixel in the G plane and B plane, respectively. G (x−1, y) and B (x−1, y) are pixel values on the left side of the target pixel in the G plane and the B plane, respectively. G (x, y + 1) and B (x, y + 1) are pixel values below the pixel of interest in the G plane and B plane, respectively. G (x, y-1) and B (x, y-1) are pixel values above and below the pixel of interest in the G plane and B plane, respectively. Although this embodiment will be described with three color planes, processing can be performed with any color plane depending on the number of color planes constituting the image.

  First, in step S106 in FIG. 1, the image processing apparatus analyzes a change in the pixel value of the input signal with respect to vertical, horizontal, and diagonal pixel sections when the target pixel of the input image is the center. Subsequently, in step S107, the image processing apparatus (determination unit) determines whether or not the pixel value change of the input signal in the pixel section has a monotonous increase / decrease characteristic.

  As a result of the determination in step S107, if the change in the pixel value of the input signal in the pixel section does not have a monotonous increase / decrease characteristic, in step S108, the image processing apparatus performs vertical and horizontal when the target pixel is the end point.・ A pixel value change of an input signal is analyzed for an oblique pixel section. Subsequently, in step S109, it is determined whether or not the pixel value change of the input signal in the pixel section has a monotonous increase / decrease characteristic.

  FIG. 2 is a diagram illustrating an example of a case where a pixel section subject to monotonous increase / decrease determination has a monotonous increase / decrease characteristic, where the vertical axis indicates the pixel value and the horizontal axis indicates the distance. FIG. 3 is a diagram illustrating an example of a case where a pixel section for which monotonous increase / decrease is determined does not have a monotonous increase / decrease characteristic, where the vertical axis indicates the pixel value and the horizontal axis indicates the distance. The image processing apparatus performs monotonous increase / decrease determination on an input signal having a change in pixel value as shown in FIGS. The white squares shown in FIGS. 2 and 3 are the target pixels.

  As shown in FIG. 2, an image in which a change in the pixel value of the input signal has a monotonous increase / decrease characteristic has a monotonous increase / decrease characteristic in a pixel section where the monotonous increase / decrease determination is performed. Therefore, as a result of the analysis in step S106, in step S107, the image processing apparatus determines that the pixel section has a monotonous increase / decrease characteristic. On the other hand, it is determined that the image having the characteristic of changing the pixel value of the input signal as shown in FIG. 3 does not have the monotonous increase / decrease characteristic.

  When the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic (monotonic increase characteristic or monotonic decrease characteristic), the image processing apparatus sets a monotonous increase / decrease flag in step S114. On the other hand, when the change in the pixel value of the input signal in the pixel section does not have the monotonous increase / decrease characteristic (monotonic increase characteristic or monotonic decrease characteristic), the image processing apparatus does not set the monotonous increase / decrease flag in step S110.

  The monotonous increase / decrease determination described above is applied to each pixel of the B plane. As a result, each pixel is associated with “1” if the monotonous increase / decrease flag is set, and “0” if the monotone increase / decrease flag is not set, and is generated and held as a monotone increase / decrease determination result plane. Used in determination (step S15). Details of the method of using the monotonous increase / decrease determination result plane will be described later.

  Next, with reference to FIG. 4 and FIG. 5, a method of setting a pixel section for performing monotonous increase / decrease determination on the target pixel will be described. FIGS. 4A to 4D are diagrams showing a pixel interval for monotonic increase / decrease determination centered on the target pixel. FIGS. 5A to 5H are diagrams showing a monotone increase / decrease pixel section with the target pixel as an end point. Of the method of setting the pixel section centered on the target pixel and the method of setting the pixel section centered on the target pixel, the method of setting the pixel section centered on the target pixel is shown in FIGS. A method of setting the horizontal direction and the vertical direction around the target pixel as in FIG.

  Further, as a method of setting a pixel section centered on the target pixel, a method of setting the target pixel in an oblique direction as shown in FIGS. 4C and 4D can be considered. In other words, the isotropic property can be imparted to the diagonal direction by setting a pixel section having a distance similar to that in the horizontal direction or the vertical direction. In this case, the angle in the oblique direction is limited to a setting of 45 degrees with respect to the horizontal direction or the vertical direction as shown in FIGS. However, the present embodiment is not limited to this, and an arbitrary angle other than 45 degrees can be set. The distance d of the pixel section at that time is calculated using the following equation (2).

In Expression (2), x represents the horizontal direction, and θ represents the angle from the horizontal.

  On the other hand, the color blur around the highlight area and the color blur around the shadow area in the image are affected by saturation and noise, respectively. There is a case where it cannot be judged correctly because it does not have an increase / decrease characteristic. In this case, as shown in FIG. 5, a method of setting a pixel section with the target pixel as an end point is effective. When the monotonous increase / decrease determination is performed by the above-described method and any one of the pixel intervals shown in FIGS. 4 and 5 has a pixel interval having the monotonous increase / decrease characteristic, the target pixel has the monotonous increase / decrease characteristic. It is determined to be a pixel.

  In the present embodiment, the pixel value of the input signal is used as a target for monotonic increase / decrease determination, but luminance gradient may be used. In this case, for example, it is determined that the color blur is caused when the change in the luminance gradient is reversed only once in a certain pixel section. It is effective to set the appropriate value of the number of pixels in the pixel section to the minimum color blur width among the color blurs generated under certain shooting conditions of the imaging apparatus.

  The color blur width varies depending on the imaging conditions of the imaging apparatus (aperture value, focal length, focusing accuracy, focus position in the image plane, coordinates on the image sensor, etc.). For this reason, it is possible to detect the color blur of the minimum width by matching the appropriate value of the number of pixels in the pixel section with the color blur of the minimum width of various color blurs generated under various shooting conditions. The maximum width color blur can be detected even by using a pixel section adapted for the minimum width color blur.

  By performing monotonous increase / decrease determination as described above, it is possible to extract a color blur to be detected. However, depending on the shooting conditions such as high ISO sensitivity, the S / N ratio may decrease due to noise included in the input signal, and as a result, the color blur may not have a monotonous increase / decrease characteristic. In that case, it is effective to perform a filtering process using a digital filter on the input signal. In the present embodiment, a case where the digital filter is a low-pass filter will be described, but the present invention is not limited to this.

  There are several methods for applying a low-pass filter to the input signal. For example, a case will be described in which a [1 2 1] low-pass filter in which the weighting coefficient of the pixel of interest is twice as large as that of adjacent pixels is applied. FIG. 6A is a diagram illustrating a 3 × 3 pixel region in an image. As shown in FIG. 6, when p is a target pixel in a certain region of 3 × 3 pixels in an image, first, a low-pass filter of [1 2 1] is applied in the horizontal direction. At this time, the target pixel P ′ is expressed as the following Expression (3).

When the adjacent pixels are similarly calculated, the result is as shown in FIG. FIG. 6B is a diagram illustrating a result of applying a low-pass filter to each pixel in a 3 × 3 pixel region in the image. Subsequently, when the low-pass filter of [1 2 1] is applied in the vertical direction, the target pixel P ″ is expressed as the following Expression (4).

  With reference to FIG. 7, an example of a change in the input signal when the low-pass filter is applied will be described. FIG. 7 is a diagram illustrating an example of a change in an input signal when a low-pass filter is applied to each pixel. In FIG. 7, the horizontal axis indicates the distance (cross section on the image, that is, the pixel section from the target pixel), and the vertical axis indicates the pixel value of the plane.

  The solid line in FIG. 7 indicates the case where the low-pass filter is not applied, the fine broken line indicates the case where the low-pass filter [1 2 1] is applied, and the coarse broken line indicates the case where the low-pass filter [1 4 6 4 1] is applied. Yes. Here, [1 4 6 4 1] means that a low-pass filter is applied by applying a weighting factor to a pixel adjacent to the target pixel and a pixel separated by another pixel. By smoothing the input signal by applying the low-pass filter in this way, it is possible to highlight the monotonous increase / decrease characteristics inherent in color fringing. In this embodiment, the low-pass filter is applied in the order of the horizontal direction and the vertical direction, but the present invention is not limited to this. The low-pass filter may be applied in the reverse order, that is, in the order of the vertical direction and the horizontal direction. It is also possible to calculate a two-dimensional low-pass filter coefficient and apply the low-pass filter simultaneously in the horizontal direction and the vertical direction. In the present embodiment, the case of performing the monotone increase / decrease determination of the B plane using the G plane as a reference color has been described. However, the present invention is not limited to this. As the reference color, the color plane set in step S105 can be used.

  Subsequently, in step S111, the image processing apparatus branches the subsequent processing according to the monotonous increase / decrease flag. When the monotonous increase / decrease flag is set, the process proceeds to step S115. In step S115, the image processing apparatus (determination unit) performs a color blur determination. That is, the image processing apparatus determines whether or not it is a color blur generation area by using the optical information of the color blur generation direction acquired in step S113 with respect to the area temporarily determined as the color blur generation area.

  Here, with reference to FIG. 8 and FIG. 9, an adverse effect in the case where the color blur area is determined only from the monotonous increase / decrease detection process will be described. An image obtained by imaging a subject with an imaging device such as a digital camera includes a blur component as an image degradation component caused by spherical aberration, coma aberration, field curvature, astigmatism, and the like of the imaging optical system. . Such a blur component is generated when a light beam emitted from one point of a subject to be collected again at one point on the imaging surface when an aberration is not caused and there is no influence of diffraction is formed by forming an image with a certain spread.

  The blur component is optically represented by a point spread function (PSF), and is different from blur due to a focus shift. In addition, color blur in a color image can also be said to be a difference in blurring for each wavelength of light with respect to axial chromatic aberration, spherical spherical aberration, and color coma aberration of the optical system. Further, the lateral color misregistration can also be referred to as a positional misalignment or a phase misalignment due to a difference in imaging magnification for each wavelength of light when the lateral chromatic aberration is caused by the optical system.

  An optical transfer function (OTF) obtained by Fourier-transforming the point spread function (PSF) is frequency component information of aberration and is represented by a complex number. The absolute value of the optical transfer function (OTF), that is, the amplitude component, is referred to as MTF (Modulation Transfer Function), and the phase component is referred to as PTF (Phase Transfer Function). MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively. Here, the phase component is expressed as the following equation (5) with the phase angle as the phase angle.

PTF = tan ⁻¹ (Im (OTF) / Re (OTF)) (5)
In formula (5), Re (OTF) and Im (OTF) are the real part and the imaginary part of OTF, respectively.

  As described above, since the optical transfer function (OTF) of the optical system deteriorates the amplitude component and the phase component of the image, each point of the subject is asymmetrically blurred with respect to the optical axis like coma aberration. It becomes a state.

  FIG. 8 is a diagram for explaining a case where monotonous increase / decrease detection is performed on an image captured in a state of optical characteristics in which each point of the subject is symmetric (rotationally symmetric) with respect to the optical axis such as spherical aberration. 8A shows the pixel value section of the subject, FIG. 8B shows the PSF section of the imaging optical system, and FIG. 8C shows the pixel value section of the subject imaged by the imaging optical system. FIG. 8 assumes an R plane or a B plane. As shown in FIG. 8B, when the imaging optical system has aberration characteristics that are symmetric (rotationally symmetric) with respect to the optical axis and the PSF is symmetric, the edge cross sections on both sides of the subject are both degraded images. Photographed as shown by the solid line in FIG. In FIG. 8C, the luminance cross section of the subject shown in FIG. 8A is indicated by a dotted line for comparison. If the monotonous increase / decrease determination is performed using the white square in FIG. 8C as the target pixel, both edges are determined as the monotonous increase / decrease area. Further, since color blur actually occurs, the color blur generation region is correctly determined.

  FIG. 9 is a diagram illustrating a case where monotonous increase / decrease detection is performed on an image captured with optical characteristics in which each point of the subject is asymmetric (rotationally asymmetric) with respect to the optical axis such as coma. 9A is a pixel value section of the subject, FIG. 9B is a PSF section of the imaging optical system, and FIG. 9C is a pixel value section of the subject imaged by the imaging optical system. As shown in FIG. 9B, when the imaging optical system has an asymmetric aberration characteristic with respect to the optical axis and the PSF is asymmetric, the edge cross section of the subject is one side particularly in the optical characteristic that greatly deteriorates only on one side edge. Only an image that is greatly degraded is taken as shown by the solid line in FIG. When monotonous increase / decrease detection is performed on such a captured image, both edges are determined as monotonous increase / decrease areas, but since the color blur actually generated by the imaging optical system is only on one side, the original color of the subject is blurred. As a result, an adverse effect occurs. If the color blur correction is performed only by monotonous increase / decrease, the original color of the subject is removed, which may result in an unnatural image. If you try to avoid this, the correction will be weak.

  Therefore, in the present embodiment, the image processing apparatus holds or acquires optical information related to the color blur occurrence direction of the imaging optical system as shown in FIG. 8B or FIG. 9B. When the monotonous increase / decrease direction of the monotonic increase / decrease detection matches the color blur generation direction, the image processing apparatus determines that the color blur generation region is present. Here, the term “matching” means not only the case of exact matching but also the case of substantial matching.

  Here, the optical information will be described with reference to FIG. FIG. 10 is an explanatory diagram of optical information in the present embodiment, and shows a PSF cross section and optical information of the imaging optical system. FIG. 10A shows optical information when the PSF of the imaging optical system shown in FIG. 8B is symmetric with respect to the optical axis. FIG. 10B shows optical information when the PSF of the imaging optical system shown in FIG. 9B is asymmetric with respect to the optical axis. For example, it has 1 in the direction in which color blur occurs due to the aberration of the imaging optical system, and 0 if it does not occur. In FIG. 10A, since the color blur occurs symmetrically on both sides, the optical information has a value of 1 on both sides. In FIG. 10B, the color blur is generated asymmetrically, no color blur is generated at the left edge, and 0 is generated at the right edge, and the value is 1 at the right edge.

  FIG. 11 is a diagram illustrating an example of determining a color blur area based on optical information and a monotonous increase / decrease detection result. As shown in FIG. 11A, for example, when monotonic decrease is detected by monotone increase / decrease detection, the optical information in the monotonic decrease direction is referred to, and if it has 1, it is determined as a color blur correction region. To do. Also, as shown in FIG. 11B, the optical information in the monotonously decreasing direction is referred to, and if it has 0, it is determined that the color of the subject. As described above, by having information on the direction in which chromatic aberration occurs and the direction in which it does not occur as the aberration of the imaging optical system, it is possible to correctly determine the color and color blur of the subject, thereby improving the accuracy of color blur correction. It becomes possible. Although the direction of occurrence of color blur is 1 and the direction of non-occurrence is 0, it is only necessary to know the direction of occurrence, and it is not necessary to be 0 and 1. Since the aberration of the imaging optical system changes depending on the image height, it is possible to correct with high accuracy by having optical information at a plurality of image heights. For image heights that do not have correction values, interpolation may be generated from nearby correction values. The aberration of the imaging optical system also changes depending on the shooting conditions (focal length of the imaging optical system, subject distance (shooting distance), aperture value (Fno)). Therefore, by storing or acquiring optical information for each shooting condition, It becomes possible to perform correction with high accuracy.

  10 and 11, the optical information has been described as one-dimensional information. However, when the captured image is two-dimensional array data, the optical information may be held as two-dimensional data. FIG. 12 is an explanatory diagram when the optical information is two-dimensional. FIG. 12C shows an example in which the optical information is held as data in eight directions of up, down, left and right, 45 degrees oblique and 135 degrees on the screen. In FIG. 12C, color coma in the imaging optical system is generated in the upward direction of the screen, and 1 optical information is held in the generation direction. 0 is maintained in the direction where no color blur occurs. Monotonic increase / decrease detection is performed in the above eight directions, and it is determined that the color is blurred when 1 of the optical information is held in the direction in which the monotonic decrease is detected. Even if a monotonic decrease is detected, if the optical information 1 is not held in the detected direction, it is determined as the color of the subject.

  As described above, the image processing apparatus holds or acquires the optical information regarding the direction in which the color blur of the imaging optical system occurs. Preferably, the image processing apparatus includes a storage unit that stores optical information. More preferably, the storage unit stores optical information for each photographing condition. The imaging condition includes at least one of a focal length of the imaging optical system, a subject distance, and an aperture value. Note that the storage unit may be provided outside the image processing apparatus. Each optical information can be stored in the storage unit 108 described later.

  The image processing apparatus can determine the color blur area with high accuracy by referring to the detection direction and the optical information when detecting monotonous increase / decrease. In the present embodiment, the monotonous increase / decrease determination has been described in the monotonic decrease direction. However, the color blur determination may be performed using the monotone increase direction. In addition, the monotonous increase / decrease determination and the correction value of the optical information have been described in eight directions, but are not limited to eight directions. In order to improve accuracy, the number of detection directions may be increased. Alternatively, the number of detection directions may be reduced in order to reduce the amount of data related to optical information.

  The image processing apparatus may be configured to hold or acquire information regarding the color blur intensity (color blur intensity information) as optical information in addition to the color blur generation direction. In this case, correction can be performed with higher accuracy. As shown in FIG. 12B, the amount of color blurring due to the aberration of the imaging optical system generally differs depending on the direction. Therefore, as shown in FIG. 12D, color blur intensity information regarding the amount of color blur may be added in the direction in which color blur occurs.

  Next, with reference to FIG. 13, color blur determination when color blur intensity information is used will be described. FIG. 13 is a diagram showing a color blur determination process using optical information, and shows an example in which the optical information has only the color blur generation direction. When determining the color blur generation region for the pixel of interest 1 surrounded by the square in the figure, first, monotonic decrease determination is performed in eight directions, up and down, left and right, 45 degrees oblique, and 135 degrees oblique, with the pixel of interest at the center. In the case of FIG. 13A, since the high-intensity subject is located below the target pixel, three directions for detecting the monotonic decrease of the target pixel 1 are determined as upper, upper right, and upper left. Since the optical information in the three directions is “1” indicating the color blur generation direction, the target pixel 1 is determined to be a color blur generation region. Similarly, if the pixel of interest 2 is also detected, the monotone decreasing direction is determined to be upper, upper right, right, and lower right. However, since the color blur occurrence direction of the optical information holds “0” indicating a direction that does not occur in the lower right, the final color blur detection direction of the target pixel 2 is monotonically decreased and the optical information is higher than the optical information. , Upper right and right. Although both the target pixel 1 and the target pixel 2 are determined to be color blur generation pixels, as shown in FIG. 12B, the amount of color blur generation in the imaging optical system differs depending on the direction, so the target pixel 1 and the target pixel 2 Then, the amount of color blur is different. If the optical information does not have intensity information, the amount of color blur cannot be determined. Therefore, correction cannot be performed by changing the intensity of correction in the subsequent color blur correction process. Therefore, if the color blur correction is increased in accordance with the target pixel 1, the target pixel 2 is overcorrected, and if the color blur correction is decreased in accordance with the target pixel 2, the target pixel 1 is insufficiently corrected.

  Therefore, as shown in FIG. 12D, it is preferable to add the color blur generation direction and the color blur intensity information in the direction to the optical information. FIGS. 13B and 13C show a method of determining a color blur area when color blur intensity information is also added to the optical information. In the target pixel 3 in FIG. 13B, the three directions of upper, upper left, and upper right are determined by monotonic decrease determination, but the color blur intensity of the optical information is 2, 3, and 2, respectively, and the strong color blur intensity is detected. can do. In the pixel of interest 4 in FIG. 13C, the three directions of upper, upper right, and lower left are determined, but the color blur intensity of the optical information in that direction is 3, 2, 1, and weaker than the pixel of interest 3, respectively. A color blur intensity is detected. Therefore, for example, by changing the intensity of correction (image correction intensity) of the subsequent color blur correction process according to the average value of the color blur intensity in the detected direction, Even when different, more accurate correction is possible.

  The target pixel 3 has an average color blur intensity value of about 2.3, and the target pixel 4 has an average color blur intensity value of 2.0, which can be used to change the correction strength. It becomes. In the present embodiment, the description has been made using the average value of the color blur intensity. However, the present invention is not limited to this. For example, the color blur correction coefficient may be changed using the color blur intensity value in the centroid direction of a plurality of detected directions as the color blur direction. At the target pixel 3, the direction of the center of gravity is upward, so the color blur intensity is 3, and at the target pixel 4, the direction of the center of gravity is upper right, so the color blur intensity is 2, and this value can be used to change the correction strength. Become. When the center of gravity direction is adopted, for example, when the center of gravity in the four directions of upper, upper right, right, and lower right is calculated, the direction closest to the center of gravity direction may be the two directions of upper right and right. In this case, it is conceivable to use an average value in two directions, a color blur intensity of the larger one of the two directions, or a color blur intensity of the smaller of the two directions. In the present embodiment, the use of the average value and the direction of the center of gravity has been described. However, the color blur intensity information of the optical information may be arbitrarily calculated to change the correction strength at the time of color blur correction.

  Next, a method for calculating optical information will be described with reference to FIG. FIG. 14 is an explanatory diagram of a method for calculating optical information, and shows a PSF cross section (solid line) of the G plane of the imaging optical system and a PSF cross section (dotted line) of the B plane in an arbitrary direction held as optical information. Yes. Here, a method for calculating the optical information regarding the color blur of the B plane using the G plane as a reference plane will be described.

  Due to the chromatic aberration of the imaging optical system, the G plane and the B plane have different PSF shapes depending on the aberration in each wavelength band. When the lateral chromatic aberration remains, the G plane and the B plane are imaged at shifted positions, so the peak positions of the respective PSFs are also calculated at shifted positions as shown in FIG. Is done. The displacement of each color plane due to lateral chromatic aberration can be corrected by a conventional geometric transformation that applies different distortions to each color plane. Therefore, as shown in FIG. 14B, the PSF peak positions of the G plane and B plane are made to coincide with each other, and the PSF area difference between the G plane and the B plane, which is the hatched portion in the figure, is blurred. A method of adopting the strength can be considered.

  In FIG. 14B, since the area difference of the right edge is larger than the area difference of the left edge than the peak, it can be seen that the color blur occurs greatly at the right edge. If this area is small, the color blur is inconspicuous, and therefore, when the area is small, the direction in which the color blur does not occur can be set. Further, the color blur intensity can be determined from the area ratio. Since it may be erroneously determined that magnification chromatic aberration remains in the image to be subjected to color blur correction, the magnification chromatic aberration correction is performed on the image in advance, and the color chromatic aberration correction data is removed as described above. It is preferable to generate it.

  As described above, the direction in which color blur occurs in the optical information and the color blur intensity can be calculated, but the calculation method is not limited to the method using the area difference of the PSF. For example, in FIG. 14B, a method of using the difference in inclination between the edges of the G plane and the B plane can be considered. When color blurring occurs, the inclination of the edge tends to be gentle, and thus optical information may be determined by using a difference in inclination.

  In the present embodiment, the method of matching the peak values when calculating the color blur intensity is described, but the color blur intensity may be calculated by matching the centroids of the PSFs instead of the peak values. As described above, according to the present embodiment, the optical correction value can be calculated for each direction in which the monotonous increase / decrease determination is performed, the image height, and the shooting conditions.

  Returning to FIG. 1, in step S <b> 116, the image processing apparatus (correction unit) corrects color blur for the region (second region) determined as the color blur region by the determination unit in step S <b> 115 (image). (Color blur correction process). That is, the correction unit corrects the image so as to reduce color blur.

  FIG. 15 is a diagram illustrating a typical pixel value change of blue blur. In FIG. 15, the horizontal axis is the distance (cross section on the image), and the vertical axis is the pixel value of the B plane and the G plane. In FIG. 15, it is assumed that there is a high-luminance subject that exceeds the saturation luminance at the left end. Then, the periphery of the high-brightness subject which is not originally bright also spreads so that the bottom of the pixel value change gradually approaches the black level due to aberration and flare. The G plane which is the reference plane is not free from bleeding, but has a certain extent. However, it is smaller than the B-plane that is subject to color blur removal. Further, the image sensor cannot measure pixel values above a certain saturation level. In such a pixel value change, if the intensity of the B-plane to be subjected to color bleeding removal exceeds the intensity of the G-plane that is the reference plane, blue bleeding occurs.

  In the present embodiment, the image processing apparatus estimates the amount of blur of the B plane based on the slope of the change in the pixel value of the B plane that is subject to color blur removal. Therefore, by multiplying the absolute value of the brightness gradient Blea of the B plane by the coefficient k1, the first estimated blur amount E1 is obtained as expressed by the following equation (6).

E1 = k1 | Blea | (6)
In Expression (6), k1 is a positive value. However, in the area A1 where the B plane is saturated, the pixel value gradient becomes 0, and the luminance gradient before saturation cannot be obtained.

  Therefore, the estimated bleeding amount E2 for the region A1 where the B plane is saturated is estimated based on the luminance gradient Glea of the pixel value change of the G plane, as shown in the following equation (7).

E2 = k2 | Glea | (7)
In Expression (7), k2 is a positive value.

  Next, nonlinear conversion is performed on the pixel values of the B plane to generate the saturation S. This non-linear transformation indicates whether or not the B plane is saturated. In the region where the intensity of the B plane is saturated, the saturation S is 1, and in the region where the intensity of the B plane is low, the saturation S is 0. Although the saturation S may be a binary value of 0 or 1, the saturation S may be a value that continuously varies from 0 to 1 as shown in FIG. FIG. 16 is a characteristic diagram of nonlinear conversion with respect to the pixel value of the B plane.

Then, the estimated blur amount E1 or the estimated blur amount E2 is selected according to the degree of saturation S. That is, if the saturation S is a binary value of 0 and 1, a new estimated bleeding amount E is obtained.
E = E1 (when S = 0)
E = E2 (when S = 1)
And If the saturation S is a value that continuously changes from 0 to 1, a new estimated bleeding amount E is
E = (1-S) E1 + SE2
And

  Next, the estimated blur amount E is corrected, and the amount E ′ to be actually removed is determined. The estimated bleeding amount (removal amount) is in line with a certain model and does not necessarily match the actual bleeding amount. For example, even in the case of light detected on the same B plane, the blurring method varies between light having a wavelength of 450 nm and light having a wavelength of 400 nm, but this is not considered in the color blur determination step (step S115). When the estimated amount of blurring (removal amount) is too small, a slight amount of blueness remains after the removal of blue blurring. On the other hand, if the estimated blurring amount (removal amount) is excessive, the B plane may be excessively reduced with respect to the gray background, resulting in a yellowish green color.

  In particular, the latter (when it becomes yellowish green) is unnatural and gives the viewer a great sense of discomfort. Therefore, in the color blur correction process (step S116), the blur removal is limited to operate only within a certain hue range. For this reason, first, in step S116, the chromaticity of the pixel is calculated. The following equation (8) is established for the strengths of the R, G, and B planes.

  FIG. 17 is a chromaticity coordinate ab plane in which a in Equation (8) is the horizontal axis and b is the vertical axis. As shown in FIG. 17, blue is in the fourth quadrant indicated by diagonal lines on the chromaticity coordinate ab plane (note that red, yellow and purple are the first quadrant, green and white are the second quadrant, and blue-green is the third quadrant. In the quadrant). When the estimated blur amount E is removed from the intensity of the B plane, B = B−E, and the chromaticity coordinate ab plane moves in the upper left direction as indicated by a dotted arrow. The starting point of the arrow is the chromaticity before the estimated blur amount E is removed, and the end point is the chromaticity after the estimated blur amount E is removed. Therefore, when the working hue range is limited to a> 0 and b <0, the following equation (9) is established.

B> 0.22R + 0.68G and B> −1.84R + 3.30G (9)
For this reason, the pixel E that does not satisfy Expression (9) is set to the removal amount E ′ = 0 that is actually removed, and is excluded from the color blur removal target. As a result, the pixel value of the pixel that does not satisfy Expression (9) is not affected. In FIG. 17, only the region in the fourth quadrant indicated by diagonal lines is the removal target.

  Further, the removal amount E ′ is set so as to satisfy the following expression (10) even for a pixel satisfying the expression (9).

E ′ = min (E, B− (0.22R + 0.68G), B − (− 1.84R + 3.30G)) (10)
As a result, the change in chromaticity due to the removal of the removal amount E ′ remains in the fourth quadrant as indicated by the solid line arrow in FIG.

  In the present embodiment, the restriction is made in the fourth quadrant of the chromaticity coordinate ab plane. However, the present invention is not limited to this, and the restriction may be made at an arbitrary angle. At this time, it is necessary to satisfy the following formula (11).

B> r1 · G + r2 · R and B> r3 · G + r4 · R (11)
In the equation (11), r1 to r4 are calculated as the following equation (12) using the limit angle θ. The hue limit is defined by two straight lines passing through the origin of the chromaticity coordinate ab plane, and θ1 and θ2 are angles representing the two straight lines.

  As a result, it is possible to prevent the B plane from decreasing beyond the hue limit range. The color plane removal amount E ′ calculated as described above is retained as the removal amount plane, and color blur correction processing is performed. A low pass filter is applied to the removal amount plane. In this embodiment, a simple a * b * plane is used for hue restriction. However, the present invention is not limited to this, and hue restriction processing is performed on the uv plane using a 3 × 3 RGB → YUV matrix. You may do.

  A new B plane is created by subtracting the removal amount E ′ from the strength of the B plane. Only the pixels that have been determined as the color blur area in the color blur determination step S115 are to be removed from the color blur. Therefore, the intensity of the new B plane is B = B−E ′ when the monotonous increase / decrease determination flag is “1”. On the other hand, when the monotonous increase / decrease determination flag is “0”, B = B. As described above, in step S112 in FIG. 1, the image processing apparatus outputs a color image obtained by correcting the B plane as an output image.

  Here, consider a case where the monotonic increase / decrease determination result in the 3 × 3 pixel region in the image shown in FIG. 6A switches the value of the monotonous increase / decrease determination flag in adjacent pixels as shown in FIG. In such a case, the removal amount fluctuates in the boundary portion of the pixel (a case where the pixel is removed by a neighboring pixel and a case where the pixel is not removed is mixed), and the change of the pixel value becomes steep, unnatural and uncomfortable to the observer. May give. Therefore, a method of applying a low-pass filter to the generated removal amount plane is effective.

  Also, the following equation (13) calculates the gain of each pixel of the color plane using the monotonic increase / decrease determination result plane, and multiplies the removal amount to perform smoothing processing on the boundary portion (monotonous increase / decrease determination result plane). You may go.

  In the example illustrated in FIG. 18, the removal amount E ″ of the pixel p is expressed as the following Expression (14).

  With the above method, it is possible to remove only the color blur without a sense of incongruity.

  Next, color blur correction data corresponding to the defocus amount will be described with reference to FIG. FIG. 19 is an example of color blur correction data (correction data for color blur optical information) corresponding to the defocus amount.

  When there is axial chromatic aberration, the relative amount of aberration between the color planes changes depending on the defocus amount, so that correction data is generated from the amount of bleeding of other colors using the color component with the smallest PSF spread as the reference color. . In the example of FIG. 19, the sharpest G component is used as a reference for the near point distance and the focusing distance, and the B component is used for the far point distance. In this way, the color blur correction data is held in advance according to the defocus amount. However, when the correction data is held with respect to the discrete defocus amount and the defocus amount does not hold the correction data, it is preferable to reduce the held data by interpolating and using nearby correction data. .

  As described above, by performing the color blur correction process according to the defocus amount, it is possible to remove color blur without a sense of incongruity even for a three-dimensional object. The above is the basic processing flow of the image processing method in the present embodiment.

  Next, with reference to FIG. 20, the imaging apparatus according to the first embodiment of the present invention will be described. FIG. 20 is a configuration diagram of the imaging apparatus 100 in the present embodiment. The imaging apparatus 100 is installed with an image processing program for performing color blur correction processing (the image processing method described with reference to FIG. 1) of the captured image, and this color blur correction processing is an image inside the imaging apparatus 100. It is executed by the processing unit 104 (image processing apparatus).

  The imaging apparatus 100 includes an imaging optical system 101 (lens) and an imaging apparatus main body (camera main body). The imaging optical system 101 includes a diaphragm 101a and a focus lens 101b, and is configured integrally with an imaging apparatus main body (camera main body). However, the present embodiment is not limited to this, and can also be applied to an imaging apparatus in which the imaging optical system 101 is attached to the imaging apparatus body in a replaceable manner.

  The image sensor 102 photoelectrically converts a subject image (optical image, imaging light) obtained via the imaging optical system 101 to generate a captured image. That is, the subject image is photoelectrically converted by the image sensor 102 and converted into an analog signal (electric signal). The analog signal is converted into a digital signal by the A / D converter 103, and the digital signal is input to the image processing unit 104.

  Here, with reference to FIG. 21, the image sensor 102 for generating a defocus map will be described. FIG. 21A is a diagram illustrating a relationship between the light receiving unit of the imaging element and the pupil of the optical system in the imaging system of the present embodiment. In FIG. 21A, ML is a microlens, and CF is a color filter. EXP is an exit pupil (pupil) of the optical system, and P1 and P2 are areas of the exit pupil EXP. G1 and G2 are pixels (light receiving units), and one pixel G1 and one pixel G2 are paired with each other (the pixels G1 and G2 are provided so as to share one microlens ML). . In the imaging device, a plurality of pairs of pixel G1 and pixel G2 (pixel pairs) are arranged. The pair of pixels G1 and G2 have a conjugate relationship with the exit pupil EXP through a common microlens ML (that is, one for each pixel pair). Note that the plurality of pixels G1 and G2 arranged in the image sensor may be collectively referred to as a pixel group G1 and G2, respectively.

  FIG. 21B is a schematic diagram of the imaging system in the present embodiment, and it is assumed that a thin lens is provided at the position of the exit pupil EXP instead of the microlens ML shown in FIG. The imaging system in the case is shown. The pixel G1 receives the light beam that has passed through the region P1 in the exit pupil EXP. The pixel G2 receives the light beam that has passed through the region P2 in the exit pupil EXP. The OSP is an object point being imaged. The object point OSP does not necessarily have an object. The light beam passing through the object point OSP is applied to either the pixel G1 or the pixel G2 according to the position (region P1 or region P2 in this embodiment) in the pupil (exit pupil EXP) through which the light beam passes. Incident. The passage of light beams through different regions in the pupil corresponds to the separation of incident light from the object point OSP by the angle (parallax). That is, among the pixels G1 and G2 provided for each microlens ML, an image generated using the output signal from the pixel G1 and an image generated using the output signal from the pixel G2 are: It becomes a plurality (here, a pair) of parallax images having parallax with each other.

  The respective parallax images made up of the pixel group G1 and the pixel group G2 form an image at the same position on the focused subject, but form an image at a different position on the non-focused subject. . Since this positional shift amount depends on the defocus amount, the defocus amount that is a relative shift amount from the in-focus distance can be obtained based on the positional shift amount. This is the principle used in the phase difference detection type autofocus mechanism. Therefore, a defocus map can be generated by obtaining the defocus amount for the entire image. If the distance from the position of the focus lens 101b to the in-focus subject is obtained, a subject distance map showing the distance to each subject can be generated.

  In FIG. 20, the image processing unit 104 (image processing apparatus) includes an acquisition unit 104a, a determination unit 104b, a detection unit 104c, and a correction unit 104d. The image processing unit 104 (image processing apparatus) performs predetermined processing on the digital signal output from the A / D converter 103 and performs the above-described color blur correction processing.

  First, the image processing unit 104 (acquisition unit 104a) acquires imaging condition information of the imaging apparatus 100 (imaging optical system 101) from the state detection unit 107. The imaging condition information is information regarding an aperture value, a shooting distance (subject distance), a focal length of the zoom lens, and the like. The state detection unit 107 can acquire the imaging condition information directly from the system controller 110, but is not limited to this. For example, the imaging condition information regarding the imaging optical system 101 can be acquired from the imaging optical system control unit 106. The image processing unit 104 (acquisition unit 104a) acquires information related to the defocus amount of an image formed via the imaging optical system 101. The information related to the defocus amount includes defocus information and distance information in the image such as the defocus amount and subject distance, but is not limited thereto. In the present embodiment, more specifically, the image processing unit 104 (acquisition unit 104a) generates a defocus map (subject distance map) of an image based on information on the defocus amount.

  Subsequently, the image processing unit 104 (determination unit 104b) uses the information about the defocus amount (defocus map or subject distance map) and the optical information about the color blur of the imaging optical system so as to reduce the color blur. It is determined whether or not to correct the image. When it is determined that the image is to be corrected, the image processing unit 104 (determination unit 104b) uses the defocus map (subject distance map) to divide the image according to the defocus amount (subject distance).

  Subsequently, the image processing unit 104 (the determination unit 104b, the detection unit 104c, and the correction unit 104) uses the color-bleed optical information (correction data) corresponding to the defocus amount (or subject distance) for each divided region. Performs blur correction processing. The color blur correction process (image processing method) of the present embodiment is as described with reference to FIG. More specifically, in the detection unit 104c, the signal level of a color (color other than the reference color) corresponding to the defocus amount is monotonously increased or monotonically decreased in the first direction in a predetermined section. Detect areas. The determination unit 104b determines a second area where color blur has occurred based on the optical information and information regarding the first area. The correction unit 104d corrects the image so as to reduce color blur.

  Color blur correction data of the imaging optical system 101 is held in the storage unit 108. The image processing unit 104 acquires color blur correction data corresponding to the shooting condition information from the storage unit 108 and performs color blur correction processing. The corrected output image is stored in the image recording medium 109 in a predetermined format. The display unit 105 displays an image that has been subjected to predetermined display processing on the image that has been subjected to the color blur correction process of the present embodiment. However, the present embodiment is not limited to this, and an image obtained by performing simple processing for high-speed display may be displayed on the display unit 105.

  A series of controls in this embodiment is performed by the system controller 110, and mechanical driving of the imaging optical system 101 is performed by the imaging optical system control unit 106 based on an instruction from the system controller 110. The imaging optical system control unit 106 controls the aperture diameter of the diaphragm 101a as the F-number imaging state setting. The imaging optical system control unit 106 controls the position of the focus lens 101b by an unillustrated autofocus (AF) mechanism or manual manual focus mechanism in order to perform focus adjustment according to the subject distance. Note that functions such as aperture diameter control of the diaphragm 101a and manual focus may not be executed according to the specifications of the imaging apparatus 100. The imaging optical system 101 can be provided with optical elements such as a low-pass filter and an infrared cut filter.

  Next, a modified example of the optical system for generating the defocus map or the subject distance map will be described. A first modification is a method disclosed in Japanese Patent Application Laid-Open No. 2013-62803, in which a subject distance is acquired from two images having different shooting conditions. In this method, for example, two images with different focus are captured by moving the focus lens, and the subject distance is generated from the difference in the blur amount of the captured images.

  The second modification uses an imaging system having the configuration shown in FIG. Ren. “Plenoptic Camera” is proposed in “Light Field Photography with a Hand-Held Plenoptic Camera” (Stanford Tech Report CTSR 2005-2) of Ng et al. By using a technique called “Light Field Photography” in this “Plenoptic Camera”, it is possible to capture information on the position and angle of light rays from the object side. Since the information on the angle of the light becomes parallax, a defocus amount or a subject distance map can be generated from a plurality of parallax images.

  In FIG. 22, the imaging optical system 301 includes a main lens (photographing lens) 301b and an aperture stop 301a. A microlens array 301c is arranged at the imaging position of the imaging optical system 301, and an imaging element 302 is arranged behind (on the image side) of the microlens array 301c. The microlens array 301c serves as a separator (separating means) so that a ray group passing through a certain point in the subject space such as the point A and a ray passing through a point near the point A are not mixed on the image sensor 302. It has a function. As can be seen from FIG. 22, the upper line, chief ray and underline from point A are received by different pixels. For this reason, a group of rays passing through the point A can be obtained separately for each angle of rays.

  Also, “Full Resolution Light Field Rendering” (Adobe Technical Report January 2008) by Todor George et al. Is known. This document proposes an imaging system shown in FIG. 23 and FIG. 24 as a method of acquiring light position and angle information (Light Field).

  In the configuration of the imaging system shown in FIG. 23, the microlens array 301c is arranged behind (image side) the imaging position of the main lens 301b, and the light ray group passing through the point A is re-imaged on the imaging element 302. Thus, the light beam group can be obtained separately for each light beam angle. In the configuration of the imaging system shown in FIG. 24, the microlens array 301c is arranged in front of the imaging position of the main lens 301b (on the object side), and a light beam passing through the point A is imaged on the imaging device 302. By doing so, the light beam group can be obtained separately for each angle of the light beam. Both configurations are the same in that the light beam passing through the pupil of the imaging optical system 301 is divided according to the passing region (passing position) in the pupil. In these configurations, as shown in FIG. 25, the imaging element 302 uses a conventional imaging element that is paired via one microlens ML, one light receiving unit G1, and a color filter CF. Can do.

  When the imaging optical system 301 shown in FIG. 22 is used, an image as shown in FIG. 26A is obtained. FIG. 26B shows an enlarged view of one of many circles arranged in FIG. One circle corresponds to the stop STP, and the inside thereof is divided by a plurality of pixels Pj (j = 1, 2, 3,...). Thereby, the intensity distribution of the pupil is obtained within one circle. Further, when the imaging optical system 301 shown in FIGS. 23 and 24 is used, a parallax image as shown in FIG. 27 is obtained. In the image of FIG. 26A obtained by the imaging optical system shown in FIG. 22, parallax is distributed in the pupil. For this reason, a plurality of parallax images as shown in FIG. 27 are obtained by separating the images for each parallax.

  The third modification uses a plurality of imaging devices (cameras) as shown in FIG. Even when the same subject is imaged using a plurality of cameras, a parallax image can be obtained. In the fourth modification example, as shown in FIG. 29, a parallax image can be obtained by providing a plurality of imaging optical systems OSj (j = 1, 2, 3,...) In one imaging apparatus.

  Next, an image processing system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 30 is a configuration diagram of the image processing system 200 in the present embodiment. Note that the color blur correction process (image processing method) of the present embodiment is as described with reference to FIG.

  In FIG. 30, an image processing apparatus 201 is a computer device equipped with image processing software 206 for causing a computer to execute the image processing method of this embodiment. The imaging device 202 is an imaging device such as a camera, a microscope, an endoscope, or a scanner. The storage medium 203 is a storage unit that stores captured images, such as a semiconductor memory, a hard disk, or a server on a network.

  The image processing apparatus 201 acquires captured image data from the imaging device 202 or the storage medium 203, and outputs the image data subjected to predetermined image processing to one or more of the output device 205, the imaging device 202, and the storage medium 203. Output. Further, the output destination can be stored in a storage unit built in the image processing apparatus 201. The output device 205 is a printer, for example. Also, the image processing software 206 can be installed in the image processing apparatus 201 from a network or a storage medium such as the CD-ROM 207.

  A display device 204 that is a monitor is connected to the image processing apparatus 201. Therefore, the user can perform the image processing operation through the display device 204 and can evaluate the corrected image. The image processing software 206 performs color blur correction processing (image processing method) of the present embodiment, and performs development and other image processing as necessary.

  It should be noted that information (correction information) relating to the contents of data for performing image processing and delivery between devices in this embodiment is preferably attached to individual image data. By adding necessary correction information to the image data, the correction processing of the present embodiment can be appropriately performed as long as the device is equipped with the image processing apparatus of the present embodiment. Further, the optical information related to the color blur generation direction of the image pickup optical system may be acquired through a network as well as a method of reading out from a recording unit of a device equipped with the image processing apparatus of the present embodiment, or may be a PC or a camera. Information may be acquired from a lens or the like.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  In each embodiment, optical information relating to the color blur of the imaging optical system is used in consideration of the defocus amount of the subject. Therefore, according to each embodiment, an image processing apparatus, an imaging apparatus, an image processing method, and an image processing apparatus that can effectively reduce color blur in a color image even when the color blur characteristic varies according to the subject distance, such as a three-dimensional subject. An image processing program and a storage medium are provided.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

104 Image processing unit (image processing apparatus)
104a acquisition unit 104b determination unit

Claims (18)

An acquisition unit that acquires information about a defocus amount of an image formed through the imaging optical system;
A determination unit that determines whether to correct the image so as to reduce the color blur using the information about the defocus amount and the optical information about the color blur of the imaging optical system. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the determination unit determines whether to correct the image so as to reduce the color blur of a different color according to information on the defocus amount. .   The image processing apparatus according to claim 2, wherein the optical information is optical information related to the color blur of the color that varies depending on the defocus amount. The acquisition unit generates a defocus map of the image based on information on the defocus amount,
4. The image processing apparatus according to claim 2, wherein the determination unit determines the color to be subjected to color blur reduction based on the defocus map and the optical information. 5. The determination unit
Based on the defocus map, the image is divided into a plurality of regions according to information on the defocus amount,
The image processing apparatus according to claim 4, wherein the image processing apparatus determines whether or not to correct the image so as to reduce the color blur of the color set for each region of the image. A detection unit that detects a first region in which the signal level of the color corresponding to the defocus amount monotonously increases or monotonously decreases in a first direction in a predetermined section;
A correction unit that corrects the image so as to reduce the color blur,
6. The determination unit according to claim 2, wherein the determination unit determines a second region in which the color blur has occurred based on the optical information and information on the first region. The image processing apparatus according to item.   The image processing apparatus according to claim 6, wherein the optical information is information related to a second direction in which the color blur occurs.   The image processing apparatus according to claim 6, wherein the determination unit determines the second region by comparing the first direction and the second direction.   9. The determination unit according to claim 8, wherein the determination unit determines that the first region is the second region when the first direction and the second direction coincide with each other. Image processing device.   The image processing apparatus according to claim 6, wherein the optical information is information regarding the intensity of the color blur.   The image processing apparatus according to claim 10, wherein the correction unit changes an intensity for correcting the image based on information on the intensity of the color blur.   When the first direction and the second direction coincide with each other in a plurality of directions, the correction unit corrects the image by calculating information on the intensity of color blur in the plurality of directions. The image processing apparatus according to claim 9.   When the first direction and the second direction coincide with each other in a plurality of directions, the correction unit corrects the image using information on the intensity of color blur in the centroid direction of the plurality of directions. The image processing apparatus according to claim 9.   The image processing apparatus according to claim 1, wherein the optical information is information related to an optical design value of the imaging optical system. An image sensor that photoelectrically converts an optical image formed via the imaging optical system and outputs an image; and
An acquisition unit for acquiring information on a defocus amount of the image;
A determination unit that determines whether to correct the image so as to reduce the color blur using the information about the defocus amount and the optical information about the color blur of the imaging optical system. An imaging device. Obtaining information relating to the defocus amount of an image formed via the imaging optical system;
Determining whether to correct the image so as to reduce the color blur using the information on the defocus amount and the optical information on the color blur of the imaging optical system. Image processing method. Obtaining information relating to the defocus amount of an image formed via the imaging optical system;
Using the information on the defocus amount and the optical information on the color blur of the imaging optical system to determine whether or not to correct the image so as to reduce the color blur. An image processing program configured as described above.   A storage medium storing the image processing program according to claim 17.