JP2016201601A - Image processing apparatus, imaging device, image processing method, image processing program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that can reduce an influence due to a manufacturing error and effectively reduce color blur in a color image.SOLUTION: An image processing apparatus 100 comprises: detection means 102 that detects a first area where a signal level in a color plane corresponding at least to one color filter in an image that is obtained from an imaging element including a plurality of color filters, or a signal level of a luminance plane that is generated from the image, is monotonously increased or monotonously decreased in a first direction in a predetermined section; determination means 103 that determines a second area where color blur occurs on the basis of first optical information on color blur in an imaging optical system, second optical information on a variation in asymmetric chromatic aberration with respect to an optical axis of the imaging optical system, and information on the first area; and correction means 104 that corrects the image to reduce color blur.SELECTED DRAWING: Figure 1B

Description

  The present invention relates to an image processing apparatus that reduces color bleeding in a color image.

  Conventionally, a color that does not originally exist around a bright portion of a color image may occur as a color blur due to chromatic aberration of the photographing optical system. The chromatic aberration of the photographing optical system can be optically reduced to some extent by combining a plurality of lenses made of materials having different dispersions. However, in recent years, the size of image pickup apparatuses has been reduced, and it has been required to increase the resolution of sensors used in image pickup elements and to reduce the size of imaging optical systems. It is difficult to sufficiently suppress chromatic aberration by the configuration of the imaging optical system. .

  Chromatic aberration is roughly classified into lateral chromatic aberration (magnification chromatic aberration) and longitudinal chromatic aberration (axial chromatic aberration). Lateral chromatic aberration is a phenomenon in which the image formation position shifts in the direction along the image plane depending on the wavelength. Longitudinal chromatic aberration is a phenomenon in which the imaging position shifts in the direction along the optical axis depending on the wavelength. As disclosed in Patent Document 1, lateral chromatic aberration can be corrected by performing a geometric transformation process on each color plane of R (red), G (green), and B (blue). On the other hand, longitudinal chromatic aberration cannot be corrected by the geometric conversion process.

  In Patent Document 2, a region in which a G (green) plane is saturated is searched using a characteristic in which color blur is mainly generated in the vicinity of a whiteout (a saturated region of a preset signal), and a region around the region is searched. A method of calculating a correction amount by integrating the pixels is disclosed. Patent Document 3 discloses a method of determining a region where a color plane pixel value in a color image monotonously increases or decreases as a color blur generation region and removing the color blur.

US Pat. No. 6,724,702 JP 2007-133582 A JP 2009-268033 A

  However, in the method disclosed in Patent Document 2, color blur that causes discomfort to the observer also occurs in a region that is not overexposed, and thus correction of color blur is insufficient. Further, according to the method disclosed in Patent Document 3, it is possible to correct the color blur of the region that is not overexposed. However, since color blur occurs due to aberrations in the photographing optical system, there are directions in which color blur occurs and does not occur in a subject in a photographing optical system having asymmetric aberrations such as coma aberration. When the color of the subject is similar to the color blur, there is a possibility that the subject color is erroneously determined as a color blur even in a direction where the color blur does not occur, and the original color of the subject is removed.

  Further, the performance (optical performance) of the photographing optical system varies (manufacturing variation occurs) due to decentration aberration caused by a manufacturing error of a lens constituting the photographing optical system or a manufacturing error of a lens barrel holding the lens. Conventionally, in order to reduce the influence of such manufacturing variations, correction (eccentric adjustment) is performed so as to cancel out decentration aberrations caused by manufacturing errors by decentering some elements of the photographic optical system and image sensor. I was going. However, even when the eccentricity adjustment is performed, it is difficult to effectively correct the variation in optical performance. For this reason, even in a photographing optical system that is rotationally symmetric with respect to the optical axis, color blur may occur in a rotationally asymmetric manner with respect to the optical axis due to manufacturing errors.

  Therefore, the present invention provides an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can reduce the influence of manufacturing errors and effectively reduce color bleeding in a color image.

  An image processing apparatus according to one aspect of the present invention includes a signal level of a color plane corresponding to at least one color filter of an image obtained from an imaging device including a plurality of color filters, or luminance generated from the image. Detection means for detecting a first region in which the signal level of the plane monotonously increases or decreases monotonously in a first direction in a predetermined section, first optical information relating to color blur of the imaging optical system, and the imaging Determination means for determining a second region in which the color blur is generated based on second optical information relating to variations in chromatic aberration that is asymmetric with respect to the optical axis of the optical system and information relating to the first region; Correction means for correcting the image so as to reduce the color blur.

  An image pickup apparatus according to another aspect of the present invention provides an image pickup device that photoelectrically converts an optical image formed through a photographing optical system, and at least one of images obtained from the image pickup device including a plurality of color filters. A first region in which the signal level of the color plane corresponding to the color filter or the signal level of the luminance plane generated from the image is monotonously increased or monotonously decreased in the first direction in a predetermined section is detected. Based on detection means, first optical information relating to color blur of the photographing optical system, second optical information relating to chromatic aberration variation asymmetric with respect to the optical axis of the photographing optical system, and information relating to the first region. And determining means for determining the second area where the color blur occurs, and correcting means for correcting the image so as to reduce the color blur.

  An image processing method according to another aspect of the present invention is generated from a signal level of a color plane corresponding to at least one color filter of an image obtained from an image sensor including a plurality of color filters, or from the image. Detecting a first region in which a signal level of the luminance plane monotonously increases or decreases monotonously in a first direction in a predetermined section; first optical information relating to color blur of the photographing optical system; and the photographing Determining a second region in which the color blur has occurred based on second optical information regarding variations in chromatic aberration that is asymmetric with respect to the optical axis of the optical system and information regarding the first region; Modifying the image to reduce color blur.

  An image processing program according to another aspect of the present invention is generated from a signal level of a color plane corresponding to at least one color filter of an image obtained from an imaging device including a plurality of color filters, or from the image. Detecting a first region in which a signal level of the luminance plane monotonously increases or decreases monotonously in a first direction in a predetermined section; first optical information relating to color blur of the photographing optical system; and the photographing Determining a second region in which the color blur has occurred based on second optical information regarding variations in chromatic aberration that is asymmetric with respect to the optical axis of the optical system and information regarding the first region; Modifying the image to reduce color bleed, and causing the computer to execute.

  A storage medium according to another aspect of the present invention stores the image processing program.

  Other objects and features of the invention are described in the following embodiments.

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can reduce the influence of manufacturing errors and can effectively reduce color blur in a color image. Can do.

It is a block diagram of the image processing apparatus in this embodiment. It is a flowchart which shows the image processing method in this embodiment. It is a flowchart which shows the monotonous increase / decrease determination process in this embodiment. In this embodiment, it is a figure which shows an example in case the pixel area of the monotonous increase / decrease determination object has a monotonous increase / decrease characteristic. In this embodiment, it is a figure which shows an example in case the pixel area of the monotonous increase / decrease determination object does not have a monotonous increase / decrease characteristic. In this embodiment, it is a figure which shows the pixel area for the monotone increase / decrease determination centering on an attention pixel. In this embodiment, it is a figure which shows the pixel area for the monotone increase / decrease determination which used the attention pixel as the end point. In this embodiment, it is a figure which shows the area | region of 3x3 pixel in an image. In this embodiment, it is a figure which shows the result of having applied the low-pass filter with respect to each pixel of the area | region of 3x3 pixel in an image. In this embodiment, it is a figure which shows the example of the change of an input signal at the time of applying a low-pass filter with respect to each pixel. In this embodiment, it is a figure which shows the example of the monotone increase / decrease detection of the imaging optical system which has a symmetrical aberration. In this embodiment, it is a figure which shows the example of the monotone increase / decrease detection of the imaging optical system which has an asymmetrical aberration. It is explanatory drawing of the optical information in this embodiment. In this embodiment, it is a figure which shows the example which determines a color bleeding area | region based on optical information and a monotonous increase / decrease detection result. It is explanatory drawing of the two-dimensional optical information in this embodiment. In this embodiment, it is a figure which shows the color blur determination process using optical information. It is explanatory drawing of the calculation method of the optical information in this embodiment. In this embodiment, it is a figure which shows the typical pixel value change of a blue blur. In this embodiment, it is a characteristic view of the nonlinear conversion with respect to the pixel value of B plane. It is a figure which shows the chromaticity coordinate ab surface in this embodiment. In this embodiment, it is a figure which shows the area | region of 3 * 3 pixel in B plane by a monotone increase / decrease determination result. 1 is a configuration diagram of an imaging apparatus in Embodiment 1. FIG. FIG. 6 is a configuration diagram of an image processing system in Embodiment 2. It is a flowchart which shows the image processing method as a modification in this embodiment. It is explanatory drawing of the color bleeding generation | occurrence | production direction by the manufacturing error of an imaging optical system. In this embodiment, it is a figure which shows the color blur determination process using the monotone increase / decrease detection result, optical information, and optical performance variation information. In this embodiment, it is explanatory drawing which added color bleed intensity | strength information as optical performance variation information. It is explanatory drawing of the color blur generation | occurrence | production direction by decentration aberration.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, an image processing apparatus and an image processing method in this embodiment will be described with reference to FIGS. 1A and 1B. The image processing apparatus and the image processing method described here are appropriately used in each embodiment described later. FIG. 1A is a block diagram of an image processing apparatus 100 in the present embodiment. FIG. 1B is a flowchart of an image processing method (image processing program) in the present embodiment. Each step in FIG. 1B is executed based on a command from the image processing apparatus 100, that is, by each unit of the image processing apparatus 100.

  As shown in FIG. 1A, the image processing apparatus 100 includes an input unit 101, a detection unit 102, a determination unit 103, a correction unit 104, an output unit 105, an acquisition unit 106, and a storage unit 107. First, in step S1 of FIG. 1B, the image processing apparatus 100 (input unit 101) acquires a captured image as an input image. The input image is a digital image (photographed image) obtained by receiving light with an image pickup device via a photographing optical system, and due to aberrations (such as axial chromatic aberration) of the photographing optical system including a lens and various optical filters. It has deteriorated. The photographing optical system can be configured using not only a lens but also a mirror (reflection surface) having a curvature.

  The color component of the input image has information on RGB color components, for example. As the color component, other commonly used color spaces such as brightness, hue, and saturation expressed in LCH, luminance expressed in YCbCr, and color difference signals can be selected and used. As other color spaces, XYZ, Lab, Yuv, and JCh can be used. Color temperature may also be used. The captured image can be acquired by connecting the imaging device and the image processing device by wire or wirelessly. The captured image can also be acquired via a storage medium.

  Subsequently, in step S2, the image processing apparatus 100 (detection means 102) is a region where the pixel value (signal level) of any one of the color planes constituting the color image is monotonously increasing or monotonically decreasing. Is detected. Then, the detection unit 102 tentatively determines the detected area as a color blur generation area (monotonic increase / decrease detection step). Several methods are conceivable as a method of determining the color blur generation region in the monotonous increase / decrease detection step. In the present embodiment, the determination is made based on the change characteristic of the pixel value that blurs in a certain pixel section (predetermined section).

  Since color blur occurs when the imaging position shifts in the direction along the optical axis depending on the wavelength, for example, in the case of blue blur, the blue plane (B plane) is out of focus (out of focus). . The color blur due to the defocusing exists over a certain pixel section, and the pixel value change of the color blur at that time has a characteristic of gradually decreasing from the highlight portion to the shadow portion of the image. Therefore, when the pixel value change of color blur in a certain pixel section has a monotonous increase or monotonic decrease characteristic, there is a possibility that color blur has occurred, so that it is temporarily determined as a color blur generation region.

  As described above, in the present embodiment, the image sensor includes a plurality of color filters. The detecting unit 102 monotonically increases the signal level (pixel value) of the color plane corresponding to at least one color filter of the image obtained from the image sensor in any direction (first direction) in a predetermined section. Alternatively, a monotonically decreasing area (first area) is detected. However, the present embodiment is not limited to this, and instead of the signal level of the color plane, for example, the signal level of the luminance plane (Y plane) generated from the image obtained from the image sensor increases monotonously or monotonically. You may comprise so that the area | region which is reducing may be detected.

  Actually, when the photographing optical system has an asymmetrical (rotationally asymmetric) aberration with respect to the optical axis (center axis) such as coma aberration, the color blur generated by the photographing optical system is the direction in which the subject is generated. There is a direction that does not occur. For this reason, if the color blur generation area is determined only from monotonous increase / decrease, if the subject has a color similar to the color blur, the subject color may be erroneously determined to be color blur. Accordingly, it is not possible to correctly determine the color blur generation area only by monotonic increase / decrease detection.

  In the present embodiment, the region for detecting monotonic increase or monotonic decrease is a horizontal, vertical or diagonal pixel section centered on the target pixel of the color image, a horizontal direction with the target pixel as an end point, It includes either a vertical or diagonal pixel section.

  Detection of monotonic increase / decrease is performed by first calculating the pixel value gradient of the color plane. When the input image is composed of, for example, three color planes of the G plane, the B plane, and the R plane, the B plane is set as a color blur removal target, and the G plane is used as a reference plane. The respective luminance gradients Blea and Glea with respect to the B plane and the G plane are calculated as represented by the following expression (1).

  In Expression (1), G (x + 1, y) and B (x + 1, y) are pixel values on the right side of the target pixel in the G plane and B plane, respectively. G (x−1, y) and B (x−1, y) are pixel values on the left side of the target pixel in the G plane and the B plane, respectively. G (x, y + 1) and B (x, y + 1) are pixel values below the pixel of interest in the G plane and B plane, respectively. G (x, y-1) and B (x, y-1) are pixel values above and below the pixel of interest in the G plane and B plane, respectively. Although this embodiment will be described with three color planes, processing can be performed with any color plane depending on the number of color planes constituting the image.

  Next, with reference to FIGS. 2 to 4, a method for determining a color blur occurrence area in the monotonous increase / decrease detection process will be described. FIG. 2 is a flowchart showing a monotonous increase / decrease determination process (monotonic increase / decrease detection step: step S2) in the present embodiment. Each step in FIG. 2 is mainly executed by the image processing apparatus 100 (detection unit 102). First, in step S1520, the image processing apparatus 100 determines whether the ISO sensitivity is high (whether the ISO sensitivity is higher than a predetermined sensitivity). If the ISO sensitivity is high, the process proceeds to step S1521. On the other hand, if the ISO sensitivity is not high, the process proceeds to step S1522.

  In step S1522, the image processing apparatus 100 analyzes a change in the pixel value of the input signal with respect to vertical, horizontal, and diagonal pixel sections (predetermined sections) when the pixel of interest of the input image is the center. In step S1523, the image processing apparatus 100 determines (detects) whether or not the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic. As a result of the determination, if the change in the pixel value of the input signal in the pixel section does not have a monotonous increase / decrease characteristic, in step S1524, the image processing apparatus 100 displays the vertical / horizontal when the target pixel is the end point.・ A pixel value change of an input signal is analyzed for an oblique pixel section. In step S1525, the image processing apparatus 100 determines whether the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic.

  FIG. 3 is a diagram illustrating an example of a case where a pixel section subject to monotonous increase / decrease determination has a monotonous increase / decrease characteristic, where the vertical axis indicates the pixel value and the horizontal axis indicates the distance. FIG. 4 is a diagram illustrating an example of a case where a pixel section for which monotonous increase / decrease is determined does not have a monotonous increase / decrease characteristic, where the vertical axis indicates the pixel value and the horizontal axis indicates the distance. The image processing apparatus 100 performs monotonous increase / decrease determination on an input signal having a change in pixel value as shown in FIGS. The white squares shown in FIGS. 3 and 4 are the target pixels.

  As shown in FIG. 3, an image in which a change in the pixel value of the input signal has a monotonous increase / decrease characteristic has a monotonous increase / decrease characteristic in a pixel section where the monotonous increase / decrease determination is performed. Therefore, as a result of the analysis in step S1522 or S1524, in step S1523 or S1525, the image processing apparatus 100 determines that the pixel section has a monotonous increase / decrease characteristic. On the other hand, it is determined that the image having the change characteristic of the pixel value of the input signal as shown in FIG. 4 does not have the monotonous increase / decrease characteristic.

  If the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic (monotonic increase characteristic or monotonic decrease characteristic), the image processing apparatus 100 sets a monotonous increase / decrease flag in step S1527. On the other hand, if the change in the pixel value of the input signal in the pixel section does not have a monotonous increase / decrease characteristic (monotonic increase characteristic or monotonic decrease characteristic), the image processing apparatus 100 sets a monotonous increase / decrease flag in step S1526. Absent.

  The monotonous increase / decrease determination described above is applied to each pixel of the B plane. As a result, each pixel is associated with “1” if the monotonous increase / decrease flag is set, and “0” if the monotone increase / decrease flag is not set, and is generated and held as a monotone increase / decrease determination result plane. Used in determination (step S5 in FIG. 1B). Details of the method of using the monotonous increase / decrease determination result plane will be described later.

  Next, with reference to FIG. 5 and FIG. 6, a method for setting a pixel section for performing monotonous increase / decrease determination on the target pixel will be described. FIGS. 5A to 5D are diagrams illustrating a monotonic increase / decrease pixel section centered on a target pixel. FIGS. 6A to 6H are diagrams showing a monotone increase / decrease pixel section with the target pixel as an end point. Of the method of setting the pixel section centered on the target pixel and the method of setting the pixel section centered on the target pixel, the method of setting the pixel section centered on the target pixel is shown in FIGS. A method of setting the horizontal direction and the vertical direction around the target pixel as in FIG.

  Further, as a method for setting a pixel section centered on the target pixel, a method of setting the target pixel in an oblique direction as shown in FIGS. 5C and 5D can be considered. In other words, the isotropic property can be imparted to the diagonal direction by setting a pixel section having a distance similar to that in the horizontal direction or the vertical direction. In this case, the angle in the oblique direction is limited to a setting of 45 degrees with respect to the horizontal direction or the vertical direction as shown in FIGS. However, the present embodiment is not limited to this, and an arbitrary angle other than 45 degrees can be set. The distance d of the pixel section at that time is calculated using the following equation (2).

In Expression (2), x represents the horizontal direction, and θ represents the angle from the horizontal.

  On the other hand, the color blur around the highlight area and the color blur around the shadow area in the image are affected by saturation and noise, respectively. There is a case where it cannot be judged correctly because it does not have an increase / decrease characteristic. In that case, as shown in FIG. 6, a method of setting a pixel section with the target pixel as an end point is effective. When the monotonous increase / decrease determination is performed by the method as described above and there is a pixel section having the monotonous increase / decrease characteristics in any one of the pixel sections shown in FIGS. 5 and 6, the target pixel has the monotonous increase / decrease characteristics. It is determined to be a pixel.

  In the present embodiment, the pixel value of the input signal is used as a target for monotonic increase / decrease determination, but luminance gradient may be used. In that case, it is determined that the color blur is caused when the change in the luminance gradient is reversed only once in a certain pixel section. It is effective to set the appropriate value of the number of pixels in the pixel section to the minimum color blur width among the color blurs generated under certain shooting conditions of the imaging apparatus.

  The color blur width varies depending on the imaging conditions of the imaging apparatus (aperture value, focal length, focusing accuracy, focus position in the image plane, coordinates on the image sensor, etc.). For this reason, it is possible to detect the color blur of the minimum width by matching the appropriate value of the number of pixels in the pixel section with the color blur of the minimum width of various color blurs generated under various shooting conditions. The maximum width color blur can be detected even by using a pixel section adapted for the minimum width color blur.

  By performing monotonous increase / decrease determination as described above, it is possible to extract a color blur to be detected. However, depending on the shooting conditions such as high ISO sensitivity, the S / N ratio may decrease due to noise included in the input signal, and as a result, the color blur may not have a monotonous increase / decrease characteristic. In that case, it is effective to perform a filtering process with a digital filter on the input signal in step S1521 in FIG. In the present embodiment, a case where the digital filter is a low-pass filter will be described, but the present invention is not limited to this.

  There are several methods for applying a low-pass filter to the input signal. For example, a case will be described in which a [1 2 1] low-pass filter in which the weighting coefficient of the pixel of interest is twice as large as that of adjacent pixels is applied. FIG. 7 is a diagram showing an area of 3 × 3 pixels in an image. As shown in FIG. 7, when p is a target pixel in a certain region of 3 × 3 pixels in an image, first, a low-pass filter of [1 2 1] is applied in the horizontal direction. At this time, the target pixel is expressed as the following Expression (3).

P = (d · 1 + p · 2 + e · 1) / 4 (3)
When the adjacent pixels are calculated in the same manner, they are as shown in FIG. FIG. 8 is a diagram illustrating a result of applying a low-pass filter to each pixel in a 3 × 3 pixel region in an image. Subsequently, when the low-pass filter of [1 2 1] is applied in the vertical direction, the target pixel is expressed as the following Expression (4).

PP = (B · 1 + P · 2 + G · 1) / 4 (4)
With reference to FIG. 9, an example of a change in the input signal when the low-pass filter is applied will be described. FIG. 9 is a diagram illustrating an example of a change in an input signal when a low pass filter is applied to each pixel. In FIG. 9, the horizontal axis indicates the distance (cross section on the image, that is, the pixel section from the target pixel), and the vertical axis indicates the pixel value of the plane.

  The solid line in FIG. 9 indicates the case where the low-pass filter is not applied, the fine broken line indicates the case where the low-pass filter [1 2 1] is applied, and the coarse broken line indicates the case where the low-pass filter [1 4 6 4 1] is applied. Yes. Here, [1 4 6 4 1] means that a low-pass filter is applied by applying a weighting factor to a pixel adjacent to the target pixel and a pixel separated by another pixel. By smoothing the input signal by applying the low-pass filter in this way, it is possible to highlight the monotonous increase / decrease characteristics inherent in color fringing. In this embodiment, the low-pass filter is applied in the order of the horizontal direction and the vertical direction, but the present invention is not limited to this. The low-pass filter may be applied in the reverse order, that is, in the order of the vertical direction and the horizontal direction. It is also possible to calculate a two-dimensional low-pass filter coefficient and apply the low-pass filter simultaneously in the horizontal direction and the vertical direction.

  Subsequently, in step S3 of FIG. 1B, the image processing apparatus 100 (acquisition unit 106) acquires optical information (first optical information) related to color blur of the photographing optical system. The first optical information is information related to the optical design value of the photographing optical system. In step S4, the acquisition unit 106 acquires optical information (second optical information) regarding variations in chromatic aberration that is asymmetric (rotationally asymmetric) with respect to the optical axis (center axis) of the imaging optical system. The second optical information is information (optical performance variation information) regarding manufacturing errors of the photographing optical system. As will be described later, the optical performance variation information reflects the manufacturing error (eccentric aberration) of the photographing optical system in the first optical information acquired in step S3, so that the color bleed generation region can be determined with higher accuracy. Used to determine. In step S4, the optical performance variation information generated by measuring the actual color blur generation direction including the decentration aberration at the time of manufacturing the photographing optical system is acquired.

  The first optical information and the second optical information are stored in advance in the storage unit 107 of the image processing apparatus 100, a storage unit 208 of the imaging apparatus 200 described later, and the optical information stored is called up and acquired. be able to. The optical performance variation information can also be acquired by connecting the imaging device 200 and the image processing device 100 by wire or wirelessly or via a storage medium. In step S4, for example, as disclosed in Japanese Patent Application Laid-Open No. 2012-195927, the amount of decentration aberration determined based on the captured image may be acquired as optical performance variation information.

Subsequently, in step S5, the image processing apparatus 100 (determination unit 103) determines that the region (first region) temporarily determined as the color blur generation region by the detection unit 102 in step S2 is the color blur generation region (second region). (Region of color blur determination step). At this time, the determination unit 103 uses the optical information (first optical information) acquired in step S3 and the optical performance variation information (second optical information) acquired in step S4.
That is, the determination unit 103 is based on the first optical information relating to the color blur of the photographing optical system, the second optical information relating to the chromatic aberration variation asymmetric with respect to the optical axis of the photographing optical system, and the information relating to the first region. Then, a region (second region) in which color blur is generated is determined.

  For example, the determination unit 103 acquires third optical information (information in which a manufacturing error is reflected in the optical design value) related to the color blur of the photographing optical system based on the first optical information and the second optical information. To do. Then, the determination unit 103 determines the second area based on the third optical information and the information related to the first area. Here, the third optical information is information regarding the second direction in which color blur occurs in the photographing optical system (optical information in the color blur generation direction). The third optical information is information regarding the second direction obtained by reflecting optical performance variation information of the photographing optical system (variation of chromatic aberration due to manufacturing errors of the photographing optical system) in the first optical information. is there. In the present embodiment, the color blur generation area includes optical information (first optical information) of the color blur generation direction of the photographing optical system that captured the input image, and optical performance variation information (second optical information) of the photographing optical system. ).

  In the present embodiment, the determination unit 103 compares the first direction (the direction in which the signal level monotonously increases or decreases monotonously) with the second direction (the direction in which the color blur of the photographing optical system occurs). 2 areas are determined. For example, when the first direction and the second direction match each other (or when they are within a predetermined range that is evaluated as substantially matching), the determination unit 103 detects the first region (detected in step S2). It is determined that the (region) is the second region (region where color bleeding is generated).

  Here, with reference to FIG. 10 and FIG. 11, an adverse effect in the case where the color blur area is determined only from the monotonous increase / decrease detection process will be described. An image obtained by imaging a subject with an imaging device such as a digital camera includes a blur component as an image degradation component caused by spherical aberration, coma aberration, field curvature, astigmatism, etc. of the photographing optical system. . Such a blur component is generated when a light beam emitted from one point of a subject to be collected again at one point on the imaging surface when an aberration is not caused and there is no influence of diffraction is formed by forming an image with a certain spread.

  The blur component is optically represented by a point spread function (PSF), and is different from blur due to a focus shift. In addition, color blur in a color image can also be said to be a difference in blurring for each wavelength of light with respect to axial chromatic aberration, spherical spherical aberration, and color coma aberration of the optical system. Further, the lateral color misregistration can also be referred to as a positional misalignment or a phase misalignment due to a difference in imaging magnification for each wavelength of light when the lateral chromatic aberration is caused by the optical system.

  An optical transfer function (OTF) obtained by Fourier-transforming the point spread function (PSF) is frequency component information of aberration and is represented by a complex number. The absolute value of the optical transfer function (OTF), that is, the amplitude component, is referred to as MTF (Modulation Transfer Function), and the phase component is referred to as PTF (Phase Transfer Function). MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively. Here, the phase component is expressed as the following equation (5) with the phase angle as the phase angle.

PTF = tan ⁻¹ (Im (OTF) / Re (OTF)) (5)
In formula (5), Re (OTF) and Im (OTF) are the real part and the imaginary part of OTF, respectively.

  As described above, since the optical transfer function (OTF) of the optical system deteriorates the amplitude component and the phase component of the image, each point of the subject is asymmetrically blurred with respect to the optical axis like coma aberration. It becomes a state.

  FIG. 10 is a diagram for explaining a case where monotonous increase / decrease detection is performed on an image captured in a state of optical characteristics in which each point of a subject is symmetric (rotationally symmetric) with respect to the optical axis such as spherical aberration. 10A shows a pixel value section of the subject, FIG. 10B shows a PSF section of the photographing optical system, and FIG. 10C shows a pixel value section of the subject photographed by the photographing optical system. As shown in FIG. 10B, when the photographing optical system has an aberration characteristic that is symmetric (rotationally symmetric) with respect to the optical axis and the PSF is symmetric, the edge cross sections on both sides of the subject are both degraded images. Photographed as shown by the solid line in FIG. In FIG. 10C, for comparison, the luminance cross section of the subject shown in FIG. If the white square in FIG. 10C is used as the target pixel and the monotone increase / decrease determination is performed, both edges are determined as the monotone increase / decrease area. Further, since color blur actually occurs, the color blur generation region is correctly determined.

  FIG. 11 is a diagram illustrating a case where monotonous increase / decrease detection is performed on an image captured with optical characteristics in which each point of a subject is asymmetric (rotationally asymmetric) with respect to the optical axis such as coma. 11A is a pixel value cross section of the subject, FIG. 11B is a PSF cross section of the photographing optical system, and FIG. 11C is a pixel cross section of the subject photographed by the photographing optical system. As shown in FIG. 11B, when the photographing optical system has an asymmetric aberration characteristic with respect to the optical axis and the PSF is asymmetric, the edge cross section of the subject is one side particularly in the optical characteristic that greatly deteriorates only on one side edge. Only an image that is greatly degraded is taken as shown by the solid line in FIG. When monotonous increase / decrease detection is performed on such a photographed image, both edges are determined as monotonous increase / decrease areas, but since the color blur actually generated by the photographing optical system is only on one side, the original color of the subject is blurred. As a result, an adverse effect occurs. If the color blur correction is performed only by monotonous increase / decrease, the original color of the subject is removed, which may result in an unnatural image. If you try to avoid this, the correction will be weak.

  Therefore, in the present embodiment, the image processing apparatus 100 includes optical information (first optical information and second optical information) regarding the color blur generation direction of the photographing optical system as shown in FIGS. 10B and 11B. Information). When the monotonous increase / decrease direction of the monotonic increase / decrease detection matches the color blur generation direction, the image processing apparatus 100 determines that the color blur generation region is present. Here, the term “matching” means not only the case of exact matching but also the case of substantial matching.

  The optical information will be described with reference to FIG. FIG. 12 is an explanatory diagram of optical information in the present embodiment, and shows a PSF cross section and optical information of the photographing optical system. FIG. 12A shows optical information when the PSF of the photographing optical system shown in FIG. 10B is symmetric with respect to the optical axis. FIG. 12B shows optical information when the PSF of the imaging optical system shown in FIG. 11B is asymmetric with respect to the optical axis. For example, it has 1 in the direction in which color blur occurs due to the aberration of the photographing optical system, and 0 if it does not occur. In FIG. 12A, since the color blur occurs symmetrically on both sides, the optical information has a value of 1 on both sides. In FIG. 12B, color blur is generated asymmetrically, and no color blur is generated at the left edge, and a value of 1 is generated because color blur is generated at the right edge.

  FIG. 13 is a diagram illustrating an example of determining a color blur area based on optical information and a monotonous increase / decrease detection result. As shown in FIG. 13 (a), for example, when monotonic decrease is detected by monotone increase / decrease detection, the optical information in the monotonic decrease direction is referred to. To do. Further, as shown in FIG. 13B, the optical information in the monotonously decreasing direction is referred to, and if it has 0, it is determined that the color of the subject. As described above, by having information on the direction in which chromatic aberration occurs and the direction in which chromatic aberration does not occur as the aberration of the photographing optical system, it is possible to correctly determine the color and color blur of the subject, thereby improving the accuracy of color blur correction. It becomes possible. Although the direction of occurrence of color blur is 1 and the direction of non-occurrence is 0, it is only necessary to know the direction of occurrence, and it is not necessary to be 0 and 1. Since the aberration of the photographic optical system changes depending on the image height, it is possible to correct with high accuracy by having optical information at a plurality of image heights. For image heights that do not have correction values, interpolation may be generated from nearby correction values. In addition, since the aberration of the photographing optical system changes depending on photographing conditions (focal length of the photographing optical system, subject distance, aperture value (Fno)), the optical information (first optical information and second optical information) for each photographing condition. By storing or acquiring (information), correction can be performed with high accuracy.

  10 and 11, the optical information has been described as one-dimensional information. However, when the captured image is two-dimensional array data, the optical information may be held as two-dimensional data. FIG. 14 is an explanatory diagram when the optical information is two-dimensional. FIG. 14C shows an example in which optical information is held as data in eight directions of up / down / left / right, 45 degrees oblique and 135 degrees on the screen. In FIG. 14, color coma in the photographing optical system is generated in the upward direction of the screen, and 1 optical information is held in the generation direction. 0 is maintained in the direction where no color blur occurs. Monotonic increase / decrease detection is performed in the above eight directions, and it is determined that the color is blurred when 1 of the optical information is held in the direction in which the monotonic decrease is detected. Even if a monotonic decrease is detected, if the optical information 1 is not held in the detected direction, it is determined as the color of the subject.

  As described above, the image processing apparatus 100 holds or acquires the optical information (first optical information and second optical information) regarding the direction in which the color blur of the photographing optical system occurs. Preferably, the image processing apparatus 100 includes a storage unit 107 that stores the first optical information and the second optical information. More preferably, the storage unit 107 stores the first optical information and the second optical information for each photographing condition. The photographing condition includes at least one of a focal length, a subject distance, and an aperture value of the photographing optical system. Note that the storage unit 107 may be provided outside the image processing apparatus 100. For example, each optical information can be stored in the storage unit 208 described later.

  Then, the image processing apparatus 100 refers to the detection direction, the first optical information (optical information), and the second optical information (optical performance variation information) at the time of monotonic increase / decrease detection, thereby blurring the color with high accuracy. The area can be determined. In the present embodiment, the monotonous increase / decrease determination has been described in the monotonic decrease direction. However, the color blur determination may be performed using the monotone increase direction. In addition, the monotonous increase / decrease determination and the correction value of the optical information have been described in eight directions, but are not limited to eight directions. In order to improve accuracy, the number of detection directions may be increased. Alternatively, the number of detection directions may be reduced in order to reduce the amount of data related to optical information.

  Incidentally, as described above, in the photographing optical system, decentration aberrations occur due to manufacturing errors and decentration adjustment, and color blur occurs asymmetrically (rotationally asymmetrically) with respect to the optical axis. For this reason, correction by color blur using the monotonous increase / decrease determination result and optical information (first optical information) regarding the color blur generation direction of the design value of the photographing optical system may not be corrected correctly.

  Here, with reference to FIG. 27, the color blur generation direction due to decentration aberration will be described. FIG. 27 is an explanatory diagram of the direction of occurrence of color blur due to decentration aberration. FIG. 27A is a schematic diagram when the photographing optical system has coma aberration in a state where no eccentricity occurs. When decentration does not occur, the aberration occurs symmetrically with respect to the optical axis. For this reason, as the optical information, it is only necessary to have information regarding the color blur occurrence direction at each image height in one direction passing through the optical axis of the photographing screen. Since aberration occurs symmetrically with respect to the optical axis, using the calculated optical information in one direction after rotational interpolation can reduce the amount of data compared to having correction data for the entire photographing screen. It is also possible to have correction data for the entire shooting screen in order to shorten the interpolation calculation time.

  FIG. 27B is a schematic diagram of decentration coma aberration when one optical element of the photographing optical system is decentered. Eccentric coma occurs asymmetrically with respect to the optical axis, and coma occurs in the decentering direction of the optical element. Actually, as shown in FIG. 27A, decentration aberration is added to the aberration that the photographing optical system has as a design value, so that color blur occurs asymmetrically on the entire screen. For this reason, when color blur correction is performed based on the optical information (first optical information) of the design value of the photographing optical system and the monotonous increase / decrease determination result, the design value and the aberration of the actually produced photographing optical system are calculated. Due to the difference between the two, correct correction may not be possible.

  Here, with reference to FIG. 24, an adverse effect in the case where decentration aberration occurs due to a manufacturing error of the photographing optical system will be described. FIG. 24 is an explanatory diagram of the direction of occurrence of color blur due to a manufacturing error of the photographing optical system. FIG. 24A is an explanatory diagram of a color blur correction process when no decentering aberration is generated in the photographing optical system in the photographing optical system having chromatic coma as a design value. When determining the color blur generation area for the target pixel 1 surrounded by the square in FIG. 24A, first, monotonously decrease in the eight directions of up and down, left and right, 45 degrees oblique, and 135 degrees oblique with the target pixel 1 as the center. Make a decision. Since the high-luminance subject is located at the lower left of the target pixel 1, the directions in which the monotonic decrease of the target pixel 1 is determined are determined to be upper, upper right, right, and lower right. At this time, since the color blur generation direction of the photographing optical system coincides with upper, upper right, and right (first optical information), the target pixel 1 is determined to be a color blur generation region.

  FIG. 24B shows a high luminance that does not occur with the design value of FIG. 24A due to a manufacturing error that causes an asymmetric aberration with respect to the optical axis such as decentering coma and decentered field curvature in the photographing optical system. An example is shown in which color blur has also occurred on the lower side of the subject. Here, a case where the color blur generation area is determined for the target pixel 2 in FIG. Since the high-intensity subject is located above the target pixel 2, the direction in which the monotonic decrease of the target pixel 2 is determined is the lower left, the lower, and the lower right. On the other hand, the optical information (first optical information) regarding the color blur generation direction generated based on the design value of the photographing optical system has information that no color blur occurs in this direction. For this reason, if the determination is performed based only on the first optical information, the target pixel 2 is not determined to be a color blur generation region, and there is a problem that the color blur correction process is not performed.

  Therefore, in the present embodiment, in order to perform the color blur correction process with higher accuracy in consideration of the manufacturing error (eccentric aberration) of the photographing optical system, the monotonous increase / decrease determination result and the optical information (first optical information) are included. In addition, optical performance variation information (second optical information) due to manufacturing errors is used.

  Here, with reference to FIG. 25, the color blur correction determination in the present embodiment will be described. FIG. 25 is a diagram showing color blur determination processing using the monotonous increase / decrease detection result, optical information, and optical performance variation information.

  FIG. 25A shows the optical design value of the photographing optical system, that is, optical information (first optical information) in a state where there is no manufacturing error in the photographing optical system. FIG. 25B shows a case where decentration aberration occurs in the downward direction of the high-luminance subject due to a manufacturing error of the photographing optical system, and color blur occurs in a direction that does not originally occur in the optical design value. In this case, the image processing apparatus 100 holds, for example, optical performance variation information (eccentric aberration occurrence direction, that is, second optical information) where 1 is a direction in which color blur occurs due to decentration aberration and 0 is a direction in which color blur does not occur. Or get. Then, using the second optical information shown in FIG. 25 (b), the first optical information shown in FIG. 25 (a) is corrected to create color blur generation direction information (third optical information). To do. As shown in FIG. 25B, the third optical information is information obtained by adding the first optical information (optical information) and the second optical information (decentration aberration generation direction). In the form, it is 1 in all directions, and indicates that color bleeding occurs in all directions. Finally, the third optical information and the monotonous increase / decrease determination result are compared to determine the color blur generation region. Information about the color blur generation direction after correction indicates that color blur occurs in all directions. For this reason, the monotonic decrease determination result and the color blur generation direction of the optical information coincide with each other even under the high-brightness subject, so that it is determined as a color blur correction pixel.

  As shown in FIG. 25C, the image processing apparatus 100 holds, as optical performance variation information, the difference between the color blur direction of the actual photographing optical system including the decentration aberration component and the optical information of the design value. Or you may acquire. The optical performance variation information is not limited to these forms, and includes actual decentration aberrations of the photographing optical system based on optical information (first optical information) that is a design value of the photographing optical system. Any information that can be corrected in the direction of occurrence of color blur (reflecting a manufacturing error) may be used.

  As described above, in the present embodiment, in the color blur correction process of a color image, the monotonic increase or monotonic decrease determination of the pixel value of each color plane, the optical information regarding the color blur generation direction of the photographing optical system, and the optical of the photographing optical system Performance variation information is used. As a result, it is possible to reduce the adverse effect of removing the original color of the subject photographed by the photographing optical system having a manufacturing error, and to effectively correct the color blur.

  By the way, since the eccentric aberration occurs asymmetrically with respect to the optical axis, the optical performance variation information needs to have different values in the upper right direction and the lower left direction of the captured image, for example. However, the direction of occurrence of color blur due to decentration is defined as, for example, the center of gravity direction of the entire photographed image, and processing is performed at high speed by processing all pixels uniformly in the photographed image with information on the direction of occurrence of decentration aberration. May be. Moreover, although the variation information of the optical performance due to decentration aberration has been described as data in eight directions, it is not limited to this form. For example, as described above, the color blur generation direction due to eccentricity may be the center of gravity direction of the entire screen, and one value indicating the direction may be the optical performance variation information.

  Further, the image processing apparatus 100 may be configured to hold or acquire information regarding the color blur intensity (color blur intensity information) as optical information in addition to the color blur generation direction. In this case, correction can be performed with higher accuracy. As shown in FIG. 14B, in general, the amount of color blur due to the aberration of the photographing optical system differs depending on the direction. Therefore, as shown in FIG. 14D, color blur intensity information regarding the amount of color blur may be added in the direction in which color blur occurs.

  Next, with reference to FIG. 15, the color blur determination when the color blur intensity information is used will be described. FIG. 15 is a diagram illustrating a color blur determination process using optical information, and illustrates an example in which the optical information has only the color blur generation direction. When determining the color blur generation region for the pixel of interest 1 surrounded by the square in the figure, first, monotonic decrease determination is performed in eight directions, up and down, left and right, 45 degrees oblique, and 135 degrees oblique, with the pixel of interest at the center. In the case of FIG. 15A, since the high-luminance subject is located below the target pixel, the upper, upper right, and upper left are determined as directions in which the monotonic decrease of the target pixel 1 is detected. Since the optical information in the three directions is “1” indicating the color blur generation direction, the target pixel 1 is determined to be a color blur generation region. Similarly, if the pixel of interest 2 is also detected, the monotone decreasing direction is determined to be upper, upper right, right, and lower right. However, since the color blur occurrence direction of the optical information holds “0” indicating a direction that does not occur in the lower right, the final color blur detection direction of the target pixel 2 is monotonically decreased and the optical information is higher than the optical information. , Upper right and right. Although it is determined that both the target pixel 1 and the target pixel 2 cause color blurring, as shown in FIG. 14B, the amount of color blur generation in the photographing optical system differs depending on the direction. Then, the amount of color blur is different. If the optical information does not have intensity information, the amount of color blur cannot be determined, so that correction cannot be performed by changing the intensity of correction in the subsequent color blur correction process. Therefore, if the color blur correction is increased in accordance with the target pixel 1, the target pixel 2 is overcorrected, and if the color blur correction is decreased in accordance with the target pixel 2, the target pixel 1 is insufficiently corrected.

  Therefore, as shown in FIG. 14D, it is preferable to add the color blur generation direction and the color blur intensity information in the direction to the optical information. FIGS. 15A and 15B show a method for determining a color blur area when color blur intensity information is also added to the optical information. In the target pixel 3 in FIG. 15B, the three directions of upper, upper left, and upper right are determined by monotonic decrease determination, but the color blur intensity of the optical information is also detected as strong color blur intensity of 2, 3, and 2, respectively. can do. In the target pixel 4 in FIG. 15C, the three directions of upper, upper right, and lower left are determined, but the color blur intensity of the optical information in that direction is 3, 2, 1 and weaker than the target pixel 3, respectively. A color blur intensity is detected. Therefore, for example, by changing the intensity of correction (image correction intensity) of the subsequent color blur correction process according to the average value of the color blur intensity in the detected direction, Even when different, more accurate correction is possible.

  The target pixel 3 has an average color blur intensity value of about 2.3, and the target pixel 4 has an average color blur intensity value of 2.0, which can be used to change the correction strength. It becomes. In the present embodiment, the description has been made using the average value of the color blur intensity. However, the present invention is not limited to this. For example, the color blur correction coefficient may be changed using the color blur intensity value in the centroid direction of a plurality of detected directions as the color blur direction. At the target pixel 3, the direction of the center of gravity is upward, so the color blur intensity is 3, and at the target pixel 4, the direction of the center of gravity is upper right, so the color blur intensity is 2, and this value can be used to change the correction strength. Become. When the center of gravity direction is adopted, for example, when the center of gravity in the four directions of upper, upper right, right, and lower right is calculated, the direction closest to the center of gravity direction may be the two directions of upper right and right. In this case, it is conceivable to use an average value in two directions, a color blur intensity of the larger one of the two directions, or a color blur intensity of the smaller of the two directions. In the present embodiment, the use of the average value and the direction of the center of gravity has been described. However, the color blur intensity information of the optical information may be arbitrarily calculated to change the correction strength at the time of color blur correction. As with optical information, the color blur generation direction due to decentration aberrations in optical performance variation information is further enhanced by adding color blur intensity information in addition to the direction of color blur generation due to decentration. Is possible.

  Next, an example in which intensity information is included as optical performance variation information (second optical information) in addition to the color blur generation direction due to decentration will be described with reference to FIG. FIG. 26 is an explanatory diagram in which color blur intensity information is added as optical performance variation information.

  FIG. 26A shows the optical design value of the photographing optical system, that is, the optical information (first optical information as the design value) when there is no manufacturing error in the photographing optical system. FIG. 26B shows an example of a case where decentration aberration occurs in the upper left direction of a high-luminance subject due to a manufacturing error of the photographing optical system, and color blur occurs in a direction that does not originally occur in the design value. Similarly, FIG. 26C shows an example in which a decentration aberration occurs in the upper left direction of a high-brightness subject due to a manufacturing error of the photographing optical system, and a color blur occurs in a direction that does not originally occur in the design value. is there. FIG. 26C shows an example in which color blurring due to eccentricity occurs more than in FIG. 26B. FIG. 26 shows a case where, as optical performance variation information (second optical information), correction of a color blur generation amount due to decentration aberration is included as a difference value with respect to optical information (first optical information) of a design value. ing.

  The amount of occurrence of color blur due to decentering changes according to the amount of decentering at the time of manufacturing the optical element constituting the photographing optical system, and is not constant. That is, as shown in FIG. 26B and FIG. 26C, even when the eccentric directions coincide, the amount of color blur generation differs depending on the eccentric amount. For this reason, there is a case where the correct correction amount cannot be given only by having the color blur generation direction due to the eccentricity as the optical performance variation information (second optical information). Therefore, as shown in FIG. 26 (c), as the optical performance variation information (second optical information), in addition to the color blur occurrence direction due to decentration, intensity information regarding the color blur generation amount is included. Correction can be performed with higher accuracy.

  In the present embodiment, the optical performance variation information (second optical information) has been described as a difference value from the optical information (first optical information) based on the design value. However, the present invention is not limited to this. Absent. In the present embodiment, the second optical information has been described as data in eight directions, but the present invention is not limited to this. Other forms may be used as long as they are information on the direction and intensity of occurrence of color blur due to eccentricity and can correct the first optical information.

  Preferably, when the first direction (the direction in which the signal level monotonously increases or decreases monotonically) and the second direction (the direction in which color blur occurs) match each other in a plurality of directions, the correcting unit 104 preferably The image is corrected by calculating information on the intensity of color blur in the direction. More preferably, when the first direction and the second direction match each other in a plurality of directions, the correcting unit 104 uses the information regarding the intensity of color blurring in the centroid direction in the plurality of directions or the average value as an image. Correct it.

  Next, a method for calculating optical information (first optical information) will be described with reference to FIG. FIG. 16 is an explanatory diagram of a method for calculating optical information, and shows a PSF cross section (solid line) of the G plane of the photographing optical system and a PSF cross section (dotted line) of the B plane in an arbitrary direction held as optical information. Yes. Here, a method for calculating the optical information regarding the color blur of the B plane using the G plane as a reference plane will be described.

  Due to the chromatic aberration of the photographing optical system, the G plane and the B plane have different PSF shapes depending on the aberration in each wavelength band. Further, since the G plane and the B plane are imaged at shifted positions due to lateral chromatic aberration, the peak positions of the respective PSFs are also calculated at shifted positions as shown in FIG. The displacement of each color plane due to lateral chromatic aberration can be corrected by a conventional geometric transformation that applies different distortions to each color plane. Therefore, as shown in FIG. 16B, the PSF peak positions of the G plane and the B plane are made to coincide with each other, and the difference between the PSF areas of the G plane and the B plane, which are the hatched portions in the figure, is blurred. A method of adopting the strength can be considered.

  In FIG. 16B, the area difference of the right edge is larger than the area difference of the left edge than the peak, so that it can be seen that the color blur is greatly generated at the right edge. If this area is small, the color blur is inconspicuous, and therefore, when the area is small, the direction in which the color blur does not occur can be set. Further, the color blur intensity can be determined from the area ratio. As described above, the direction in which color blur occurs in the optical information and the color blur intensity can be calculated, but the calculation method is not limited to the method using the area difference of the PSF. For example, in FIG. 16B, a method of using the difference in inclination between the edges of the G plane and the B plane can be considered. When color blurring occurs, the inclination of the edge tends to be gentle, and thus optical information may be determined by using a difference in inclination.

  In this embodiment, the method of matching the peak values when calculating the color blur intensity is described. However, the color blur intensity may be calculated by matching the center of gravity of the PSF instead of the peak value. As described above, according to the present embodiment, it is possible to calculate the color blur generation direction and the color blur intensity information of the optical information.

  Referring back to FIG. 1B, in step S6, the image processing apparatus 100 (correction unit 104) corrects color blur for the region (second region) determined as the color blur region by the determination unit 103 in step S5. Processing (processing for correcting an image) is performed (color blur correction step). That is, the correcting unit 104 corrects the image so as to reduce color blur.

  FIG. 17 is a diagram illustrating a typical pixel value change of blue blur. In FIG. 17, the horizontal axis is the distance (cross section on the image), and the vertical axis is the pixel value of the B plane and the G plane. In FIG. 17, it is assumed that there is a high brightness subject exceeding the saturation brightness at the left end. Further, the tail of the pixel value change exponentially spreads around the light source, which is originally not bright, due to light that oozes from the light source due to aberration or flare.

  The G plane which is the reference plane is not free from bleeding, but has a certain extent. However, it is smaller than the B-plane that is subject to color blur removal. Further, the image sensor cannot measure pixel values above a certain saturation level. In such a pixel value change, if the intensity of the B-plane to be subjected to color bleeding removal exceeds the intensity of the G-plane that is the reference plane, blue bleeding occurs.

  In the present embodiment, the image processing apparatus 100 estimates the amount of blurring of the B plane based on the slope of the change in pixel value of the B plane that is subject to color blur removal. Therefore, by multiplying the absolute value of the brightness gradient Blea of the B plane by the coefficient k1, the first estimated blur amount E1 is obtained as expressed by the following equation (6).

E1 = k1 | Blea | (6)
In Expression (6), k1 is a positive value. However, in the area A1 where the B plane is saturated, the pixel value gradient becomes 0, and the luminance gradient before saturation cannot be obtained.

  Therefore, the estimated bleeding amount E2 for the region A1 where the B plane is saturated is estimated based on the luminance gradient Glea of the pixel value change of the G plane, as shown in the following equation (7).

E2 = k2 | Glea | (7)
In Expression (7), k2 is a positive value.

  Next, nonlinear conversion is performed on the pixel values of the B plane to generate the saturation S. This non-linear transformation indicates whether or not the B plane is saturated. In the region where the intensity of the B plane is saturated, the saturation S is 1, and in the region where the intensity of the B plane is low, the saturation S is 0. Although the saturation S may be a binary value of 0 or 1, the saturation S may be a value that continuously varies from 0 to 1 as shown in FIG. FIG. 18 is a characteristic diagram of nonlinear conversion with respect to the pixel value of the B plane.

Based on the generated saturation S, the estimated blur amount E1 or the estimated blur amount E2 calculated as described above is selected. That is, if the saturation S is a binary value of 0 and 1, a new estimated bleeding amount E is obtained.
E = E1 (when S = 0)
E = E2 (when S = 1)
And If the saturation S is a value that continuously changes from 0 to 1, a new estimated bleeding amount E is
E = (1-S) E1 + SE2
And

  Next, the estimated blur amount E is corrected to determine the amount E ′ to be actually removed. The estimated bleeding amount (removal amount) is in line with a certain model and does not necessarily match the actual bleeding amount. For example, even in the case of light detected on the same B plane, the blurring method varies between light having a wavelength of 450 nm and light having a wavelength of 400 nm, but this is not considered in step S5 (color blur determination step). When the estimated amount of blurring (removal amount) is too small, a slight amount of blueness remains after the removal of blue blurring. On the other hand, if the estimated blurring amount (removal amount) is excessive, the B plane may be excessively reduced with respect to the gray background, resulting in a yellowish green color.

  In particular, the latter (when it becomes yellowish green) is unnatural and gives the viewer a great sense of discomfort. Accordingly, in step S6 (color blur correction process), the blur removal is limited to operate only within a certain hue range. For this reason, first, in step S6, the chromaticity of the pixel is calculated. The following equation (8) is established for the strengths of the R, G, and B planes.

  FIG. 19 is a chromaticity coordinate ab plane in which a in equation (8) is the horizontal axis and b is the vertical axis. As shown in FIG. 19, blue is in the fourth quadrant indicated by the diagonal lines on the chromaticity coordinate ab plane (note that red, yellow and purple are the first quadrant, green and white are the second quadrant, and blue-green is the third quadrant. In the quadrant). When the estimated blur amount E is removed from the intensity of the B plane, B = B−E, and the chromaticity coordinate ab plane moves in the upper left direction as indicated by a dotted arrow. The starting point of the arrow is the chromaticity before the estimated blur amount E is removed, and the end point is the chromaticity after the estimated blur amount E is removed. Therefore, when the working hue range is limited to a> 0 and b <0, the following equation (9) is established.

B> 0.22R + 0.68G and B> −1.84R + 3.30G (9)
For this reason, the pixel E that does not satisfy Expression (9) is set to the removal amount E ′ = 0 that is actually removed, and is excluded from the color blur removal target. As a result, the pixel value of the pixel that does not satisfy Expression (9) is not affected. In FIG. 19, only the region in the fourth quadrant indicated by diagonal lines is the removal target.

  Further, the removal amount E ′ is set so as to satisfy the following expression (10) even for a pixel satisfying the expression (9).

E ′ = min (E, B− (0.22R + 0.68G), B − (− 1.84R + 3.30G)) (10)
As a result, the change in chromaticity due to the removal of the removal amount E ′ remains in the fourth quadrant as indicated by the solid line arrow in FIG.

  In the present embodiment, the restriction is made in the fourth quadrant of the chromaticity coordinate ab plane. However, the present invention is not limited to this, and the restriction may be made at an arbitrary angle. At this time, it is necessary to satisfy the following formula (11).

B> r1 · G + r2 · R and B> r3 · G + r4 · R (11)
In the equation (11), r1 to r4 are calculated as the following equation (12) using the limit angle θ. The hue limit is defined by two straight lines passing through the origin of the chromaticity coordinate ab plane, and θ1 and θ2 are angles representing the two straight lines.

  As a result, it is possible to prevent the B plane from decreasing beyond the hue limit range. The color plane removal amount E ′ calculated as described above is retained as the removal amount plane, and color blur correction processing is performed. The above-described low-pass filter is applied to the removal amount plane. In this embodiment, a simple a * b * plane is used for hue restriction. However, the present invention is not limited to this, and hue restriction processing is performed on the uv plane using a 3 × 3 RGB → YUV matrix. You may do.

A new B plane is created by subtracting the removal amount E ′ from the strength of the B plane. Only the pixels that have been determined as the color blur area in the color blur determination step S4 are to be removed from the color blur. Therefore, the strength of the new B plane is as follows:
B = BE '
If the monotonous increase / decrease determination flag is “0”,
B = B
It becomes. As described above, in step S7 in FIG. 1, the image processing apparatus 100 (output unit 105) outputs a color image obtained by correcting the B plane as an output image.

  Here, consider a case where the monotonic increase / decrease determination result in a certain region of 3 × 3 pixels in the image shown in FIG. 7 switches the value of the monotonic increase / decrease determination flag in adjacent pixels as shown in FIG. In such a case, the removal amount fluctuates in the boundary portion of the pixel (a case where the pixel is removed by a neighboring pixel and a case where the pixel is not removed is mixed), and the change of the pixel value becomes steep, unnatural and uncomfortable to the observer. May give. Therefore, a method of applying a low-pass filter to the generated removal amount plane is effective.

  Also, the following equation (13) calculates the gain of each pixel of the color plane using the monotonic increase / decrease determination result plane, and multiplies the removal amount to perform smoothing processing on the boundary portion (monotonous increase / decrease determination result plane). You may go.

  In the example shown in FIG. 20, the removal amount E ″ of the pixel p is expressed as the following Expression (14).

  With the above method, it is possible to remove only the color blur without a sense of incongruity.

  In the image processing method (color blur correction process) of this embodiment described with reference to FIG. 1B, the order of the steps is not limited to the order of steps shown in FIG. 1B, and can be changed as appropriate. It is. For example, the order of steps S2, S3, and S4 in FIG. 1B may be changed to the order of steps S4, S3, and S2.

  Next, a modification of this embodiment will be described. In the flowchart shown in FIG. 1B, it is assumed that the optical information (first optical information) based on the optical design value and the optical performance variation information (second optical information) are information independent from each other, that is, the first optical information. The case where both the second optical information and the second optical information are acquired is described. However, the present embodiment is not limited to this, and information including optical information based on optical design values and variation information on optical performance due to manufacturing errors may be acquired as optical information (third optical information). Good.

  FIG. 23 is a flowchart of an image processing method as a modification of the present embodiment. Each step in FIG. 23 is executed based on a command from the image processing apparatus 100, that is, by each unit of the image processing apparatus 100. The flowchart in FIG. 23 differs from the flowchart in FIG. 1B in that step S13 is provided instead of steps S3 and S4. Steps S11, S12, and S14 to S16 in FIG. 23 are the same as steps S1, S2, and S5 to S7 in FIG.

  In step S <b> 13, the image processing apparatus 100 (acquisition unit 106) acquires optical information (third optical information) related to color blur (color blur occurrence information) of the photographing optical system that reflects a manufacturing error of the photographing optical system. To do. The third optical information acquired in step S13 reflects the first optical information acquired in step S3 of FIG. 1B and the second optical information acquired in step S4 of FIG. 1B. It corresponds to the optical information obtained in this way (information in which a manufacturing error is reflected in the optical design value). The third optical information can be stored in a storage unit (for example, the storage unit 107 or the storage unit 208). In step S14, the determination unit 103 determines the second area based on the third optical information and the information related to the first area. In addition, also in the flowchart of FIG. 23, the order of each step can be changed suitably.

  Next, with reference to FIG. 21, an imaging apparatus according to Embodiment 1 of the present invention will be described. FIG. 21 is a configuration diagram of the imaging apparatus 200 in the present embodiment. The imaging apparatus 200 is installed with an image processing program for performing color blur correction processing (the above-described image processing method) of a captured image. This color blur correction processing is performed by an image processing unit 204 (image processing) in the imaging apparatus 200. Device).

  The imaging device 200 includes a photographing optical system 201 (lens) and an imaging device main body (camera main body). The photographing optical system 201 includes a diaphragm 201a and a focus lens 201b, and is configured integrally with an imaging apparatus main body (camera main body). However, the present embodiment is not limited to this, and can also be applied to an imaging apparatus in which the imaging optical system 201 is replaceably attached to the imaging apparatus main body.

  The image sensor 202 photoelectrically converts a subject image (optical image) formed via the photographing optical system 201 to obtain an image (captured image). That is, the subject image is photoelectrically converted by the image sensor 202 and converted into an analog signal (electric signal). The analog signal is converted into a digital signal by the A / D converter 203, and the digital signal is input to the image processing unit 204.

  An image processing unit 204 (corresponding to the image processing apparatus 100) performs predetermined processing on the digital signal and performs the above-described color blur correction processing. First, the image processing unit 204 (acquiring unit 106) acquires the imaging conditions (imaging condition information) of the imaging apparatus 200 (imaging optical system 201) from the state detection unit 207. The shooting condition information is information related to an aperture, a shooting distance, a focal length of a zoom lens, and the like. The state detection unit 207 can acquire the shooting condition information directly from the system controller 210, but is not limited thereto. For example, shooting condition information related to the shooting optical system 201 can be acquired from the optical system control unit 206. Note that the color blur correction process (image processing method) of this embodiment is as described with reference to FIG.

  Optical information (first optical information and second optical information or third optical information) regarding the color blur generation direction of the photographing optical system 201 can be held in the storage unit 208. The image processing unit 204 acquires optical information (first optical information and second optical information, or third optical information) corresponding to the shooting condition information from the storage unit 208, performs color blur correction processing, The output image is stored in the image recording medium 209 in a predetermined format. The display unit 205 displays an image obtained by performing a predetermined display process on the image subjected to the color blur correction process of the present embodiment. However, the present embodiment is not limited to this, and an image subjected to simple processing for high-speed display may be displayed on the display unit 205.

  A series of controls in this embodiment is performed by the system controller 210, and the imaging optical system 201 is mechanically driven by the optical system control unit 206 based on an instruction from the system controller 210. The optical system control unit 206 controls the aperture diameter of the aperture 201a as the F number shooting state setting. The optical system control unit 206 controls the position of the focus lens 201b by an unillustrated autofocus (AF) mechanism or a manual manual focus mechanism in order to perform focus adjustment according to the subject distance. Note that functions such as aperture diameter control and manual focus of the aperture 201a may not be executed according to the specifications of the imaging apparatus 200. The photographing optical system 201 can be provided with optical elements such as a low-pass filter and an infrared cut filter.

  Next, an image processing system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 22 is a configuration diagram of an image processing system 300 in the present embodiment. Note that the color blur correction process (image processing method) of the present embodiment is as described with reference to FIG.

  In FIG. 22, an image processing apparatus 301 (corresponding to the image processing apparatus 100) is a computer device equipped with image processing software 306 for causing a computer to execute the image processing method of this embodiment. The imaging device 302 is an imaging device such as a camera, a microscope, an endoscope, or a scanner. The storage medium 303 is a storage unit that stores captured images, such as a semiconductor memory, a hard disk, or a server on a network.

  The image processing apparatus 301 acquires captured image data from the imaging device 302 or the storage medium 303, and outputs the image data subjected to predetermined image processing to one or more of the output device 305, the imaging device 302, and the storage medium 303. Output. Further, the output destination can be stored in a storage unit built in the image processing apparatus 301. The output device 305 is a printer, for example. Further, the image processing software 306 can be installed in the image processing apparatus 301 from a network or a storage medium such as a CD-ROM 307.

  A display device 304 that is a monitor is connected to the image processing apparatus 301. Therefore, the user can perform the image processing work through the display device 304 and evaluate the corrected image. The image processing software 306 performs color blur correction processing (image processing method) of the present embodiment, and performs development and other image processing as necessary.

  It should be noted that information (correction information) related to the contents of data for image processing and the transfer between devices in this embodiment is preferably attached to individual image data. By adding necessary correction information to the image data, the correction processing of the present embodiment can be appropriately performed as long as the device is equipped with the image processing apparatus of the present embodiment.

  According to each embodiment, the optical blur related to the color blur generation direction of the photographing optical system and the optical performance variation information are held, and the color blur correction region is based on the optical information and the optical performance variation information as a result of the monotonous increase / decrease detection means To decide. As a result, the adverse effect of removing the color of the subject can be reduced, and a high-quality image can be obtained. Therefore, according to each embodiment, it is possible to provide an image processing method, an image processing apparatus, an imaging apparatus, and an image processing program capable of performing a good image restoration process.

  Note that the optical information related to the color blur generation direction of the photographing optical system may be acquired through a network as well as a method of reading out from a recording unit of a device equipped with the image processing apparatus of the present embodiment, or may be a PC or a camera. Information may be acquired from a lens or the like.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  In each embodiment, when performing color blur correction processing of a color image, monotonic increase / decrease determination information of pixel values of each color plane, information on the color blur occurrence direction of the photographing optical system, and variation information on optical performance of the photographing optical system are used. . As a result, it is possible to effectively correct the color blur while reducing the adverse effect of removing the original color of the subject on the subject photographed by the photographing optical system having a manufacturing error. Therefore, according to each embodiment, an image processing apparatus, an imaging apparatus, an image processing method, an image processing program, and an image processing apparatus capable of effectively reducing color blur in a color image while reducing the influence due to manufacturing errors of the optical system, and A storage medium can be provided.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 102 Detection means 103 Determination means 104 Correction means

Claims (17)

A signal level of a color plane corresponding to at least one color filter of an image obtained from an image sensor having a plurality of color filters or a signal level of a luminance plane generated from the image is first in a predetermined section. Detecting means for detecting a first region monotonously increasing or monotonically decreasing in the direction of
Based on the first optical information relating to the color blur of the photographing optical system, the second optical information relating to the variation of chromatic aberration that is asymmetric with respect to the optical axis of the photographing optical system, and the information relating to the first region, the color blur. Determining means for determining the second region where the
An image processing apparatus comprising: correction means for correcting the image so as to reduce the color blur. The first optical information is information relating to an optical design value of the photographing optical system,
The image processing apparatus according to claim 1, wherein the second optical information is information relating to a manufacturing error of the photographing optical system. And further comprising storage means for storing the first optical information and the second optical information,
The determination means includes
Based on the first optical information and the second optical information, obtain third optical information related to the color blur of the photographing optical system,
The image processing apparatus according to claim 1, wherein the second area is determined based on the third optical information and information related to the first area.   The image processing apparatus according to claim 3, wherein the storage unit stores the first optical information and the second optical information for each photographing condition.   The image processing apparatus according to claim 4, wherein the photographing condition includes at least one of a focal length, a subject distance, and an aperture value of the photographing optical system. Storage means for storing third optical information relating to color blur of the photographing optical system;
The third optical information is information obtained by reflecting the first optical information and the second optical information,
The image processing apparatus according to claim 1, wherein the determination unit determines the second area based on the third optical information and information related to the first area.   The image processing apparatus according to claim 3, wherein the third optical information is information related to a second direction in which the color blur of the photographing optical system occurs.   The image processing apparatus according to claim 7, wherein the determination unit determines the second area by comparing the first direction with the second direction.   9. The determination unit according to claim 8, wherein the determination unit determines that the first region is the second region when the first direction and the second direction coincide with each other. Image processing device.   The image processing apparatus according to claim 3, wherein the third optical information is information related to the intensity of the color blur of the photographing optical system.   The image processing apparatus according to claim 10, wherein the correction unit changes a strength for correcting the image based on information on the strength of the color blur.   When the first direction and the second direction coincide with each other in a plurality of directions, the correcting unit calculates the information on the intensity of color blur in the plurality of directions and corrects the image. The image processing apparatus according to claim 9.   When the first direction and the second direction coincide with each other in a plurality of directions, the correction unit corrects the image using information on the intensity of color blur in the centroid direction of the plurality of directions. The image processing apparatus according to claim 9. An image sensor that photoelectrically converts an optical image formed through the imaging optical system;
A signal level of a color plane corresponding to at least one color filter of an image obtained from the image pickup device including a plurality of color filters or a signal level of a luminance plane generated from the image is set in a predetermined section. Detecting means for detecting a first region monotonously increasing or monotonically decreasing in the direction of 1;
Based on the first optical information relating to the color blur of the photographing optical system, the second optical information relating to chromatic aberration variation asymmetric with respect to the optical axis of the photographing optical system, and the information relating to the first region, the color Determination means for determining the second area where the blur occurs;
An imaging apparatus comprising: correction means for correcting the image so as to reduce the color blur. A signal level of a color plane corresponding to at least one color filter of an image obtained from an image sensor having a plurality of color filters or a signal level of a luminance plane generated from the image is first in a predetermined section. Detecting a first region monotonically increasing or monotonically decreasing in the direction of
Based on the first optical information relating to the color blur of the photographing optical system, the second optical information relating to the variation of chromatic aberration that is asymmetric with respect to the optical axis of the photographing optical system, and the information relating to the first region, the color blur. Determining a second region in which occurrence occurs;
Correcting the image so as to reduce the color blur. An image processing method comprising: A signal level of a color plane corresponding to at least one color filter of an image obtained from an image sensor having a plurality of color filters or a signal level of a luminance plane generated from the image is first in a predetermined section. Detecting a first region monotonically increasing or monotonically decreasing in the direction of
Based on the first optical information relating to the color blur of the photographing optical system, the second optical information relating to the variation of chromatic aberration that is asymmetric with respect to the optical axis of the photographing optical system, and the information relating to the first region, the color blur. Determining a second region in which occurrence occurs;
An image processing program configured to cause a computer to execute the step of correcting the image so as to reduce the color blur.   A storage medium storing the image processing program according to claim 16.