JP2017092800A - Image processing apparatus, imaging apparatus, image processing method, image processing program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of effectively reducing color blur in a color image in consideration of manufacturing error of an imaging optical system.SOLUTION: An image processing apparatus 100 includes: an acquiring unit 101 that acquires information on chromatic aberration of magnification of an image formed via an imaging optical system; and a determination unit 104 for determining whether to modify the image so as to reduce color blur by using optical information concerning color blur of the imaging optical system on the basis of the information regarding the chromatic aberration of magnification.SELECTED DRAWING: Figure 1B

Description

  The present invention relates to an image processing apparatus that reduces color bleeding in a color image.

  Conventionally, a color that does not originally exist around a bright portion of a color image may occur as a color blur due to chromatic aberration of the imaging optical system. The chromatic aberration of the imaging optical system can be optically reduced to some extent by combining a plurality of lenses made of materials having different dispersions. However, in recent years, the downsizing of the image pickup apparatus has progressed, and it is required to increase the resolution of the sensor used for the image pickup element and the size of the image pickup optical system, and it is difficult to sufficiently suppress chromatic aberration by the configuration of the image pickup optical system. .

  Chromatic aberration is roughly classified into lateral chromatic aberration (magnification chromatic aberration) and longitudinal chromatic aberration (axial chromatic aberration). Lateral chromatic aberration is a phenomenon in which the image formation position shifts in the direction along the image plane depending on the wavelength. Longitudinal chromatic aberration is a phenomenon in which the imaging position shifts in the direction along the optical axis depending on the wavelength. Patent Document 1 discloses a method of correcting lateral chromatic aberration by performing geometric transformation processing on each color plane of R (red), G (green), and B (blue). Thus, the lateral aberration can be corrected by the geometric conversion process. On the other hand, longitudinal chromatic aberration cannot be corrected by the geometric conversion process.

  In Patent Document 2, a region in which a G (green) plane is saturated is searched using a characteristic in which color blur is mainly generated in the vicinity of a whiteout (a saturated region of a preset signal), and a region around the region is searched. A method of calculating a correction amount by integrating the pixels is disclosed. Patent Document 3 discloses a method of determining a region where a color plane pixel value in a color image monotonously increases or decreases as a color blur generation region and removing the color blur.

US Pat. No. 6,724,702 JP 2007-133582 A JP 2009-268033 A

  However, in the method disclosed in Patent Document 2, color blur that causes discomfort to the observer also occurs in a region that is not overexposed, and thus correction of color blur is insufficient. Further, according to the method disclosed in Patent Document 3, it is possible to correct the color blur of the region that is not overexposed. However, since color blur occurs due to aberration of the imaging optical system, there are directions where color blur occurs and does not occur with respect to the subject in an imaging optical system having asymmetric aberrations such as coma. When the color of the subject is similar to the color blur, there is a possibility that the subject color is erroneously determined as a color blur even in a direction where the color blur does not occur, and the original color of the subject is removed. Further, in order to further reduce erroneous determination regarding color blur, it is necessary to consider manufacturing variations (manufacturing errors) of the imaging optical system.

  Therefore, the present invention provides an image processing apparatus, an imaging apparatus, an image processing method, an image processing program, and a storage medium that can effectively reduce color blur in a color image in consideration of a manufacturing error of the imaging optical system.

  An image processing apparatus according to an aspect of the present invention includes an acquisition unit that acquires information about lateral chromatic aberration of an image formed through an imaging optical system, and color blurring of the imaging optical system according to the information about the lateral chromatic aberration. And a determination unit that determines whether or not to correct the image so as to reduce the color blur.

  An imaging apparatus according to another aspect of the present invention includes an imaging element that photoelectrically converts an optical image formed via an imaging optical system and outputs an image, an acquisition unit that acquires information about the chromatic aberration of magnification of the image, A determination unit configured to determine whether to correct the image so as to reduce the color blur using the optical information regarding the color blur of the imaging optical system in accordance with the information regarding the chromatic aberration of magnification.

  According to another aspect of the present invention, there is provided an image processing method, the step of acquiring information on lateral chromatic aberration of an image formed through an imaging optical system, and color blurring of the imaging optical system according to the information on the lateral chromatic aberration. Determining whether to correct the image so as to reduce the color blur using the optical information on the image.

  According to another aspect of the present invention, there is provided an image processing program that obtains information on lateral chromatic aberration of an image formed through an imaging optical system, and that blurs the color of the imaging optical system according to the information on the lateral chromatic aberration. And determining whether to correct the image so as to reduce the color blur using the optical information on the computer, and causing the computer to execute.

  A storage medium according to another aspect of the present invention stores the image processing program.

  Other objects and features of the invention are described in the following embodiments.

  According to the present invention, there are provided an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium capable of effectively reducing color blur in a color image in consideration of a manufacturing error of the imaging optical system. be able to.

It is a block diagram of the image processing apparatus in this embodiment. It is a flowchart which shows the image processing method in this embodiment. It is explanatory drawing of lateral chromatic aberration (color shift). It is a flowchart which shows the correction process of the magnification chromatic aberration in this embodiment. It is explanatory drawing of the preparation procedure of the correction data of the magnification chromatic aberration in this embodiment. It is a flowchart which shows the monotonous increase / decrease determination process in this embodiment. In this embodiment, it is a figure which shows an example in case the pixel area of the monotonous increase / decrease determination object has a monotonous increase / decrease characteristic. In this embodiment, it is a figure which shows an example in case the pixel area of the monotonous increase / decrease determination object does not have a monotonous increase / decrease characteristic. In this embodiment, it is a figure which shows the pixel area for the monotone increase / decrease determination centering on an attention pixel. In this embodiment, it is a figure which shows the pixel area for the monotone increase / decrease determination which used the attention pixel as the end point. In this embodiment, it is a figure which shows the area | region of 3x3 pixel in an image, and a figure which shows the result of having applied the low-pass filter with respect to each pixel. In this embodiment, it is a figure which shows the example of the change of an input signal at the time of applying a low-pass filter with respect to each pixel. In this embodiment, it is a figure which shows the example of the monotone increase / decrease detection of the imaging optical system which has a symmetrical aberration. In this embodiment, it is a figure which shows the example of the monotone increase / decrease detection of the imaging optical system which has an asymmetrical aberration. It is explanatory drawing of the optical information in this embodiment. In this embodiment, it is a figure which shows the example which determines a color bleeding area | region based on optical information and a monotonous increase / decrease detection result. It is explanatory drawing of the two-dimensional optical information in this embodiment. In this embodiment, it is a figure which shows the color blur determination process using optical information. It is explanatory drawing of the calculation method of the optical information in this embodiment. In this embodiment, it is a figure which shows the typical pixel value change of a blue blur. In this embodiment, it is a characteristic view of the nonlinear conversion with respect to the pixel value of B plane. It is a figure which shows the chromaticity coordinate ab surface in this embodiment. In this embodiment, it is a figure which shows the area | region of 3 * 3 pixel in B plane by a monotone increase / decrease determination result. It is a flowchart which shows another image processing method in this embodiment. It is explanatory drawing of correction | amendment of the color blur correction data (optical information regarding a color blur) in this embodiment. 1 is a configuration diagram of an imaging apparatus in Embodiment 1. FIG. FIG. 6 is a configuration diagram of an image processing system in Embodiment 2.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, an image processing apparatus and an image processing method in this embodiment will be described with reference to FIGS. 1A and 1B. The image processing apparatus and the image processing method described here are appropriately used in each embodiment described later. FIG. 1A is a block diagram of an image processing apparatus 100 in the present embodiment. FIG. 1B is a flowchart of an image processing method (image processing program) in the present embodiment. Each step in FIG. 1B is realized by each unit of the image processing apparatus 100 based on a command of a computer (processor) that executes an image processing program stored in a storage unit (memory), for example. The memory or the processor may be provided either inside or outside the image processing apparatus 100.

  As illustrated in FIG. 1A, the image processing apparatus 100 includes an acquisition unit 101, a detection unit 102, a correction unit 103, a determination unit 104, a determination unit 105, a storage unit 106, and an output unit 107. First, in step S101 of FIG. 1B, the image processing apparatus 100 (acquisition unit 101) acquires a captured image as an input image. The input image is a digital image (captured image) obtained by receiving light with the image sensor via the imaging optical system, and is deteriorated by the aberration of the imaging optical system including the lens and various optical filters. Note that the imaging optical system can be configured using not only a lens but also a mirror (reflection surface) having a curvature.

  The color component of the input image has information on RGB color components, for example. As the color component, other commonly used color spaces such as brightness, hue, and saturation expressed in LCH, luminance expressed in YCbCr, and color difference signals can be selected and used. As other color spaces, XYZ, Lab, Yuv, and JCh can be used. Color temperature may also be used. The captured image can be acquired by connecting the imaging device and the image processing device by wire or wirelessly. The captured image can also be acquired via a storage medium.

  Subsequently, in step S102, the image processing apparatus 100 (acquisition unit 101) acquires imaging conditions (imaging condition information) of the imaging optical system. The imaging conditions are the state of the imaging optical system such as the focal length, aperture value, and imaging distance (focus lens position). The shooting condition can be acquired from information added to the image such as Exif information.

  Subsequently, in step S103, the image processing apparatus 100 (acquisition unit 101) acquires magnification chromatic aberration correction data corresponding to the imaging conditions. The correction data for the chromatic aberration of magnification is data (stored in the storage unit) that is stored in advance in the storage unit (memory) of the imaging device or the image processing device based on the design value of the imaging optical system or the data measured in the manufacturing process. Information on chromatic aberration of magnification). For example, when the imaging device holds correction data and the image processing device does not hold correction data, it is transmitted from the imaging device to the image processing device by communication, or the correction data is added to the image data. It can be transmitted indirectly.

  Subsequently, in step S104, the image processing apparatus 100 (detection unit 102) detects information relating to the chromatic aberration of magnification (color shift) from the input image (captured image) acquired in step S101. In step S105, the image processing apparatus 100 (correction unit 103) corrects (corrects) the lateral chromatic aberration of the image based on the information on the lateral chromatic aberration detected in step S104. In the present embodiment, the correction of the lateral chromatic aberration can be corrected by comparing the information on the lateral chromatic aberration detected in step S104 with the correction data of the lateral chromatic aberration acquired in step S103. By comparing the chromatic aberration of magnification detected from the image with the correction data held in advance, it is possible to limit erroneous correction due to erroneous detection. Note that the correction of the lateral chromatic aberration in step S105 is performed in a step prior to the color blur correction in step S111 described later. The reason will be described below.

  FIG. 2 is an explanatory diagram of chromatic aberration of magnification (color shift), FIG. 2A is a simulated view of an input image in which chromatic aberration of magnification remains, and FIG. 2B is a horizontal sectional view of FIG. Each is shown. In FIG. 2A, the horizontal axis indicates the x-axis (horizontal axis), and the vertical axis indicates the y-axis (vertical axis). In FIG. 2B, the horizontal axis indicates the x-axis, and the vertical axis indicates the pixel value. Note that the intersection of the x-axis and the y-axis is the center of the image. 2A and 2B, the solid line represents the G plane, the dotted line represents the R plane, and the alternate long and short dash line represents the B plane. In FIG. 2, the imaging optical system is a coaxial system, and aberrations other than lateral chromatic aberration have been corrected. The intensity shapes of the original images coincide with each other in RGB and are shifted by the amount of lateral chromatic aberration. Shall.

  In such a case, as shown in FIG. 2, the pixel value of the B plane is higher than that of the R plane or the G plane on the low image height side (the intersection side of the x axis and the y axis). For this reason, the area on the low image height side appears blurred in blue. On the other hand, on the high image height side, the pixel value of the R plane is higher than that of the G plane and B plane. For this reason, the area on the high image height side appears blurred in red. When color blur correction is performed in such a state, as shown in FIG. 2, even if the plane of each color is just shifted due to the chromatic aberration of magnification, the color blur is processed. There is a possibility that it cannot be corrected correctly and the original color cannot be restored.

  For example, in the case shown in FIG. 2, when correcting the color blur so that the shapes of RGB are aligned, the portion where RGB overlaps is a part of the inside, so the edge portion is cut out from the shape of the original image. There is a thing that will be. Or, when trying to match the G plane so that the edge portion is not scraped, the pixel value of the R plane becomes low on the low image height side, and the pixel value of the B plane becomes low on the high image height side, which causes color loss. Can occur. Therefore, in the present embodiment, by correcting the lateral chromatic aberration first and then performing the color blur correction process described later, the color blur of the subject generated by the imaging optical system can be accurately performed without causing such a problem. Correct appropriately.

  Here, with reference to FIG. 3, the chromatic aberration correction process (steps S104 and S105 in FIG. 1B) in the present embodiment will be described. FIG. 3 is a flowchart showing the correction process of the lateral chromatic aberration.

  First, in step S201, the image processing apparatus 100 (acquisition unit 101) acquires a captured image as an input image. Subsequently, in step S202, the image processing apparatus 100 (detection unit 102) detects, from the image (captured image), an edge in which a color shift due to lateral chromatic aberration appears significantly. For detection of an edge, a Y (luminance) plane is used, but the present invention is not limited to this. The edge detected here is limited to the edge where the pixel value changes greatly in the radial direction from the optical center, so that a highly accurate color misregistration amount can be acquired. In the Y plane, the color shift due to the chromatic aberration of magnification appears mixed together. For this reason, it is preferable to target an edge having a certain width such that the pixel value monotonously increases or decreases monotonically for a plurality of pixels.

  Subsequently, in step S203, the image processing apparatus 100 (acquisition unit 101) acquires the amount of color misregistration at each edge detected in step S202. Depending on the positional relationship between the optical center and each edge, one of up / down, left / right, right up / down left, down left / up right / down is applied as the color misregistration direction. Thus, the correction process can be simplified. The correlation between the color components is used to acquire the color misregistration amount at each edge. For example, the amount of color misregistration can be acquired by determining the sum of absolute differences between color components. While the R plane (or B plane) is moved in the color shift direction with respect to the G plane, a place where the sum of absolute differences between color components is minimized is searched for in the pixels near the detected edge. It is possible to determine the color misregistration amount of the R plane (or B plane) with respect to the G plane from the position where the sum of absolute differences detected here is minimized. The color misregistration amount acquired in step S203 is a negative value when, for example, the R plane (or B plane) is shifted in the optical center direction with respect to the G plane, and the R plane (or B plane with respect to the G plane). A positive value is obtained when the plane is displaced in the direction opposite to the optical center.

  Subsequently, in step S204, the image processing apparatus 100 (correction unit 103) determines the image height based on the image height of each edge detected in step S202 and the color shift amount of each edge acquired in step S203. The correction data is created by obtaining the relationship between the color shift and the color shift. Here, the image height is a distance from a pixel corresponding to the optical center (hereinafter simply referred to as “optical center”). Hereinafter, the procedure for creating correction data will be described in detail with reference to FIG. FIG. 4 is an explanatory diagram of a procedure for creating correction data for chromatic aberration of magnification in the present embodiment. FIG. 4A is a diagram in which an image is divided into regions h1 to h8 for each image height, and FIG. And the relationship between the color misregistration ratios.

  First, assuming that the image height of the edge detected in step S202 of FIG. 3 is L and the color shift amount acquired in step S203 is D, the color shift rate M with respect to the image height is M = L / (L + D). It is expressed as Subsequently, as shown in FIG. 4A, the image is divided into eight regions h1 to h8 for each image height, and the region to which the edge belongs is selected. Subsequently, with respect to each of the plurality of edges detected in the image, calculation of the color misregistration rate M and selection of an area to be removed by the edge are performed. Then, the color misregistration rates M are tabulated for each of the eight regions divided according to the image height, an average value of the color misregistration rates M is obtained for each region, and the color misregistration rate of each region is determined.

  Subsequently, as shown in FIG. 4B, using the image height and the color misregistration rate, a high-order polynomial approximate expression F (L) is expressed as an approximate curve representing the relationship between the image height and the color misregistration rate. This is calculated and used as correction data. FIG. 4B shows an example in which correction data is calculated using a cubic polynomial. However, the present invention is not limited to this. Note that edge detection and color shift amount acquisition may be performed for all edges in the screen. However, for each of the eight regions divided according to the image height, the edge detection and the color misregistration amount acquisition are completed when the number of color misregistration ratios equal to or greater than a predetermined threshold is aggregated. It is possible to improve efficiency. In addition, by using only the area where the corresponding edge is found among the eight areas divided for each image height for calculating the higher-order polynomial approximation, even if there is an area where the corresponding edge is not found, the correction data Can be created.

  Subsequently, in step S205, the image processing apparatus 100 (correction unit 103) corrects the color misregistration of the image using the correction data created in step S204. First, in the pixel (X, Y) of the plane to be corrected (R plane, B plane), the color misregistration rate M from the image height L of the pixel (X, Y) using the relationship of M = F (L). Ask for. It is assumed that the optical center position is a coordinate system of (0, 0).

  Subsequently, the image processing apparatus 100 obtains the coordinates (X1, Y1) of the pixel generated by the color misregistration correction using the relationship of X1 = M × X and Y1 = M × Y. In the plane to be corrected, a pixel value corresponding to the coordinates (X1, Y1) is generated by a general interpolation process, and is set as the pixel value of the pixel (X, Y). Color misregistration correction is performed by performing these operations for all pixels. In step S206, the image processing apparatus 100 outputs a corrected image.

  Next, returning to FIG. 1, in step S106, the image processing apparatus 100 (determination unit 104) performs a manufacturing error determination. In this embodiment, the image processing apparatus 100 compares the correction data for the chromatic aberration of magnification acquired in step S103 with the correction data created in step S104 (S204), and the manufacturing error of the imaging optical system (the extent of the error). ). Here, the correction data of the lateral chromatic aberration acquired in step S103 is referred to as first correction data, and the correction data generated in step S204 is referred to as second correction data. The first correction data is a design value of the imaging optical system or data measured in the manufacturing process (information on lateral chromatic aberration based on the design value of the imaging optical system, that is, information on lateral chromatic aberration stored in the storage unit). On the other hand, the second correction data is correction data (information relating to the chromatic aberration of magnification of the image) of the chromatic aberration of magnification (a chromatic aberration of magnification detected from the captured image) that has occurred during shooting.

  When the first correction data is created based on the design value of the imaging optical system, and the first correction data and the second correction data match each other, no manufacturing error of the imaging optical system has occurred. Conversely, if the first correction data and the second correction data do not match each other, a manufacturing error of the imaging optical system has occurred. In the present embodiment, the image processing apparatus 100 (determination unit 104) determines whether or not the first correction data and the second correction data match each other based on the first correction data and the second correction data. Judgment is made based on whether or not the difference is within a predetermined range.

  On the other hand, when the first correction data is created based on the measurement in the manufacturing process of the imaging optical system, and the first correction data and the second correction data match each other, an error at the time of photographing is a manufacturing process. It means that there is no change from the time of measurement at. Conversely, if the first correction data and the second correction data do not match each other, an error has occurred after measurement in the manufacturing process. In this manner, the image processing apparatus 100 (determination unit 104) determines that there is no manufacturing error when no error has occurred after measurement in the manufacturing process. On the other hand, if an error occurs after measurement in the manufacturing process, the image processing apparatus 100 determines that there is a manufacturing error. Further, the manufacturing error means an error in a broad sense, and includes, for example, errors depending on the state at the time of shooting such as aberration fluctuation due to an anti-vibration lens, aberration fluctuation due to an attitude difference of the imaging apparatus, and aberration fluctuation due to environmental temperature.

  Then, the image processing apparatus 100 (storage unit 106) stores the result of the manufacturing error determination in step S106. For example, “1” is stored when there is a manufacturing error, and “0” is stored when there is no manufacturing error, as a determination flag for the presence or absence of manufacturing error. Note that the manufacturing error determination in step S106 is performed by comparing the correction data of the magnification chromatic aberration acquired in step S103 with the correction data created in step S104, and therefore is performed before the correction of magnification chromatic aberration in step S105. May be.

  Subsequently, in step S107, the image processing apparatus 100 (detection unit 102) is a region in which the pixel value (signal level) of any one of the plurality of color planes constituting the color image is monotonously increasing or monotonically decreasing. Is detected. Then, the image processing apparatus 100 (detection unit 102) tentatively determines the detected area as a color blur generation area (monotonic increase / decrease detection process). Several methods are conceivable as a method of determining the color blur generation region in the monotonous increase / decrease detection step. In the present embodiment, the determination is made based on the change characteristic of the pixel value that blurs in a certain pixel section (predetermined section).

  Since color blur occurs when the imaging position shifts in the direction along the optical axis depending on the wavelength, for example, in the case of blue blur, the blue plane (B plane) is out of focus (out of focus). . The color blur due to the defocusing exists over a certain pixel section, and the pixel value change of the color blur at that time has a characteristic of gradually decreasing from the highlight portion to the shadow portion of the image. Therefore, when the pixel value change of color blur in a certain pixel section has a monotonous increase or monotonic decrease characteristic, there is a possibility that color blur has occurred, so that it is temporarily determined as a color blur generation region.

  As described above, in the present embodiment, the image sensor includes a plurality of color filters. Then, the detection unit 102 monotonically increases the signal level (pixel value) of the color plane corresponding to at least one color filter of the image obtained from the image sensor in any direction (first direction) in a predetermined section. Alternatively, a monotonically decreasing area (first area) is detected. However, the present embodiment is not limited to this, and instead of the signal level of the color plane, for example, the signal level of the luminance plane (Y plane) generated from the image obtained from the image sensor increases monotonously or monotonically. You may comprise so that the area | region which is reducing may be detected.

  Actually, when the imaging optical system has an asymmetric (rotationally asymmetric) aberration with respect to the optical axis (center axis) such as coma aberration, the color blur generated by the imaging optical system is the direction in which the subject is generated. There is a direction that does not occur. For this reason, if the color blur generation area is determined only from monotonous increase / decrease, if the subject has a color similar to the color blur, the subject color may be erroneously determined to be color blur. Accordingly, it is not possible to correctly determine the color blur generation area only by monotonic increase / decrease detection.

  In the present embodiment, the region for detecting monotonic increase or monotonic decrease is a horizontal, vertical or diagonal pixel section centered on the target pixel of the color image, a horizontal direction with the target pixel as an end point, It includes either a vertical or diagonal pixel section.

  Detection of monotonic increase / decrease is performed by first calculating the pixel value gradient of the color plane. When the input image is composed of, for example, three color planes of the G plane, the B plane, and the R plane, the B plane is set as a color blur removal target, and the G plane is used as a reference plane. The respective luminance gradients Blea and Glea with respect to the B plane and the G plane are calculated as represented by the following expression (1).

  In Expression (1), G (x + 1, y) and B (x + 1, y) are pixel values on the right side of the target pixel in the G plane and B plane, respectively. G (x−1, y) and B (x−1, y) are pixel values on the left side of the target pixel in the G plane and the B plane, respectively. G (x, y + 1) and B (x, y + 1) are pixel values below the pixel of interest in the G plane and B plane, respectively. G (x, y-1) and B (x, y-1) are pixel values above and below the pixel of interest in the G plane and B plane, respectively. Although this embodiment will be described with three color planes, processing can be performed with any color plane depending on the number of color planes constituting the image.

  Next, with reference to FIGS. 5 to 7, a method for determining a color blur occurrence area in the monotonous increase / decrease detection process will be described. FIG. 5 is a flowchart showing a monotonous increase / decrease determination process (monotonic increase / decrease detection step: step S107) in the present embodiment. Each step in FIG. 5 is mainly executed by the image processing apparatus 100 (detection unit 102). First, in step S1520, the image processing apparatus 100 determines whether the ISO sensitivity is high (whether the ISO sensitivity is higher than a predetermined sensitivity). If the ISO sensitivity is high, the process proceeds to step S1521. On the other hand, if the ISO sensitivity is not high, the process proceeds to step S1522.

  In step S1522, the image processing apparatus 100 analyzes a change in the pixel value of the input signal with respect to vertical, horizontal, and diagonal pixel sections (predetermined sections) when the pixel of interest of the input image is the center. In step S1523, the image processing apparatus 100 determines (detects) whether or not the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic. As a result of the determination, if the change in the pixel value of the input signal in the pixel section does not have a monotonous increase / decrease characteristic, in step S1524, the image processing apparatus 100 displays the vertical / horizontal when the target pixel is the end point.・ A pixel value change of an input signal is analyzed for an oblique pixel section. In step S1525, the image processing apparatus 100 determines whether the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic.

  FIG. 6 is a diagram illustrating an example of a case where a pixel section subject to monotonous increase / decrease determination has a monotonous increase / decrease characteristic, where the vertical axis indicates the pixel value and the horizontal axis indicates the distance. FIG. 7 is a diagram illustrating an example of a case where a pixel section that is a monotonous increase / decrease determination target does not have a monotonous increase / decrease characteristic, where the vertical axis indicates the pixel value, and the horizontal axis indicates the distance. The image processing apparatus 100 performs monotonous increase / decrease determination with respect to an input signal having a pixel value change as shown in FIGS. The white squares shown in FIGS. 6 and 7 are the target pixels.

  As shown in FIG. 6, an image in which a change in pixel value of an input signal has a monotonous increase / decrease characteristic has a monotonous increase / decrease characteristic in a pixel section in which a monotonous increase / decrease determination is performed. Therefore, as a result of the analysis in step S1522 or S1524, in step S1523 or S1525, the image processing apparatus 100 determines that the pixel section has a monotonous increase / decrease characteristic. On the other hand, it is determined that the image having the change characteristic of the pixel value of the input signal as shown in FIG. 7 does not have the monotonous increase / decrease characteristic.

  If the change in the pixel value of the input signal in the pixel section has a monotonous increase / decrease characteristic (monotonic increase characteristic or monotonic decrease characteristic), the image processing apparatus 100 sets a monotonous increase / decrease flag in step S1527. On the other hand, if the change in the pixel value of the input signal in the pixel section does not have a monotonous increase / decrease characteristic (monotonic increase characteristic or monotonic decrease characteristic), the image processing apparatus 100 sets a monotonous increase / decrease flag in step S1526. Absent.

  The monotonous increase / decrease determination described above is applied to each pixel of the B plane. As a result, each pixel is associated with “1” if the monotonous increase / decrease flag is set, and “0” if the monotone increase / decrease flag is not set, and is generated and held as a monotone increase / decrease determination result plane. Used in determination (step S110 in FIG. 1B). Details of the method of using the monotonous increase / decrease determination result plane will be described later.

  Next, with reference to FIG. 8 and FIG. 9, a method of setting a pixel section for performing monotonous increase / decrease determination on the target pixel will be described. FIGS. 8A to 8D are diagrams showing a monotonic increase / decrease pixel section centered on the target pixel. FIGS. 9A to 9H are diagrams illustrating a monotone increase / decrease pixel section with the target pixel as an end point. Of the method of setting the pixel section centered on the target pixel and the method of setting the pixel section centered on the target pixel, the method of setting the pixel section centered on the target pixel is shown in FIGS. A method of setting the horizontal direction and the vertical direction around the target pixel as in FIG.

  Further, as a method of setting a pixel section centered on the target pixel, a method of setting the target pixel in an oblique direction as shown in FIGS. 8C and 8D can be considered. In other words, the isotropic property can be imparted to the diagonal direction by setting a pixel section having a distance similar to that in the horizontal direction or the vertical direction. In this case, the angle in the oblique direction is limited to a setting of 45 degrees with respect to the horizontal direction or the vertical direction as shown in FIGS. However, the present embodiment is not limited to this, and an arbitrary angle other than 45 degrees can be set. The distance d of the pixel section at that time is calculated using the following equation (2).

In Expression (2), x represents the horizontal direction, and θ represents the angle from the horizontal.

  On the other hand, the color blur around the highlight area and the color blur around the shadow area in the image are affected by saturation and noise, respectively. There is a case where it cannot be judged correctly because it does not have an increase / decrease characteristic. In this case, as shown in FIG. 9, a method of setting a pixel section with the target pixel as an end point is effective. When the monotonous increase / decrease determination is performed by the method as described above, and any one of the pixel intervals illustrated in FIGS. 8 and 9 has a pixel interval having the monotonous increase / decrease characteristic, the target pixel has the monotonous increase / decrease characteristic. It is determined to be a pixel.

  In the present embodiment, the pixel value of the input signal is used as a target for monotonic increase / decrease determination, but luminance gradient may be used. In this case, for example, it is determined that the color blur is caused when the change in the luminance gradient is reversed only once in a certain pixel section. It is effective to set the appropriate value of the number of pixels in the pixel section to the minimum color blur width among the color blurs generated under certain shooting conditions of the imaging apparatus.

  The color blur width varies depending on the imaging conditions of the imaging apparatus (aperture value, focal length, focusing accuracy, focus position in the image plane, coordinates on the image sensor, etc.). For this reason, it is possible to detect the color blur of the minimum width by matching the appropriate value of the number of pixels in the pixel section with the color blur of the minimum width of various color blurs generated under various shooting conditions. The maximum width color blur can be detected even by using a pixel section adapted for the minimum width color blur.

  By performing monotonous increase / decrease determination as described above, it is possible to extract a color blur to be detected. However, depending on the shooting conditions such as high ISO sensitivity, the S / N ratio may decrease due to noise included in the input signal, and as a result, the color blur may not have a monotonous increase / decrease characteristic. In that case, it is effective to perform a filtering process using a digital filter on the input signal in step S1521 in FIG. In the present embodiment, a case where the digital filter is a low-pass filter will be described, but the present invention is not limited to this.

  There are several methods for applying a low-pass filter to the input signal. For example, a case will be described in which a [1 2 1] low-pass filter in which the weighting coefficient of the pixel of interest is twice as large as that of adjacent pixels is applied. FIG. 10A is a diagram illustrating a 3 × 3 pixel region in an image. As shown in FIG. 7, when p is a target pixel in a certain region of 3 × 3 pixels in an image, first, a low-pass filter of [1 2 1] is applied in the horizontal direction. At this time, the target pixel P ′ is expressed as the following Expression (3).

When the adjacent pixels are similarly calculated, the result is as shown in FIG. FIG. 10B is a diagram illustrating a result of applying a low-pass filter to each pixel in a 3 × 3 pixel region in the image. Subsequently, when the low-pass filter of [1 2 1] is applied in the vertical direction, the target pixel P ″ is expressed as the following Expression (4).

  With reference to FIG. 11, an example of a change in the input signal when the low-pass filter is applied will be described. FIG. 11 is a diagram illustrating an example of a change in the input signal when a low-pass filter is applied to each pixel. In FIG. 11, the horizontal axis indicates the distance (cross section on the image, that is, the pixel section from the target pixel), and the vertical axis indicates the pixel value of the plane.

  The solid line in FIG. 11 indicates the case where the low-pass filter is not applied, the fine broken line indicates the case where the low-pass filter [1 2 1] is applied, and the rough broken line indicates the case where the low-pass filter [1 4 6 4 1] is applied. Yes. Here, [1 4 6 4 1] means that a low-pass filter is applied by applying a weighting factor to a pixel adjacent to the target pixel and a pixel separated by another pixel. By smoothing the input signal by applying the low-pass filter in this way, it is possible to highlight the monotonous increase / decrease characteristics inherent in color fringing. In this embodiment, the low-pass filter is applied in the order of the horizontal direction and the vertical direction, but the present invention is not limited to this. The low-pass filter may be applied in the reverse order, that is, in the order of the vertical direction and the horizontal direction. It is also possible to calculate a two-dimensional low-pass filter coefficient and apply the low-pass filter simultaneously in the horizontal direction and the vertical direction.

  Subsequently, in step S108 of FIG. 1, the subsequent processing is branched according to the result of the manufacturing error determination in step S106. First, the case where it is determined in step S106 that there is no manufacturing error will be described. At this time, the process proceeds to step S109, and the image processing apparatus 100 (acquisition unit 101) acquires color blur correction data (optical information relating to color blur of the imaging optical system) corresponding to the imaging condition acquired in step S102.

  Subsequently, in step S110, the image processing apparatus 100 (determination unit 105) compares the monotonic increase / decrease detection result in step S107 with the color blur correction data acquired in step S109, and determines whether or not to perform color blur correction. judge. That is, in the image processing apparatus 100 (determination unit 105), the region (first region) provisionally determined as the color blur generation region by the detection unit 102 in step S107 is the color blur generation region (second region). Whether or not (color blur determination step). In other words, the image processing apparatus 100 (decision unit 105) has an area (second area) in which color blur is generated based on the optical information on color blur of the imaging optical system and the information on the first area. To decide. Here, the optical information is information related to the optical design value of the imaging optical system.

  In the present embodiment, the image processing apparatus 100 (determination unit 105) includes a first direction (a direction in which the signal level monotonously increases or decreases monotonously) and a second direction (a direction in which color blur of the imaging optical system occurs). To determine the second region. For example, when the first direction and the second direction match each other (or when they are within a predetermined range that is evaluated as substantially matching), the determination unit 105 detects the first region (detected in step S107). It is determined that the (region) is the second region (region in which color blur occurs).

  Here, with reference to FIG. 12 and FIG. 13, an adverse effect in the case where the color blur area is determined only from the monotonous increase / decrease detection process will be described. An image obtained by imaging a subject with an imaging device such as a digital camera includes a blur component as an image degradation component caused by spherical aberration, coma aberration, field curvature, astigmatism, and the like of the imaging optical system. . Such a blur component is generated when a light beam emitted from one point of a subject to be collected again at one point on the imaging surface when an aberration is not caused and there is no influence of diffraction is formed by forming an image with a certain spread.

  The blur component is optically represented by a point spread function (PSF), and is different from blur due to a focus shift. In addition, color blur in a color image can also be said to be a difference in blurring for each wavelength of light with respect to axial chromatic aberration, spherical spherical aberration, and color coma aberration of the optical system. Further, the lateral color misregistration can also be referred to as a positional misalignment or a phase misalignment due to a difference in imaging magnification for each wavelength of light when the lateral chromatic aberration is caused by the optical system.

  An optical transfer function (OTF) obtained by Fourier-transforming the point spread function (PSF) is frequency component information of aberration and is represented by a complex number. The absolute value of the optical transfer function (OTF), that is, the amplitude component, is referred to as MTF (Modulation Transfer Function), and the phase component is referred to as PTF (Phase Transfer Function). MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively. Here, the phase component is expressed as the following equation (5) with the phase angle as the phase angle.

PTF = tan ⁻¹ (Im (OTF) / Re (OTF)) (5)
In formula (5), Re (OTF) and Im (OTF) are the real part and the imaginary part of OTF, respectively.

  As described above, since the optical transfer function (OTF) of the optical system deteriorates the amplitude component and the phase component of the image, each point of the subject is asymmetrically blurred with respect to the optical axis like coma aberration. It becomes a state.

  FIG. 12 is a diagram for explaining a case where monotonous increase / decrease detection is performed on an image photographed in a state of optical characteristics in which each point of a subject is symmetric (rotationally symmetric) with respect to the optical axis such as spherical aberration. 12A shows the pixel value section of the subject, FIG. 12B shows the PSF section of the imaging optical system, and FIG. 12C shows the pixel value section of the subject imaged by the imaging optical system. Note that FIG. 12 assumes an R plane or a B plane. As shown in FIG. 12B, when the imaging optical system has an aberration characteristic that is symmetric (rotationally symmetric) with respect to the optical axis and the PSF is symmetric, the edge cross sections on both sides of the subject are both degraded images. Photographed as shown by the solid line in FIG. In FIG. 12C, the luminance cross section of the subject shown in FIG. 12A is indicated by a dotted line for comparison. When the monotonous increase / decrease determination is performed using the white square in FIG. 12C as the target pixel, both edges are determined as the monotonous increase / decrease area. Further, since color blur actually occurs, the color blur generation region is correctly determined.

  FIG. 13 is a diagram illustrating a case where monotonous increase / decrease detection is performed on an image captured with optical characteristics in which each point of the subject is asymmetric (rotationally asymmetric) with respect to the optical axis such as coma. 13A is a pixel value section of the subject, FIG. 13B is a PSF section of the imaging optical system, and FIG. 13C is a pixel value section of the subject imaged by the imaging optical system. As shown in FIG. 13B, when the imaging optical system has an asymmetric aberration characteristic with respect to the optical axis and the PSF is asymmetric, the edge cross section of the subject is one side particularly in the optical characteristic that greatly deteriorates only on one side edge. Only an image that is greatly degraded is taken as shown by the solid line in FIG. When monotonous increase / decrease detection is performed on such a captured image, both edges are determined as monotonous increase / decrease areas, but since the color blur actually generated by the imaging optical system is only on one side, the original color of the subject is blurred. As a result, an adverse effect occurs. If the color blur correction is performed only by monotonous increase / decrease, the original color of the subject is removed, which may result in an unnatural image. If you try to avoid this, the correction will be weak.

  Therefore, in the present embodiment, the image processing apparatus 100 holds or acquires optical information related to the color blur generation direction of the imaging optical system as shown in FIGS. 12B and 13B. When the monotonous increase / decrease direction of the monotonic increase / decrease detection matches the color blur generation direction, the image processing apparatus 100 determines that the color blur generation region is present. Here, the term “matching” means not only the case of exact matching but also the case of substantial matching.

  Here, the optical information will be described with reference to FIG. FIG. 14 is an explanatory diagram of the optical information in the present embodiment, and shows a PSF cross section and optical information of the imaging optical system. FIG. 14A shows optical information when the PSF of the imaging optical system shown in FIG. 12B is symmetric with respect to the optical axis. FIG. 14B shows optical information when the PSF of the imaging optical system shown in FIG. 13B is asymmetric with respect to the optical axis. For example, it has 1 in the direction in which color blur occurs due to the aberration of the imaging optical system, and 0 if it does not occur. In FIG. 14A, since color blur occurs symmetrically on both sides, the optical information has a value of 1 on both sides. In FIG. 14B, the color blur is generated asymmetrically, the color blur is not generated at the left edge, and 0 is generated at the right edge, and the value is 1 because the color blur is generated at the right edge.

  FIG. 15 is a diagram illustrating an example of determining a color blur area based on optical information and a monotonous increase / decrease detection result. As shown in FIG. 15A, for example, when monotonic decrease is detected by monotonic increase / decrease detection, the optical information in the monotonic decrease direction is referred to, and if it has 1, it is determined as a color blur correction region. To do. Further, as shown in FIG. 15B, the optical information in the monotonously decreasing direction is referred to, and if it has 0, it is determined that the color of the subject. As described above, by having information on the direction in which chromatic aberration occurs and the direction in which it does not occur as the aberration of the imaging optical system, it is possible to correctly determine the color and color blur of the subject, thereby improving the accuracy of color blur correction. It becomes possible. Although the direction of occurrence of color blur is 1 and the direction of non-occurrence is 0, it is only necessary to know the direction of occurrence, and it is not necessary to be 0 and 1. Since the aberration of the imaging optical system changes depending on the image height, it is possible to correct with high accuracy by having optical information at a plurality of image heights. For image heights that do not have correction values, interpolation may be generated from nearby correction values. The aberration of the imaging optical system also changes depending on the shooting conditions (focal length of the imaging optical system, subject distance (shooting distance), aperture value (Fno)). Therefore, by storing or acquiring optical information for each shooting condition, It becomes possible to perform correction with high accuracy.

  14 and 15, the optical information has been described as one-dimensional information. However, when the photographed image is two-dimensional array data, the optical information may be held as two-dimensional data. FIG. 16 is an explanatory diagram when the optical information is two-dimensional. FIG. 16C shows an example in which optical information is held as data in eight directions of up, down, left and right, 45 degrees oblique and 135 degrees on the screen. In FIG. 16, the coma of the color of the imaging optical system is generated in the upward direction of the screen, and 1 optical information is held in the generation direction. 0 is maintained in the direction where no color blur occurs. Monotonic increase / decrease detection is performed in the above eight directions, and it is determined that the color is blurred when 1 of the optical information is held in the direction in which the monotonic decrease is detected. Even if a monotonic decrease is detected, if the optical information 1 is not held in the detected direction, it is determined as the color of the subject.

  As described above, the image processing apparatus 100 holds or acquires the optical information regarding the direction in which the color blur of the imaging optical system occurs. Preferably, the image processing apparatus 100 includes a storage unit 106 that stores optical information. More preferably, the storage unit 106 stores optical information for each shooting condition. The shooting conditions include at least one of a focal length of the imaging optical system, a subject distance (shooting distance), and an aperture value. Note that the storage unit 106 may be provided outside the image processing apparatus 100. For example, each optical information can be stored in the storage unit 208 described later.

  The image processing apparatus 100 can determine the color blur area with high accuracy by referring to the detection direction and the optical information when detecting monotonous increase / decrease. In the present embodiment, the monotonous increase / decrease determination has been described in the monotonic decrease direction. However, the color blur determination may be performed using the monotone increase direction. In addition, the monotonous increase / decrease determination and the correction value of the optical information have been described in eight directions, but are not limited to eight directions. In order to improve accuracy, the number of detection directions may be increased. Alternatively, the number of detection directions may be reduced in order to reduce the amount of data related to optical information.

  Further, the image processing apparatus 100 may be configured to hold or acquire information regarding the color blur intensity (color blur intensity information) as optical information in addition to the color blur generation direction. In this case, correction can be performed with higher accuracy. As shown in FIG. 16B, generally, the amount of color blur due to the aberration of the imaging optical system differs depending on the direction. Therefore, as shown in FIG. 16D, the color blur intensity information regarding the color blur generation amount may be added in the direction in which the color blur occurs.

  Next, with reference to FIG. 17, color blur determination when color blur intensity information is used will be described. FIG. 17 is a diagram showing a color blur determination process using optical information, and shows an example in which the optical information has only the color blur generation direction. When determining the color blur generation region for the pixel of interest 1 surrounded by the square in the figure, first, monotonic decrease determination is performed in eight directions, up and down, left and right, 45 degrees oblique, and 135 degrees oblique, with the pixel of interest at the center. In the case of FIG. 17A, since the high-luminance subject is located below the target pixel, three directions in which the monotonic decrease of the target pixel 1 is detected are determined as the upper, upper right, and upper left. Since the optical information in the three directions is “1” indicating the color blur generation direction, the target pixel 1 is determined to be a color blur generation region. Similarly, if the pixel of interest 2 is also detected, the monotone decreasing direction is determined to be upper, upper right, right, and lower right. However, since the color blur occurrence direction of the optical information holds “0” indicating a direction that does not occur in the lower right, the final color blur detection direction of the target pixel 2 is monotonically decreased and the optical information is higher than the optical information. , Upper right and right. Although the noticeable pixel 1 and the noticeable pixel 2 are determined to have no color bleed, the amount of color bleed in the imaging optical system varies depending on the direction as shown in FIG. In No. 2, the amount of color bleeding is different. If the optical information does not have intensity information, the amount of color blur cannot be determined, so that correction cannot be performed by changing the intensity of correction in the subsequent color blur correction process. Therefore, if the color blur correction is increased in accordance with the target pixel 1, the target pixel 2 is overcorrected, and if the color blur correction is decreased in accordance with the target pixel 2, the target pixel 1 is insufficiently corrected.

  Therefore, as shown in FIG. 16D, it is preferable to add the color blur generation direction and the color blur intensity information in the direction to the optical information. FIGS. 17A and 17B show a method of determining a color blur area when color blur intensity information is also added to the optical information. In the target pixel 3 in FIG. 17B, the three directions of upper, upper left, and upper right are determined by monotonic decrease determination, but the color blur intensity of the optical information is also detected as strong color blur intensity of 2, 3, and 2, respectively. can do. In the pixel of interest 4 in FIG. 17C, the three directions of upper, upper right, and lower left are determined, but the color blur intensity of the optical information in that direction is 3, 2, 1 and weaker than the pixel of interest 3, respectively. A color blur intensity is detected. Therefore, for example, by changing the intensity of correction (image correction intensity) of the subsequent color blur correction process according to the average value of the color blur intensity in the detected direction, Even when different, more accurate correction is possible.

  The target pixel 3 has an average color blur intensity value of about 2.3, and the target pixel 4 has an average color blur intensity value of 2.0, which can be used to change the correction strength. It becomes. In the present embodiment, the description has been made using the average value of the color blur intensity. However, the present invention is not limited to this. For example, the color blur correction coefficient may be changed using the color blur intensity value in the centroid direction of a plurality of detected directions as the color blur direction. At the target pixel 3, the direction of the center of gravity is upward, so the color blur intensity is 3, and at the target pixel 4, the direction of the center of gravity is upper right, so the color blur intensity is 2, and this value can be used to change the correction strength. Become. When the center of gravity direction is adopted, for example, when the center of gravity in the four directions of upper, upper right, right, and lower right is calculated, the direction closest to the center of gravity direction may be the two directions of upper right and right. In this case, it is conceivable to use an average value in two directions, a color blur intensity of the larger one of the two directions, or a color blur intensity of the smaller of the two directions. In the present embodiment, the use of the average value and the direction of the center of gravity has been described. However, the color blur intensity information of the optical information may be arbitrarily calculated to change the correction strength at the time of color blur correction.

  Preferably, when the first direction (the direction in which the signal level monotonously increases or decreases monotonically) and the second direction (the direction in which color blur occurs) match each other in a plurality of directions, the correcting unit 103 preferably The image is corrected by calculating information on the intensity of color blur in the direction. More preferably, when the first direction and the second direction match each other in a plurality of directions, the correcting unit 103 uses the information regarding the intensity of color blurring in the centroid direction in the plurality of directions or the average value as an image. Correct it.

  Next, a method for calculating optical information will be described with reference to FIG. FIG. 18 is an explanatory diagram of a method for calculating optical information, and shows the PSF cross section (solid line) of the G plane of the imaging optical system and the PSF cross section (dotted line) of the B plane in any direction held as optical information. Yes. Here, a method for calculating the optical information regarding the color blur of the B plane using the G plane as a reference plane will be described.

  Due to the chromatic aberration of the imaging optical system, the G plane and the B plane have different PSF shapes depending on the aberration in each wavelength band. Further, when lateral chromatic aberration remains, the G plane and B plane are imaged at shifted positions, so the peak positions of the respective PSFs are also calculated at shifted positions as shown in FIG. Is done. The displacement of each color plane due to lateral chromatic aberration can be corrected by a conventional geometric transformation that applies different distortions to each color plane. Therefore, as shown in FIG. 18B, the PSF peak positions of the G plane and B plane are made to coincide with each other, and the PSF area difference between the G plane and the B plane, which is the hatched portion in the figure, is blurred. A method of adopting the strength can be considered.

  In FIG. 18B, since the area difference of the right edge is larger than the area difference of the left edge than the peak, it can be seen that the color fringing occurs greatly at the right edge. If this area is small, the color blur is inconspicuous, and therefore, when the area is small, the direction in which the color blur does not occur can be set. Further, the color blur intensity can be determined from the area ratio. As described above, the direction in which color blur occurs in the optical information and the color blur intensity can be calculated, but the calculation method is not limited to the method using the area difference of the PSF. For example, in FIG. 18B, a method using the difference in inclination between the edges of the G plane and the B plane can be considered. When color blurring occurs, the inclination of the edge tends to be gentle, and thus optical information may be determined by using a difference in inclination.

  In this embodiment, the method of matching the peak values when calculating the color blur intensity is described. However, the color blur intensity may be calculated by matching the centroids of the PSFs instead of the peak values. As described above, according to the present embodiment, it is possible to calculate the color blur generation direction and the color blur intensity information of the optical information. In addition, an optical correction value can be calculated for each direction in which monotonous increase / decrease determination, image height, and photographing conditions are performed.

  Returning to FIG. 1B, in step S111, the image processing apparatus 100 (correction unit 103) corrects color blur for the region (second region) determined as the color blur region by the determination unit 104 in step S110. Processing (processing for correcting an image) is performed (color blur correction step). That is, the correction unit 103 corrects the image so as to reduce color blur.

  FIG. 19 is a diagram illustrating a typical pixel value change of blue blur. In FIG. 19, the horizontal axis is the distance (cross section on the image), and the vertical axis is the pixel value of the B plane and the G plane. In FIG. 19, it is assumed that there is a high-luminance subject that exceeds the saturation luminance at the left end. Then, the periphery of the high-brightness subject which is not originally bright also spreads so that the bottom of the pixel value change gradually approaches the black level due to aberration and flare. The G plane which is the reference plane is not free from bleeding, but has a certain extent. However, it is smaller than the B-plane that is subject to color blur removal. Further, the image sensor cannot measure pixel values above a certain saturation level. In such a pixel value change, if the intensity of the B-plane to be subjected to color bleeding removal exceeds the intensity of the G-plane that is the reference plane, blue bleeding occurs.

  In the present embodiment, the image processing apparatus 100 estimates the amount of blurring of the B plane based on the slope of the change in pixel value of the B plane that is subject to color blur removal. Therefore, by multiplying the absolute value of the brightness gradient Blea of the B plane by the coefficient k1, the first estimated blur amount E1 is obtained as expressed by the following equation (6).

E1 = k1 | Blea | (6)
In Expression (6), k1 is a positive value. However, in the area A1 where the B plane is saturated, the pixel value gradient becomes 0, and the luminance gradient before saturation cannot be obtained.

  Therefore, the estimated bleeding amount E2 for the region A1 where the B plane is saturated is estimated based on the luminance gradient Glea of the pixel value change of the G plane, as shown in the following equation (7).

E2 = k2 | Glea | (7)
In Expression (7), k2 is a positive value.

  Next, nonlinear conversion is performed on the pixel values of the B plane to generate the saturation S. This non-linear transformation indicates whether or not the B plane is saturated. In the region where the intensity of the B plane is saturated, the saturation S is 1, and in the region where the intensity of the B plane is low, the saturation S is 0. The saturation S may be a binary value of 0 or 1. However, as shown in FIG. 20, the saturation S may be a value that continuously varies from 0 to 1. FIG. 20 is a characteristic diagram of nonlinear conversion with respect to the pixel value of the B plane.

Based on the generated saturation S, the estimated blur amount E1 or the estimated blur amount E2 calculated as described above is selected. That is, if the saturation S is a binary value of 0 and 1, a new estimated bleeding amount E is obtained.
E = E1 (when S = 0)
E = E2 (when S = 1)
And If the saturation S is a value that continuously changes from 0 to 1, a new estimated bleeding amount E is
E = (1-S) E1 + SE2
And

  Next, the estimated blur amount E is corrected to determine the amount E ′ to be actually removed. The estimated bleeding amount (removal amount) is in line with a certain model and does not necessarily match the actual bleeding amount. For example, even in the case of light detected on the same B plane, the blurring method varies between light having a wavelength of 450 nm and light having a wavelength of 400 nm, but this is not taken into consideration in step S110 (color blur determination step). When the estimated amount of blurring (removal amount) is too small, a slight amount of blueness remains after the removal of blue blurring. On the other hand, if the estimated blurring amount (removal amount) is excessive, the B plane may be excessively reduced with respect to the gray background, resulting in a yellowish green color.

  In particular, the latter (when it becomes yellowish green) is unnatural and gives the viewer a great sense of discomfort. Therefore, in step S111 (color blur correction process), the blur removal is limited to operate only within a certain hue range. For this reason, first, in step S111, the chromaticity of the pixel is calculated. The following equation (8) is established for the strengths of the R, G, and B planes.

  FIG. 21 is a chromaticity coordinate ab plane in which a in equation (8) is the horizontal axis and b is the vertical axis. As shown in FIG. 21, blue is in the fourth quadrant indicated by diagonal lines on the chromaticity coordinate ab plane (note that red, yellow and purple are the first quadrant, green and white are the second quadrant, and blue-green is the third quadrant. In the quadrant). When the estimated blur amount E is removed from the intensity of the B plane, B = B−E, and the chromaticity coordinate ab plane moves in the upper left direction as indicated by a dotted arrow. The starting point of the arrow is the chromaticity before the estimated blur amount E is removed, and the end point is the chromaticity after the estimated blur amount E is removed. Therefore, when the working hue range is limited to a> 0 and b <0, the following equation (9) is established.

B> 0.22R + 0.68G and B> −1.84R + 3.30G (9)
For this reason, the pixel E that does not satisfy Expression (9) is set to the removal amount E ′ = 0 that is actually removed, and is excluded from the color blur removal target. As a result, the pixel value of the pixel that does not satisfy Expression (9) is not affected. In FIG. 21, only the region in the fourth quadrant indicated by diagonal lines is the removal target.

  Further, the removal amount E ′ is set so as to satisfy the following expression (10) even for a pixel satisfying the expression (9).

E ′ = min (E, B− (0.22R + 0.68G), B − (− 1.84R + 3.30G)) (10)
As a result, the change in chromaticity due to the removal of the removal amount E ′ remains in the fourth quadrant, as indicated by the solid line arrow in FIG.

  In the present embodiment, the restriction is made in the fourth quadrant of the chromaticity coordinate ab plane. However, the present invention is not limited to this, and the restriction may be made at an arbitrary angle. At this time, it is necessary to satisfy the following formula (11).

B> r1 · G + r2 · R and B> r3 · G + r4 · R (11)
In the equation (11), r1 to r4 are calculated as the following equation (12) using the limit angle θ. The hue limit is defined by two straight lines passing through the origin of the chromaticity coordinate ab plane, and θ1 and θ2 are angles representing the two straight lines.

  As a result, it is possible to prevent the B plane from decreasing beyond the hue limit range. The color plane removal amount E ′ calculated as described above is retained as the removal amount plane, and color blur correction processing is performed. The above-described low-pass filter is applied to the removal amount plane. In this embodiment, a simple a * b * plane is used for hue restriction. However, the present invention is not limited to this, and hue restriction processing is performed on the uv plane using a 3 × 3 RGB → YUV matrix. You may do.

  A new B plane is created by subtracting the removal amount E 'from the strength of the B plane. Only the pixels that have been determined as the color blur area in the color blur determination step S4 are to be removed from the color blur. Therefore, the intensity of the new B plane is B = B−E ′ when the monotonous increase / decrease determination flag is “1”. On the other hand, when the monotonous increase / decrease determination flag is “0”, B = B. As described above, in step S112 of FIG. 1B, the image processing apparatus 100 (output unit 107) outputs a color image obtained by correcting the B plane as an output image.

  Here, consider a case where the monotonous increase / decrease determination result in the 3 × 3 pixel region in the image shown in FIG. 10A switches the value of the monotonous increase / decrease determination flag in adjacent pixels as shown in FIG. In such a case, the removal amount fluctuates in the boundary portion of the pixel (a case where the pixel is removed by a neighboring pixel and a case where the pixel is not removed is mixed), and the change of the pixel value becomes steep, unnatural and uncomfortable to the observer. May give. Therefore, a method of applying a low-pass filter to the generated removal amount plane is effective.

  Also, the following equation (13) calculates the gain of each pixel of the color plane using the monotonic increase / decrease determination result plane, and multiplies the removal amount to perform smoothing processing on the boundary portion (monotonous increase / decrease determination result plane). You may go.

  In the example shown in FIG. 22, the removal amount E ″ of the pixel p is expressed as the following Expression (14).

  With the above method, it is possible to remove only the color blur without a sense of incongruity.

  Subsequently, a case where it is determined that there is a manufacturing error in the manufacturing error determination in step S106 will be described. In this case, the image processing apparatus 100 skips steps S109 and S110 and directly executes step S111. That is, in step S111, the image processing apparatus 100 (correction unit 103) corrects color blur for the region (first region) that is temporarily determined as a color blur generation region by the detection unit 102 in step S107. Processing (processing for correcting an image) is performed (color blur correction step). In other words, the correction unit 103 corrects the image so as to reduce color blur in the region detected in the monotonous increase / decrease detection process in step S107.

  This is because if there is a manufacturing error in the imaging optical system, the reliability of the color blur correction data (optical information relating to the color blur of the imaging optical system, that is, the optical design value) in step S109 decreases. The color blur correction data is aberration information of the imaging optical system as described above, and is information on the direction and intensity of the color flare. If the direction or intensity changes due to a manufacturing error, even if color blur due to chromatic aberration is detected in the monotonic increase / decrease detection process in step S107, there is a possibility that the color blur determination in step S110 may be excluded from the correction target. On the other hand, even if the subject color is used, it is determined that the color is blurred and erroneous correction may be performed.

  Next, another image processing method in the present embodiment will be described with reference to FIG. FIG. 23 is a flowchart showing another image processing method according to this embodiment. Steps S301 to S307 in FIG. 23 are the same as steps S101 to S107 in FIG.

  Subsequently, in step S308, the image processing apparatus 100 (acquisition unit 101) acquires color blur correction data (optical information regarding color blur of the imaging optical system). Subsequently, in step S309, the subsequent processing is branched according to the result of the manufacturing error determination in step S306. If it is determined in step S306 that there is a manufacturing error, the process proceeds to step S313, and the image processing apparatus 100 (correction unit 103) acquires the color blur correction data (optical information relating to the color blur of the imaging optical system) acquired in step S308. ) Is corrected (corrected). For example, when the image processing apparatus 100 (detection unit 102) compares the lateral chromatic aberration detected in step S104 with the lateral chromatic aberration correction data acquired in step S103, the lateral chromatic aberration is a factor in manufacturing the imaging optical system. It detects how it is shifted due to an error. Then, the image processing apparatus 100 (correction unit 103) corrects the color blur correction data (optical information related to the color blur of the imaging optical system) acquired in step S308 based on the shift (shift amount, shift direction) of the chromatic aberration of magnification. To do.

  FIG. 24 is an explanatory diagram of correction of color blur correction data (optical information relating to color blur) in step S308. For example, when the chromatic aberration of magnification of the entire screen is shifted (shifted) in the direction of the arrow shown in FIG. 24B (downward), the correction data (optical information) shown in FIG. Then, the correction data (optical information) shown in FIG. That is, correction data (optical information) is shifted in the shift direction (shift direction) of lateral chromatic aberration. Correction of correction data (correction amount, correction direction) depends on the imaging optical system. For this reason, it is preferable to store the relationship between the lateral chromatic aberration and the color blur calculated in advance in the storage unit 106 as table data or a function, for example.

  Subsequently, in step S310 of FIG. 23, the image processing apparatus 100 (determination unit 104) performs color blur determination using the color blur correction data (optical information after correction) corrected in step S313. On the other hand, when it is determined in step S306 that the imaging optical system has no manufacturing error, step S313 is not executed.

  Subsequently, in step S311, the image processing apparatus 100 (correction unit 103) performs color blur correction using the color blur correction data acquired in step S308 or the color blur correction data corrected in step S311. In step S312, the image processing apparatus 100 (output unit 107) outputs an image corrected (corrected) so as to reduce color blur.

  As described above, in the present embodiment, the image processing apparatus 100 includes the storage unit 106 that stores the optical information related to the color blur of the imaging optical system and the information related to the chromatic aberration of magnification of the imaging optical system. When the difference between the lateral chromatic aberration acquired from the image (captured image) and the lateral chromatic aberration stored in the storage unit 106 does not exceed a predetermined threshold value, the correcting unit 103 corrects the image using the optical information.

  The manufacturing error can be determined based only on the lateral chromatic aberration detected from the image. That is, when there is no manufacturing error, the distance from the center of the screen is the same, and lateral chromatic aberration at different positions such as the upper right and lower left of the image is generated in the same amount symmetrically with respect to the screen center. On the other hand, when eccentric chromatic aberration of magnification occurs due to a manufacturing error, a color shift in one direction occurs on the entire screen, and the appearance of the chromatic aberration of magnification becomes asymmetric. For this reason, it is possible to estimate a manufacturing error even if there is no design value by comparing the generation direction and generation amount of the lateral chromatic aberration at a position diagonal to the center of the screen. Therefore, the image processing apparatus 100 may include a storage unit 106 that stores optical information. The correction unit 103 corrects the image using the optical information when the difference in the chromatic aberration of magnification at a position symmetrical with respect to the center of the image does not exceed a predetermined threshold.

  On the other hand, the correction unit 103 corrects the first region of the image when the difference exceeds a predetermined threshold. As a modified example, when the difference exceeds a predetermined threshold, the image processing apparatus 100 (decision unit 105) corrects the optical information according to the chromatic aberration of magnification acquired from the image, and the corrected optical information and the first information are corrected. The second area is determined based on the information regarding the first area. However, this embodiment is not limited to these. For example, the correction unit 103 may not correct the image when the difference exceeds a predetermined threshold (that is, do not perform color blur correction of the image).

  According to the image processing method of the present embodiment, it is determined whether or not to perform color blur correction using optical information related to color blur of the imaging optical system in accordance with manufacturing errors due to manufacturing variations and lens eccentricity adjustment. For this reason, according to the present embodiment, it is possible to provide an image processing method capable of effectively reducing color bleeding in a color image in consideration of a manufacturing error of the imaging optical system.

  Next, with reference to FIG. 25, an imaging apparatus according to Embodiment 1 of the present invention will be described. FIG. 25 is a configuration diagram of the imaging apparatus 200 in the present embodiment. The imaging apparatus 200 is installed with an image processing program for performing color blur correction processing (the above-described image processing method) of a captured image. This color blur correction processing is performed by an image processing unit 204 (image processing) in the imaging apparatus 200. Device).

  The imaging apparatus 200 includes an imaging optical system 201 (lens) and an imaging apparatus main body (camera main body). The imaging optical system 201 includes a diaphragm 201a and a focus lens 201b, and is configured integrally with an imaging apparatus main body (camera main body). However, the present embodiment is not limited to this, and can also be applied to an imaging apparatus in which the imaging optical system 201 is attached to the imaging apparatus body in a replaceable manner.

  The image sensor 202 photoelectrically converts a subject image (optical image) formed via the imaging optical system 201 to obtain an image (captured image). That is, the subject image is photoelectrically converted by the image sensor 202 and converted into an analog signal (electric signal). The analog signal is converted into a digital signal by the A / D converter 203, and the digital signal is input to the image processing unit 204.

  An image processing unit 204 (image processing apparatus) performs predetermined processing on the digital signal and performs the above-described color blur correction processing. First, the image processing unit 204 (acquisition unit 101) acquires imaging condition information of the imaging apparatus 200 (imaging optical system 201) from the state detection unit 207. The imaging condition information is information regarding an aperture value, a shooting distance (subject distance), a focal length of a zoom lens, and the like. The state detection unit 207 can acquire the imaging condition information directly from the system controller 210, but is not limited to this. For example, the imaging condition information regarding the imaging optical system 201 can be acquired from the optical system control unit 206. Note that the color blur correction process (image processing method) of this embodiment is as described with reference to FIG. 1B.

  Correction data relating to chromatic aberration of magnification of the imaging optical system 201 is held in the storage unit 208. The correction data relating to the chromatic aberration of magnification is data held in advance by the imaging device and the image processing device based on the design value of the imaging optical system and data measured in the manufacturing process. The image processing unit 204 acquires correction data related to the chromatic aberration of magnification corresponding to the shooting condition information from the storage unit 208. The image processing unit 204 compares the correction data with the chromatic aberration of magnification (correction amount) detected from the image, and performs manufacturing error determination based on the comparison result. When it is determined that there is a manufacturing error, the image processing unit 204 stores a determination flag “1” in the storage unit 208. On the other hand, if it is determined that there is no manufacturing error, the determination flag “0” is stored in the storage unit 208.

  In addition, correction data relating to color bleeding of the imaging optical system 201 is also held in the storage unit 208. The image processing unit 204 acquires correction data related to color blur corresponding to the shooting condition information from the storage unit 208, and performs color blur correction processing. The corrected output image is stored in the image recording medium 209 in a predetermined format. The display unit 205 displays an image obtained by performing a predetermined display process on the image subjected to the color blur correction process of the present embodiment. However, the present embodiment is not limited to this, and an image subjected to simple processing for high-speed display may be displayed on the display unit 205.

  A series of controls in this embodiment is performed by the system controller 210, and mechanical driving of the imaging optical system 201 is performed by the optical system controller 206 based on an instruction from the system controller 210. The optical system control unit 206 controls the aperture diameter of the aperture 201a as the F number shooting state setting. The optical system control unit 206 controls the position of the focus lens 201b by an unillustrated autofocus (AF) mechanism or a manual manual focus mechanism in order to perform focus adjustment according to the subject distance. Note that functions such as aperture diameter control and manual focus of the aperture 201a may not be executed according to the specifications of the imaging apparatus 200. The imaging optical system 201 can be provided with optical elements such as a low-pass filter and an infrared cut filter.

  Next, an image processing system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 26 is a configuration diagram of an image processing system 300 in the present embodiment. Note that the color blur correction process (image processing method) of this embodiment is as described with reference to FIG.

  In FIG. 26, an image processing apparatus 301 is a computer device equipped with image processing software 306 for causing a computer to execute the image processing method of this embodiment. The imaging device 302 is an imaging device such as a camera, a microscope, an endoscope, or a scanner. The storage medium 303 is a storage unit that stores captured images, such as a semiconductor memory, a hard disk, or a server on a network.

  The image processing apparatus 301 acquires captured image data from the imaging device 302 or the storage medium 303, and outputs the image data subjected to predetermined image processing to one or more of the output device 305, the imaging device 302, and the storage medium 303. Output. Further, the output destination can be stored in a storage unit built in the image processing apparatus 301. The output device 305 is a printer, for example. Further, the image processing software 306 can be installed in the image processing apparatus 301 from a network or a storage medium such as a CD-ROM 307.

  A display device 304 that is a monitor is connected to the image processing apparatus 301. Therefore, the user can perform the image processing work through the display device 304 and evaluate the corrected image. The image processing software 306 performs color blur correction processing (image processing method) of the present embodiment, and performs development and other image processing as necessary.

  It should be noted that information (correction information) related to the contents of data for image processing and the transfer between devices in this embodiment is preferably attached to individual image data. By adding necessary correction information to the image data, the correction processing of the present embodiment can be appropriately performed as long as the device is equipped with the image processing apparatus of the present embodiment.

  Note that the optical information related to the color blur generation direction of the imaging optical system may be acquired via a network as well as a method of reading from a recording unit of a device equipped with the image processing apparatus of the present embodiment, or may be a PC or camera Information may be acquired from a lens or the like.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  According to each embodiment, an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can effectively reduce color blur in a color image in consideration of manufacturing errors of the imaging optical system are provided. can do.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

100 Image processing apparatus 101 Acquisition unit 104 Determination unit

Claims (24)

An acquisition unit for acquiring information on lateral chromatic aberration of an image formed via the imaging optical system;
A determination unit that determines whether to correct the image so as to reduce the color blur using the optical information regarding the color blur of the imaging optical system according to the information about the chromatic aberration of magnification. An image processing apparatus. A signal level of a color plane corresponding to at least one color filter of the image obtained from an image pickup device including a plurality of color filters or a signal level of a luminance plane generated from the image is set in a predetermined section. A detection unit for detecting a first region monotonously increasing or monotonically decreasing in the direction of 1;
A determination unit that determines a second region in which the color blur is generated based on the optical information and the information on the first region;
The image processing apparatus according to claim 1, further comprising: a correction unit that corrects the image so as to reduce the color blur according to information on the chromatic aberration of magnification. A storage unit that stores the optical information and information about the chromatic aberration of magnification of the imaging optical system;
When the difference between the information about the lateral chromatic aberration acquired from the image and the information about the lateral chromatic aberration stored in the storage unit does not exceed a predetermined threshold, the correcting unit uses the optical information to display the image. The image processing apparatus according to claim 2, wherein the correction is performed.   The image processing apparatus according to claim 3, wherein the correction unit corrects the first region of the image when the difference exceeds the predetermined threshold.   The determination unit corrects the optical information according to information on the lateral chromatic aberration acquired from the image when the difference exceeds the predetermined threshold, and relates to the corrected optical information and the first region. The image processing apparatus according to claim 3, wherein the second area is determined based on the information.   The image processing apparatus according to claim 3, wherein the correction unit does not correct the image when the difference exceeds the predetermined threshold. A storage unit for storing the optical information;
The correction unit corrects the image using the optical information when a difference in lateral chromatic aberration at a position symmetric with respect to the center of the image does not exceed a predetermined threshold. The image processing apparatus described.   The image processing apparatus according to claim 7, wherein the correction unit corrects the first region of the image when the difference exceeds the predetermined threshold.   When the difference exceeds the predetermined threshold, the determination unit corrects the optical information according to information on the lateral chromatic aberration acquired from the image, and relates to the corrected optical information and the first region. The image processing apparatus according to claim 7, wherein the second area is determined based on the information.   The image processing apparatus according to claim 7, wherein the correction unit does not correct the image when the difference exceeds the predetermined threshold.   The image processing apparatus according to claim 3, wherein the storage unit stores the optical information for each shooting condition.   The image processing apparatus according to claim 11, wherein the photographing condition includes at least one of a focal length, a subject distance, and an aperture value of the imaging optical system.   The image processing apparatus according to claim 2, wherein the optical information is information related to a second direction in which the color blur of the imaging optical system occurs.   The image processing apparatus according to claim 13, wherein the determination unit determines the second region by comparing the first direction and the second direction.   The said determination part determines that the said 1st area | region is the said 2nd area | region, when the said 1st direction and the said 2nd direction correspond mutually. Image processing device.   The image processing apparatus according to claim 2, wherein the optical information is information relating to the intensity of the color blur of the imaging optical system.   The image processing apparatus according to claim 16, wherein the correction unit changes a strength for correcting the image based on information on the strength of the color blur.   When the first direction and the second direction coincide with each other in a plurality of directions, the correction unit corrects the image by calculating information on the intensity of color blur in the plurality of directions. The image processing apparatus according to claim 15.   When the first direction and the second direction coincide with each other in a plurality of directions, the correction unit corrects the image using information on the intensity of color blur in the centroid direction of the plurality of directions. The image processing apparatus according to claim 15.   The image processing apparatus according to claim 1, wherein the optical information is information related to an optical design value of the imaging optical system. An image sensor that photoelectrically converts an optical image formed via the imaging optical system and outputs an image; and
An acquisition unit for acquiring information on lateral chromatic aberration of the image;
A determination unit that determines whether to correct the image so as to reduce the color blur using the optical information regarding the color blur of the imaging optical system according to the information about the chromatic aberration of magnification. An imaging device. Obtaining information on lateral chromatic aberration of an image formed via the imaging optical system;
Determining whether to correct the image so as to reduce the color blur using the optical information related to the color blur of the imaging optical system according to the information about the chromatic aberration of magnification. Image processing method. Obtaining information on lateral chromatic aberration of an image formed via the imaging optical system;
Determining whether to correct the image so as to reduce the color blur using the optical information related to the color blur of the imaging optical system according to the information about the chromatic aberration of magnification. An image processing program configured as described above.   24. A storage medium storing the image processing program according to claim 23.