JP2007028040A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To execute chromatic aberration correction of an image picked up through an optical system. <P>SOLUTION: A control apparatus 111 obtains information about a difference in an MTF (modulation transfer function) characteristic between an omitted color component and a reference color component, in a first image in which at least one of a plurality of color components is omitted in one of pixels picked up by an image pickup element 102 through a lens 101, and the MTF characteristic is different between the reference color component and at least another omitted color component in an image pickup surface. The apparatus 111 uses the obtained information to generate a second image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that corrects an image captured through an optical system.

  The following chromatic aberration correction apparatus is known from Patent Document 1. In this chromatic aberration correction device, for a color image in which three colors are aligned, the defocus between RGB caused by the arrangement inclination of the RGB plane image sensor is smoothed or sharpened to another color plane with respect to the reference color plane. A filter is applied to search and correct the variable filter parameters so that the correlation between the color planes is maximized.

JP 2001-103358 A

  However, if such a technique is applied to the on-axis chromatic aberration correction technique of a single-plate image sensor such as a Bayer array, a predetermined interpolation algorithm is used to form a color image in which three colors are aligned, and then between RGB planes. It is necessary to see the correlation regarding the defocusing for each color. Therefore, depending on the type of interpolation algorithm, the generation of false color associated with the image structure is more difficult to detect than the amount of color blur due to axial chromatic aberration, and filtering for false color suppression is often used. In such a case, the local chromatic aberration of axial chromatic aberration spreads greatly, and the axial chromatic aberration is stable and easily affected by various interpolation algorithms such as those restored to an MTF state completely different from the original MTF state. There has been a problem that it may not be possible to correct.

According to the first aspect of the present invention, an image is picked up through an optical system, and at least one of a plurality of color components is missing in one pixel, and an MTF characteristic is present between the reference color component and at least one other missing color component on the imaging surface. An image processing apparatus for converting a different first image into a second image having matched MTF characteristics, wherein an MTF between a missing color component and a reference color component in a pixel having a missing color component of the first image It is characterized by comprising image generation means for obtaining information relating to the difference in characteristics and generating a second image using the obtained information.
According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the image generating means uses the obtained information to correct the missing color component to match the MTF characteristic of the reference color component, A second image in which at least one missing color component is interpolated for each pixel is generated.
According to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the image generation means uses the obtained information to change the MTF characteristic of the missing color component of the first image to the MTF characteristic of the reference color component. The second image having the same color component arrangement as that of the first image is generated by performing correction so as to be matched.
According to a fourth aspect of the present invention, in the image processing apparatus according to the third aspect of the present invention, the image processing apparatus further includes an interpolation unit that interpolates the missing color component of the second image.
According to a fifth aspect of the present invention, in the image processing device according to any one of the second to fourth aspects, the image generation means uses the reference color component as at least one missing color component having a different MTF characteristic of the first image. Smoothing means for smoothing pixels having the reference color component so as to approximate the MTF characteristic, and based on the result of smoothing by the smoothing means, the MTF characteristic of the missing color component of the first image is determined as the reference color component It is characterized by matching to the MTF characteristics of
According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, since the smoothing means generates the color difference component used for generating the second image, the smoothing means generates a reference color component for the first image. The smoothing process is performed, and the smoothing process is not performed on the reference color component of the first image for generating the luminance component used for generating the second image.
According to a seventh aspect of the present invention, in the image processing apparatus according to the first aspect, a color component bearing luminance information of the first image is associated with the reference color component of the first image.
According to an eighth aspect of the present invention, in the image processing apparatus according to the first aspect, as the reference color component of the first image, the color component arranged at the highest density of the first image is made to correspond. .
According to a ninth aspect of the present invention, in the image processing apparatus according to the first aspect, information relating to the difference in MTF characteristics is determined based on external information relating to the state of the optical system at the time of imaging.
According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the external information relating to the state of the optical system at the time of shooting includes the lens type, zoom position, focus position, aperture value, and in-screen used at the time of shooting. It includes at least one piece of information on the focus position.
The invention of claim 11 is the image processing apparatus according to claim 5 or 6, wherein the smoothing means determines the degree of smoothing for each pixel.
According to a twelfth aspect of the present invention, in the image processing apparatus according to the fifth or sixth aspect, the smoothing means determines a common smoothing level across a plurality of pixels.
According to a thirteenth aspect of the present invention, in the image processing apparatus according to the sixth aspect, when the smoothing means generates a plurality of types of color difference components representing different color information for one pixel, each type of color difference component The degree of smoothing of the smoothing process is determined for each color component of the first image that constitutes.

  According to the present invention, not the color image that has undergone the interpolation process, but the color image before the interpolation process, that is, the image in which at least one of a plurality of color components is missing from one pixel is maintained. Thus, axial chromatic aberration can be corrected stably and accurately without being affected by the interpolation processing algorithm.

-First embodiment-
In the first embodiment, there is shown a method for correcting by self-detecting from an image itself without having any optical system information regarding axial chromatic aberration. The self-detection method of lateral chromatic aberration, which is a lateral aberration, is relatively easy to measure by measuring the similarity between the RGB planes in the spatial direction, but what is a self-detection method for longitudinal chromatic aberration, which is a longitudinal aberration? Since it is not obvious whether it should be used as an indicator, the method is shown. As a result, it is possible to cope with axial chromatic aberration that varies depending on the subject distance even in the same lens state. In other words, in a landscape photograph with an infinite focus, the subject in the foreground is red-fogged and the subject in the distance is blue-fogged, or in a close-up shooting due to the centimeter-order relationship of the subject position relative to the lens focus position. It is possible to cope with a situation in which a change in response of a longitudinal chromatic aberration occurs.

  FIG. 1 is a block diagram showing a configuration of an embodiment when the image processing apparatus according to the first embodiment is mounted on a digital camera. The digital camera 100 includes a lens 101, an image sensor 102 composed of a CCD or the like, and an image processing device 110. In the image sensor 102, for example, the most representative R (red), G (green), and B (blue) color filters of a single-plate color image sensor are arranged in a Bayer array. The image data captured by the image sensor 102 is assumed to be expressed in the RGB color system, and color information of any one color component of RGB exists in each pixel constituting the image data. To do. That is, it is assumed that the image is a Bayer image.

  The image processing device 110 includes a CPU and other peripheral circuits, and displays a control device 111 that executes image processing to be described later, an image memory 112 that stores an image captured by the image sensor 102, and an image after image processing. And a monitor 113. In this digital camera, Bayer image data captured by the image sensor 102 through the lens 101 is A / D converted by the control device 111 to be converted into a digital signal, and stored in the image memory 112.

  As described above, in the image data picked up by the image pickup device 102 and stored in the image memory, there is a possibility that color blur accompanying axial chromatic aberration has occurred. That is, when an achromatic black line subject is imaged and there is no axial chromatic aberration, no blur signal is generated in each RGB component as shown in FIG. When there is a large axial chromatic aberration in the R component, a blur signal is generated in the R component as shown in FIG. That is, the G component and B component signals are clear color component signals, while the R component signal is a blurred color component signal.

  In this case, the blur signal generated in the R component wraps around the black line subject position and floats only by red, so that it is captured as a reddish line as a whole, and red blurring occurs particularly near the edge of the black line subject. When the B component has a large axial chromatic aberration, blue blur occurs. When both R and B have a large axial chromatic aberration, magenta blur occurs.

  Such an image is generally captured using a single-plate image sensor such as a Bayer array, and the missing color component is interpolated. Therefore, the dependency on the interpolation algorithm occurs and the situation becomes more complicated, but a high-performance algorithm In many cases, axial chromatic aberration is reproduced as faithfully as possible. Therefore, in this embodiment, the luminance component is interpolated as it is so as not to impair the sharpness in the Bayer array state that does not depend on the interpolation algorithm, and the difference in MTF characteristics between the color components is generated in the generation of the color difference component. The conventional color-difference component generation processing is performed after the conversion to match the blurred side to prevent the occurrence of color blur due to axial chromatic aberration.

  Here, a description will be given of the principle of occurrence of color blur due to axial chromatic aberration as described above when an achromatic black line subject is imaged. When an achromatic black line subject is imaged through the optical system, if there is no longitudinal chromatic aberration, the RGB components are in focus on the imaging surface as shown in FIG. That is, as shown in FIG. 3B, the MTF characteristics of the RGB components are matched. On the other hand, when there is axial chromatic aberration in the R component, only the R component is defocused from the imaging surface as shown in FIG. 3C, and as shown in FIG. This is a state in which there is a mismatch in MTF characteristics.

  In the first embodiment, the control device 111 corrects an inconsistency in the MTF characteristics between the color components when any one of the RGB color components has axial chromatic aberration and color blur is caused accordingly. Thus, the color blur caused by the longitudinal chromatic aberration is eliminated. That is, the axial chromatic aberration is eliminated by matching the MTF characteristics of the respective color components by matching or bringing them close to each other. Specifically, the color blur accompanying axial chromatic aberration is eliminated by the following principle.

  For example, with respect to an image in which the signal level of each RGB component is as shown in each of FIGS. 4A to 4C, the MTF characteristics of the blurred color component signal such as the R component and the B component are shown. The MTF is matched by sharpening using the MTF characteristic of a clear color component signal such as the G component. For this purpose, first, the clear color component signal is used as a reference color component, and the MTF characteristic of the reference color component is smoothed and blurred so as to approach or match the MTF characteristic of the blurred color component signal. That is, in the example illustrated in FIG. 4, the MTF characteristics of the G component are smoothed so as to approach the MTF characteristics of the R component and the B component.

As a result, MTF characteristics of the G component as shown in the MTF characteristics of the G component as shown in FIG. 4 (d) blurred strongly closer to the MTF characteristics of the R component and <G> _strong, FIG 4 (e) <G> _Weak is obtained by weakly blurring so as to approach the MTF characteristics of the B component. Then, as shown in FIG. 4 (f), the color difference components Cr and Cb are generated using the smoothed G components corresponding to the R component and the B component, that is, <G> _strong and <G> _weak. To do.

Then, if the RGB color system is restored based on the generated color difference components Cr and Cb and the original G component that has not been smoothed, the MTF characteristics of the R, G, and B components are changed to the MTF characteristics of the G component. They can be matched or brought close to each other so that they can be matched. In other words, by adding to each of the R component and the B component the difference between the G component and each of the <G> _strong and <G> _weak obtained by smoothing the G component with different smoothing degrees, The MTF characteristics of the R and B components can be matched with the clear GTF MTF characteristics.

That is, the R component is corrected by the following equation (1), the B component is corrected by the following equation (2), and the other components (G component) having a high MTF characteristic are used to correct the R and B components. It is possible to correct the longitudinal chromatic aberration.
R ′ = R + (G− <G> _strong ) (1)
B '= B + (G- <G> _weak ) (2)

  In this way, by reconstructing fine structure information that cannot be recovered from only a color component having a low MTF characteristic by using another color component having a high MTF characteristic, high-precision axial chromatic aberration correction that has not been conventionally possible can be achieved. It becomes possible. Further, axial chromatic aberration correction is performed using color signals that have undergone smoothing for MTF matching for only the chrominance component, and luminance components that are not conspicuous by the axial chromatic aberration are the unsmoothed color components. Since the image is generated using the signal, it is possible to perform correction with less image structure destructiveness while maintaining sharpness.

  Hereinafter, a process for correcting the mismatch of the MTF characteristics between the color components and eliminating the color blur due to the longitudinal chromatic aberration will be described in detail.

(1-1) Bayer Image Input First, a Bayer image captured by the image sensor 102 is read from the image memory 112 and set as an image processing target. As shown in FIG. 5, each pixel [i, j] of the Bayer image has color information of any color component of R, G, B, that is, a color component value (a value corresponding to a CCD signal value). ) Exists. [I, j] is a coordinate indicating the pixel position.

(1-2) Direction Determination Similarity determination at the R / B pixel position is performed using the Bayer image as in the conventional interpolation process. In the first embodiment, as will be described below, direction determination is performed by a general known method.

(1-2-1) Calculating Similarity According to the following equations (3) and (4), the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j for the R and B positions, respectively. ] Is calculated.
Cv [i, j] = {(| G [i, j-1] -Z [i, j] | + | G [i, j + 1] -Z [i, j] |) / 2 + | G [i, j-1] -G [i, j + 1] |} / 2 (3)
Ch [i, j] = {(| G [i-1, j] -Z [i, j] | + | G [i + 1, j] -Z [i, j] |) / 2 + | G [i-1, j] -G [i + 1, j] |} / 2 (4)
In the formulas (3) and (4), Z represents R or B.

(1-2-2) Similarity Determination Next, based on the following conditional expression (5), the similarities are compared and converted into direction indicators.
If | Cv [i, j] -Ch [i, j] | = <Th1 THEN HV [i, j] = 0 → Vertical / horizontal similarity unknown
else if Cv [i, j] <Ch [i, j] THEN HV [i, j] = 1 → vertical similarity
else HV [i, j] = -1 THEN Lateral Similarity (5)
The threshold value Th1 takes a value of around 10 when there are 256 gradations, and is set higher when there is a lot of image noise.

(1-3) Creation of Plural Blur Bayer Images In order to perform MTF matching between RGB color components, a plurality of blur images are generated for each color component. However, in this embodiment, since the arrangement density of the G component representing luminance in the Bayer array is high, it is considered that the MTF of the R and B components is originally inferior with the Bayer sampling. What is necessary is just to perform blurring (smoothing). Therefore, two types of blurred images are generated for the G component using a typical weak smoothing filter (FIG. 6A) and a strong smoothing filter (FIG. 6B) as shown in FIG. It should be noted that more detailed methods of blurring may be prepared, or an intermediate state may be considered by linear combination of the original image and the blurred image by these.

  In this way, a blurred image is generated using the two types of smoothing filters shown in FIG. 6, and “Bayer0: unblurred image” shown in FIG. 7A and “Bayer1: blurred image weakly” shown in FIG. 7B. (Image blurred using the weak smoothing filter shown in FIG. 6A), and “Bayer 2: blur strong image” shown in FIG. 7C (strong smoothing filter shown in FIG. 6B) 3 types of Bayer images are obtained. That is, smoothing is performed at a plurality of smoothing levels including the case where no blurring is performed.

(1-4) Generation of Temporary Color Difference Surfaces by Combination of Blur Using 3 G surfaces in each blurred image obtained in (1-3) Create a Cb plane. That is, Cr planes created on the basis of Bayer's R signal and G, G ', G "signals are respectively Cr0, Cr1, Cr2, and Cb created on the basis of Bayer's B signal and G, G', G" signals. The surfaces are Cb0, Cb1, and Cb2, respectively, and three types of Cr surfaces and three types of Cb surfaces are created. The following formulas (6) to (11) schematically represent these Cr0, Cb0, Cr1, Cb1, Cr2, and Cb2.

Cr0 = R- <G> (6)
Cb0 = B- <G> (7)
Cr1 = R- <G '> (8)
Cb1 = B- <G '> (9)
Cr2 = R- <G "> (10)
Cb2 = B- <G "> (11)

Among these, the generation of the Cr0 plane will be described. Specifically, a Cr0 surface is generated at the R position by the following equation (12), and the Cr0 surface is interpolated at a position other than the R position by the following equations (13) to (15). In addition, it can calculate similarly about each surface of other Cr1, Cr2, Cb0, Cb1, and Cb2.
If HV [i, j] = 1 THEN Cr0 [i, j] = R [i, j]-(G [i, j-1] + G [i, j + 1]) / 2
else if HV [i, j] =-1 THEN Cr0 [i, j] = R [i, j]-(G [i-1, j] + G [i + 1, j]) / 2
else Cr0 [i, j] = R [i, j]-(G [i, j-1] + G [i, j + 1] + G [i-1, j] + G [i + 1, j ]) / 4 (12)

Interpolate Cr0 surface at B position
Cr0 [i, j] = (Cr0 [i-1, j-1] + Cr0 [i-1, j + 1] + Cr0 [i + 1, j-1] + Cr0 [i + 1, j + 1 ]) / 4 ... (13)
Interpolate Cr0 surface at G position (same lines as R rows)
Cr0 [i, j] = (Cr0 [i-1, j] + Cr0 [i + 1, j]) / 2 (14)
Interpolate the Cr0 plane at the G position (same lines as B rows)
Cr0 [i, j] = (Cr0 [i, j-1] + Cr0 [i, j + 1]) / 2 (15)
In addition, about the production | generation method of a color difference surface, it is not limited to the method mentioned above, You may produce | generate by another well-known method.

(1-5) Creation of Color Index Next, a color index for comparing color changes of virtual color images that can be generated from the images blurred in (1-3) described above, that is, saturation Create indicators. As described above, when the MTF mismatch due to the axial chromatic aberration is matched, the coloring and color blurring at the edge portion due to the axial chromatic aberration acts in a decreasing direction. Therefore, in order to accurately measure how to decrease the color index, a color index is created in consideration of a combination of the two types of color difference surfaces Cr and Cb.

  The color index indicates the color characteristics of each pixel and is an index that is referred to in order to determine whether each pixel has low saturation or high saturation, and changes the MTF characteristics of the input image. This is an index for viewing the color response of a color image that can be generated when In other words, how the color response changes according to the degree of smoothing for changing the MTF characteristic, i.e., how much the color change depends on the degree of smoothing for changing the MTF characteristic. It is an index for monitoring how it changes.

  Here, the color response means a color response when observing what color is changed as a result of smoothing at an arbitrary smoothing degree, and the change in color response is a plurality of different smoothing. This means the change in color response between the two when smoothed with a degree of conversion.

  The smoothing described here includes the meaning of an intermediate process for obtaining a correction signal for sharpening other color components via the equations (1) and (2). 4 shows the color response of the finally obtained R′GB ′ image when the blur amount in FIG. 4 is changed little by little. That is, the smoothing process also refers to an unsharp process for obtaining an MTF corrected image by an unsharp mask process. More specifically, it can be said that a smoothing process is performed on the G component and a sharpening process is performed on the R and B components.

If the color index to be created is Cdiff, the basic form is defined by the following equation (16). In the present embodiment, Cdiff is defined as a color index. However, for example, only a color difference may be defined and used as a color index.
Cdiff [i, j] = | Cr [i, j] | + | Cb [i, j] | + | Cr [i, j] -Cb [i, j] | (16)

In the color index defined by Expression (16), the combination of color indices to be evaluated for each of the two types of color difference surfaces Cr and Cb is expressed by the following Expressions (17) to (25). There are 9 ways in total.
Cdiff_r0b0 [i, j] = | Cr0 * [i, j] | + | Cb0 * [i, j] | + | Cr0 * [i, j] -Cb0 * [i, j] | (17)
Cdiff_r1b0 [i, j] = | Cr1 [i, j] | + | Cb0 [i, j] | + | Cr1 [i, j] -Cb0 [i, j] | (18)
Cdiff_r0b1 [i, j] = | Cr0 [i, j] | + | Cb1 [i, j] | + | Cr0 [i, j] -Cb1 [i, j] | (19)
Cdiff_r1b1 [i, j] = | Cr1 [i, j] | + | Cb1 [i, j] | + | Cr1 [i, j] -Cb1 [i, j] | (20)
Cdiff_r2b0 [i, j] = | Cr2 [i, j] | + | Cb0 [i, j] | + | Cr2 [i, j] -Cb0 [i, j] | (21)
Cdiff_r0b2 [i, j] = | Cr0 [i, j] | + | Cb2 [i, j] | + | Cr0 [i, j] -Cb2 [i, j] | (22)
Cdiff_r2b1 [i, j] = | Cr2 [i, j] | + | Cb1 [i, j] | + | Cr2 [i, j] -Cb1 [i, j] | (23)
Cdiff_r1b2 [i, j] = | Cr1 [i, j] | + | Cb2 [i, j] | + | Cr1 [i, j] -Cb2 [i, j] | (24)
Cdiff_r2b2 [i, j] = | Cr2 [i, j] | + | Cb2 [i, j] | + | Cr2 [i, j] -Cb2 [i, j] | (25)

  When only the R component MTF is low, only the G component on the Cr generation side needs to be blurred, and when only the B component MTF is low, only the G component on the Cb generation side needs to be blurred. If the MTF is low, the G components on both sides must be blurred.

  Here, only Cdiff_r0b0 without blurring of the G component for both R and B does not use the value that merely generated the Cr and Cb components, and further uses Cr0 * and Cb0 * for which color difference plane correction processing has been performed on the values. Used. This is a stable color evaluation on the non-blurred side, which is more likely to provide a safer solution than the risky blurring side, with respect to the variation factors of the color index due to the occurrence of false colors due to the image structure. This is to ensure that Further, even if there is an image structure such as an opposite color pair, intentionally lowering the saturation of Cdiff_r0b0 reduces the saturation due to the image structure unlike MTF matching of axial chromatic aberration. This is also for changing to a color index evaluation value that prevents the image structure destruction in opposition to the possible blur side Cdiff.

As an example of the color difference plane correction process for Cr0 and Cb0, there is a low-pass process as shown in the following equation (26).
Cr * [i, j]
= {36 × Cr [i, j]
+ 24 × (Cr [i-2, j] + Cr [i + 2, j] + Cr [i, j-2] + Cr [i, j + 2])
+ 16 × (Cr [i-2, j-2] + Cr [i + 2, j-2] + Cr [i-2, j + 2] + Cr [i + 2, j + 2])
+ 6 × (Cr [i-4, j] + Cr [i + 4, j] + Cr [i, j-4] + Cr [i, j + 4])
+ 4 × (Cr [i-2, j-4] + Cr [i + 2, j-4] + Cr [i-2, j + 4] + Cr [i + 2, j + 4]
Cr [i-4, j-2] + Cr [i + 4, j-2] + Cr [i-4, j + 2] + Cr [i + 4, j + 2])
+ (Cr [i-4, j-4] + Cr [i + 4, j-4] + Cr [i-4, j + 4] + Cr [i + 4, j + 4])} / 256 (26)

  This color difference surface correction process is not limited to the low-pass process shown in Expression (26), and other methods such as a median filter may be used, or the filter range may be changed. Moreover, although the example which calculates Cr * [i, j] was demonstrated in Formula (26), Cb * [i, j] is similarly computable.

  Note that very subtle color change information with respect to the amount of blurring due to MTF mismatch due to aberrations may be lost by extra filtering processing, so it is preferable to compare in the state immediately after the color difference plane generation. Therefore, it is defined that the color difference plane correction processing is not performed except for Cdiff_r0b0. Also, Cdiff_r0b0 may be defined not to perform the color difference plane correction process.

(1-6) Determination of blurring amount It is considered that the color difference surface generated by eliminating the MTF mismatch due to the longitudinal chromatic aberration reduces the color generation due to the color blur and changes to the low saturation side. In the blurring process described above, since the color index that has been pre-measured to shift the factor of interpolation false color that occurs with the image structure to the low saturation side is generated, the color index is compared with each other. A combination of Cr and Cb having a blur amount that provides the lowest saturation level can be finally used as a blur amount to be used for image generation. That is, according to the conditional expression shown in the following equation (27), the smallest value among Cdiff_r0b0 to Cdiff_r2b2 is extracted for each pixel, and the blurring amounts used to generate Cr and Cb at that time are respectively By determining whether each pixel is G, G ′, or G ″, it is possible to finally determine a discrete blurring amount that is used for image generation, that is, a degree of smoothing.

(G blur amount at Cr generation, G blur amount at Cb generation)
= arg min (Cdiff_r0b0, Cdiff_r1b0, ..., Cdiff_r2b2)
(G, G ', G "for Cr),
(G, G ', G "for Cb) (27)
Thus, each of the generated color indexes can be evaluated on a pixel basis, and it can be determined which color component is to be blurred when generating which color difference plane for each pixel. This can be dealt with even when the state of axial chromatic aberration changes depending on the subject distance.

(1-7) Actual Color Difference Surface Generation The color difference component Cr [i, used in the final output image is obtained by using the discrete blur amount G component determined for each pixel in the above-described process as information on the difference in MTF characteristics. j] and Cb [i, j] are obtained as follows. First, the Cr value at the R pixel position is set by the following equation (28), and the Cb value is set at the B pixel position by the following equation (29).
Cr [i, j] = one of {Cr0, Cr1, Cr2} R position (28)
Cb [i, j] = one of {Cb0, Cb1, Cb2} B position (29)
Then, the Cr surface and the Cb surface are interpolated by the above-described equations (13) to (15). Thereafter, color difference surface correction is performed using various known color difference surface correction methods. For example, an adaptive color difference correction method for selecting whether to correct or not according to conditions is used. This is to incorporate a false color suppression process that occurs in normal Bayer interpolation. Thus, the Cr surface and the Cb surface from which the longitudinal chromatic aberration is removed, that is, the actual color difference surface is generated.

(1-8) Actual luminance plane generation An actual luminance plane is generated using the original Bayer signal without blurring. This is because a luminance surface that is not significantly affected by axial chromatic aberration can be restored in a state in which a sharp resolution is maintained by using the original Bayer signal. In the present embodiment, as shown in the following equation (30), the process is performed using the luminance as the G plane, but the process may be performed using the Y plane as the luminance. The luminance plane can be generated by various known methods. Further, as disclosed in International Publication No. WO2002 / 071761, it may be generated directly from the Bayer surface.

R / B position on the Bayer surface
if HV [i, j] = 1 THEN Gout [i, j] = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 × Z [i, j] -Z [i, j-2] -Z [i, j + 2]) / 4
else if HV [i, j] =-1 THEN Gout [i, j] = (G [i-1, j] + G [i + 1, j]) / 2
+ (2 × Z [i, j] -Z [i-2, j] -Z [i + 2, j]) / 4
else Gout [i, j] = (G [i, j-1] + G [i, j + 1] + G [i-1, j] + G [i + 1, j]) / 4
+ (4 × Z [i, j] -Z [i, j-2] -Z [i, j + 2] -Z [i-2, j] -Z [i + 2, j]) / 8 (30)
In Equation (30), Z is Z = R at the R position and Z = B at the B position. Further, the G position Gout [i, j] on the Bayer plane can be obtained by substituting the Bayer signal as it is.

(1-9) Color system conversion From the three color information of the Cr surface and Cb surface from which axial chromatic aberration has been removed and the G surface retaining sharpness in the above-described processing, the following equations (31) and ( 32), conversion to the RGB color system is performed.
Rout [i, j] = Cr [i, j] + Gout [i, j] (31)
Bout [i, j] = Cb [i, j] + Gout [i, j] (32)

  With the above processing, in the read Bayer image, if any of the RGB color components has axial chromatic aberration, and color blur occurs with this, the MTF characteristic mismatch between the color components is corrected, Color blur due to axial chromatic aberration can be eliminated. Then, the RGB image in which the color blur due to the longitudinal chromatic aberration is eliminated is output to the monitor 113 and displayed.

  FIG. 8 is a flowchart showing the operation of the image processing apparatus 110 according to the first embodiment. The processing shown in FIG. 8 is executed by the control device 111 as a program that is activated when an image captured by the image sensor 102 via the lens 101 is stored in the image memory 112. In step S10, as described above in (1-1), the Bayer image captured by the image sensor 102 is read from the image memory 112, and the process proceeds to step S20. In step S20, as described above in (1-2), the direction determination at the R / B pixel position is performed using the Bayer image. Then, it progresses to step S30.

  In step S30, the plurality of blur Bayer image creation processes described above in (1-3) are executed, and the process proceeds to step S40. In step S40, a plurality of provisional color difference plane generation processes by the blur combination described in (1-4) are executed, and the process proceeds to step S50. In step S50, the color index creating process described in (1-5) is executed, and the process proceeds to step S60. In step S60, as described above in (1-6), the blurring amount to be finally used for image generation is determined, and the process proceeds to step S70.

  In step S70, the actual color difference plane generation process described in (1-7) is executed, and the process proceeds to step S80. In step (1-8), the actual luminance plane generation process is executed. Then, the process proceeds to step S90, and the color system conversion process is performed as described above in (1-9), and the Cr surface and the Cb surface from which the longitudinal chromatic aberration has been removed, and the G surface 3 that maintains sharpness. Conversion from one color information to the RGB color system is performed. Thereafter, the process proceeds to step S100, an image in which the color blur associated with the longitudinal chromatic aberration is eliminated is output to the monitor 113, and the process ends.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The color difference component is generated after the MTF characteristic mismatch between the color components caused by the longitudinal chromatic aberration is matched with the color component on the lower side by blurring the color component on the higher MTF side. Accordingly, it is possible to suppress color bleeding that occurs due to a difference in MTF difference due to axial chromatic aberration.
(2) The correction amount for performing axial chromatic aberration correction on the captured image is determined based on the captured image itself. As a result, even if the axial chromatic aberration has a characteristic that changes depending on the distance to the subject, an appropriate correction amount can be determined for each pixel to perform the axial chromatic aberration correction.
(3) Furthermore, since an appropriate correction amount can be determined using the captured image itself and axial chromatic aberration correction can be performed in this way, even if information regarding the imaging optical system is unknown, On-axis chromatic aberration correction can be performed.
(4) In creating the color index, only Cdiff_r0b0 without blurring of the G component in both R and B is not subjected to the value that merely generated the Cr and Cb components, but is further subjected to color difference plane correction processing. Cr0 * and Cb0 * were used. As a result, the non-blurred side, which is more likely to provide a safer solution than the highly dangerous blurring processing side, can be stably prevented with respect to the variation factors of the color index received due to the generation of false colors due to Bayer sampling and the image structure. Color can be evaluated. In addition, even if there is an image structure such as an opposite color pair, intentionally lowering the saturation of Cdiff_r0b0 reduces the saturation due to the image structure unlike MTF matching of axial chromatic aberration. It can be changed to a color index evaluation value so as to prevent destruction of the image structure against the possible blur side Cdiff.

-Second embodiment-
In the first embodiment described above, in (1-7) actual color difference plane generation, the color difference components Cr [i, j] and Cb [i, j] are obtained by the expressions (28) and (29), and the expression After interpolating the Cr surface and the Cb surface according to (13) to (15), the processing shown in (1-8) actual luminance surface generation and (1-9) color system conversion is performed. explained.

  On the other hand, in the second embodiment, after obtaining the color difference components Cr [i, j] and Cb [i, j] by the equations (28) and (29), the Cr surface at the R position, the B position (1-8) actual luminance plane generation at that position and (1-9) color system conversion processing are performed using only the Cb plane of rewrite. Thus, an MTF-matched Bayer image before interpolation processing, that is, a Bayer image in which axial chromatic aberration is eliminated can be obtained. Thereafter, an interpolation process is performed on the Bayer image from which the longitudinal chromatic aberration has been eliminated to obtain an RGB image after interpolation.

  1 is a block diagram of the digital camera 100 equipped with the image processing apparatus 110 shown in FIG. 1, FIG. 2 is a diagram showing the principle of color blurring caused by axial chromatic aberration, and FIG. Since the diagram showing the principle of correcting the axial chromatic aberration by matching the characteristics is the same as in the first embodiment, the description thereof is omitted.

  After obtaining the color difference components Cr [i, j] and Cb [i, j] by the equations (28) and (29), the Cr surface before interpolation of each surface and the Cb surface are used (1-8). By performing each process of actual luminance plane generation and (1-9) color system conversion, a Bayer image in which MTF is matched, that is, a Bayer image in which axial chromatic aberration is corrected, is output as shown in FIG. Is done. By performing a known interpolation process on the Bayer image shown in FIG. 9, an RGB image after interpolation can be obtained.

  FIG. 10 is a flowchart illustrating processing of the image processing apparatus 110 according to the second embodiment. The processing shown in FIG. 8 is executed by the control device 111 as a program that is activated when an image captured by the image sensor 102 via the lens 101 is stored in the image memory 112. In FIG. 10, the same processing numbers as those in the first embodiment shown in FIG. 8 are given the same step numbers, and differences will be mainly described.

  In step S71, the color difference components Cr [i, j] and Cb [i, j] are obtained by the equations (28) and (29), and the process proceeds to step S80. Thereafter, in step S91, conversion from the three color information of the Cr surface and Cb surface before interpolation and the G surface retaining sharpness into the RGB color system is performed by equations (31) and (32). Thereafter, the process proceeds to step S92. In step S92, a Bayer image in which axial chromatic aberration is corrected is output, and the process proceeds to step S93. A known Bayer interpolation process is performed on the Bayer image to obtain an RBG image in which axial chromatic aberration is corrected. .

  According to the second embodiment described above, the execution position of the interpolation process in the axial chromatic aberration correction process is changed from the first embodiment, and a Bayer image in which the MTF is matched is output once. Interpolation processing was performed later. Accordingly, the axial chromatic aberration correction process and the interpolation process can be handled as separate processes, and various interpolation processes can be applied to the MTF-matched Bayer image.

-Third embodiment-
In the first and second embodiments, axial chromatic aberration correction has been described for image data before interpolation, such as a Bayer image captured by a single-plate color image sensor, but in the third embodiment, 3 The axial chromatic aberration of the color image photographed by the plate type color image sensor and the axial chromatic aberration of the interpolated color image are corrected. That is, correction of axial chromatic aberration of color image data in which all R, G, and B color component information exists in each pixel will be described.

  Specifically, in order to divide the original color image data into luminance and chrominance components and replace the chrominance component data with data without axial chromatic aberration, a plurality of types of color component data are provided as in the first embodiment. Blur. Then, a combination of blurs that gives a color index with the lowest saturation is searched for from among a set of color difference components that can be combined, and is replaced with a color difference component obtained through the blur.

  1 is a block diagram of the digital camera 100 equipped with the image processing apparatus 110 shown in FIG. 1, a diagram showing the principle of color blurring due to axial chromatic aberration shown in FIGS. 2 and 3, and FIG. The diagram showing the principle of correcting the axial chromatic aberration by matching the MTF characteristics is the same as in the first embodiment, and thus the description thereof is omitted. However, when using a three-plate color image sensor to obtain a color image in which R, G, and B components are aligned for each pixel, the image sensor 102 is replaced with a three-plate color image sensor. To do. In the third embodiment, not only the blur for the G component but also the blur for the R component and the B component are performed. As a result, even when there is axial chromatic aberration in the G component, it can be corrected.

(2-1) Color Image Input In the third embodiment, a color image captured by a three-plate color image sensor or a color image that has been previously subjected to interpolation processing is stored in the image memory 112 as R [i, j ], G [i, j], and B [i, j] are stored in a state in which they are prepared, and these color images are read out and subjected to image processing.

(2-2) Luminance component saving by color system conversion The read color image is color system converted by the following equation (33) to save the luminance component. That is, the luminance component of the image in the state before blurring is saved in the memory space of the control device 111 in a process described later.
Y [i, j] = (R [i, j] + 2 × G [i, j] + B [i, j]) / 4 (33)

(2-3) Creation of Plural Blur R, G, B Surface Images In the third embodiment, the smoothed filter shown in FIG. 11 is used to create a blurred image (R, G, B for each component of RGB. (Surface image). In this embodiment, for the sake of simplification of explanation, an example in which only one smoothing filter shown in FIG. 11 is used will be described. However, similarly to the first embodiment, a weak smoothing filter and a strong filter are used. R, G, and B plane images may be created using two types of smoothing filters.

  Then, as shown in FIG. 12, FIGS. 12D to 12F created using the non-blurred images for the RGB components of FIGS. 12A to 12C and the smoothing filter of FIG. ) And the blurred image are used to perform the following processing.

(2-4) Generation of Temporary Color Difference Surfaces by Combination of Blur The above six RGB planes, that is, non-blurred images of FIGS. 12 (a) to 12 (c) and blurs of FIGS. 12 (d) to (f). Using the images, three Cr surfaces and three Cb surfaces are created for each pixel by the following equations (34) to (39).
Cr00 = RG (34)
Cb00 = BG (35)
Cr01 = RG '(36)
Cb01 = BG '(37)
Cr10 = R'-G (38)
Cb10 = B'-G (39)

(2-5) Creation of color index For each Cr and Cb blurring method created by the equations (34) to (39), the color index to be evaluated, that is, the combination of the indexes related to saturation, is expressed by the following equation (40) There are nine ways in total as represented by (48).
Cdiff_r00b00 [i, j] = | Cr00 [i, j] | + | Cb00 [i, j] | + | Cr00 [i, j] -Cb00 [i, j] | (40)
Cdiff_r01b00 [i, j] = | Cr01 [i, j] | + | Cb00 [i, j] | + | Cr01 [i, j] -Cb00 [i, j] | (41)
Cdiff_r00b01 [i, j] = | Cr00 [i, j] | + | Cb01 [i, j] | + | Cr00 [i, j] -Cb01 [i, j] | (42)
Cdiff_r01b01 [i, j] = | Cr01 [i, j] | + | Cb01 [i, j] | + | Cr01 [i, j] -Cb01 [i, j] | (43)
Cdiff_r10b00 [i, j] = | Cr10 [i, j] | + | Cb00 [i, j] | + | Cr10 [i, j] -Cb00 [i, j] | (44)
Cdiff_r10b01 [i, j] = | Cr10 [i, j] | + | Cb01 [i, j] | + | Cr10 [i, j] -Cb01 [i, j] | (45)
Cdiff_r01b10 [i, j] = | Cr01 [i, j] | + | Cb10 [i, j] | + | Cr01 [i, j] -Cb10 [i, j] | (46)
Cdiff_r00b10 [i, j] = | Cr00 [i, j] | + | Cb10 [i, j] | + | Cr00 [i, j] -Cb10 [i, j] | (47)
Cdiff_r10b10 [i, j] = | Cr10 [i, j] | + | Cb10 [i, j] | + | Cr10 [i, j] -Cb10 [i, j] | (48)

  In the third embodiment, as shown in Expression (40), Cdiff_r00b00 is defined not to perform the color difference plane correction process, but as in Expression (17) in the first embodiment. , Cr, and Cb components may be subjected to correction processing such as low-pass processing shown in Expression (26), for example.

(2-6) Determination of blurring amount Similar to (1-6) in the first embodiment, the color index calculated by the equations (40) to (48) is compared, and the blurring amount giving the minimum value The combination of Cr and Cb is finally determined as the blurring amount used for image generation. That is, according to the following equation (49), the components having the smallest value among Cdiff_r00b00 to Cdiff_r10b10 are extracted for each pixel, and each component of RGB is shown in FIG. 11 in order to generate Cr and Cb at that time. Determine whether it is necessary to blur with a smoothing filter.

(Cr blurring amount for Cr generation, G blurring amount for Cr generation, G blurring amount for Cb generation, B blurring amount for Cb generation)
= arg min (Cdiff_r00b00, Cdiff_r01b00, ..., Cdiff_r10b10)
(R, R 'for Cr)
(G, G 'for Cr)
(G, G 'for Cb)
(B, B 'for Cb) (49)

(2-7) Generation of Replacement Color Difference Surface Based on the result of (2-6) described above, the replacement value Cr [i, j of the color difference component used for the final output image is obtained by the following equations (50) and (51). ] And Cb [i, j] are set.
Cr [i, j] = one of {Cr00, Cr01, Cr10} (50)
Cb [i, j] = one of {Cb00, Cb01, Cb10} (51)

(2-8) Color Conversion Next, with the following equations (52) to (54), the luminance component saved in the above (2-2) and the axial chromatic aberration are corrected by the processing of (2-7). The final color image is generated by integrating the color difference components.
R [i, j] = Y [i, j] + (3/4) × Cr [i, j] − (1/4) × Cb [i, j] (52)
G [i, j] = Y [i, j]-(1/4) × Cr [i, j]-(1/4) × Cb [i, j] (53)
B [i, j] = Y [i, j]-(1/4) × Cr [i, j] + (3/4) × Cb [i, j] (54)

  With the above processing, if there is axial chromatic aberration in any of the RGB color components in the color image, and color blur occurs with this, the mismatch of the MTF characteristics between the color components is corrected, and the axial Color blur due to chromatic aberration can be eliminated. Then, the color image in which the color blur due to the longitudinal chromatic aberration is eliminated is output to the monitor 113 and displayed.

  FIG. 13 is a flowchart illustrating the operation of the image processing apparatus 110 according to the third embodiment. The process illustrated in FIG. 13 is executed by the control device 111 as a program that is activated when an image captured by the image sensor 102 via the lens 101 is stored in the image memory 112. In step S110, as described above in (2-1), the color image is read from the image memory 112, and the process proceeds to step S120. In step S120, as described above in (2-2), the read color image is subjected to color system conversion, and the luminance component is saved in the memory space of the control device 111. Thereafter, the process proceeds to step S130.

  In step S130, the plurality of blur R, G, and B plane image creation processes described above in (2-3) are executed, and the process proceeds to step S140. In step S140, a plurality of provisional color difference plane generation processes by the combination of blurs described above in (2-4) are executed, and the process proceeds to step S150. In step S150, the color index creation processing described above in (2-5) is executed, and the flow advances to step S160. In step S160, as described above in (2-6), the blurring amount to be finally used for image generation is determined, and the process proceeds to step S170.

  In step S170, the replacement color difference plane generation process described above in (2-7) is executed, and the process proceeds to step S180. As described above in (2-8), the luminance component saved in (2-2). Then, the final output image is generated by integrating the color difference components whose axial chromatic aberration is corrected in the processing of (2-7). Thereafter, the process proceeds to step S190, where a color image from which color blur due to axial chromatic aberration has been eliminated is output and displayed on the monitor 113, and the process ends.

  According to the third embodiment described above, unlike the prior art, the low MTF color component information is corrected using other color component information of the high MTF, so that sharp axial chromatic aberration is obtained. Can be corrected. In addition to blurring for the G component, blurring for the R component and the B component is also performed. As a result, even when there is axial chromatic aberration in the G component, it can be corrected.

-Fourth embodiment-
In the fourth embodiment, the digital camera 100 focuses on a subject separated by a predetermined distance, and corrects the general tendency of axial chromatic aberration that the subject image covers at the focus position of the lens 101. 1 is a block diagram of the digital camera 100 equipped with the image processing apparatus 110 shown in FIG. 1, a diagram showing the principle of color blurring due to axial chromatic aberration shown in FIGS. 2 and 3, and FIG. The diagram showing the principle of correcting the axial chromatic aberration by matching the MTF characteristics is the same as in the first embodiment, and thus the description thereof is omitted.

  First, axial chromatic aberration information inherently possessed by the lens 101 is examined in advance. That is, a reference subject placed at a predetermined distance is photographed, and the difference in MTF characteristics between RGB for each of the variable parameters that the lens 101 can take, such as the lens type, zoom position, aperture value, focus position, etc. To investigate. Then, while changing each of these variable parameters, for example, an imaging chart for evaluation as shown in FIG. 14 is imaged, and the appearance of the black line signal level at that time is observed, and the difference in MTF characteristics between RGB Is calculated.

  Based on this calculation result, a uniform LPF necessary to match the blurred color component is identified for the sharpest color component in each variable parameter setting state. As shown in FIG. For each setting situation, the position of the subject (distance to the subject) and the specified type of uniform LPF are associated with each other and stored in the memory of the control device 111.

  Then, in order to correct the axial chromatic aberration in the portion where the main subject in the target image captured by the image sensor 102 is reflected, the following processing is performed. First, the main subject position in the target image is detected. At this time, taking into account that the main subject generally exists at or near the focus position (focus position), first, the above-mentioned variable parameters are obtained from the image data, and the focus in the target image is obtained. Identify the location. Each variable parameter is stored as external information related to the optical system in, for example, Exif information in captured image data, and the control device 113 can specify the focus position with reference to the image data.

  Assuming that the main subject is spreading around the specified focus position, the predetermined range around the focus position is set as the target range, edge detection is performed within that range, and the detected edges are grouped. Thus, the object is extracted, and the object is regarded as a main subject, and axial chromatic aberration correction is performed on a range including the object. Specifically, the distance from the specified focus position to the main subject is determined, and further, the setting state of the variable parameter at the time of imaging is determined from the image data. Then, by referring to the table shown in FIG. 15 corresponding to the determined setting state of the variable parameter, the uniform LPF associated with the distance to the main subject is selected.

  Then, the target range described above is blurred by the selected uniform LPF, and axial chromatic aberration correction within the target range is performed as described above in the first embodiment and the third embodiment. Thus, based on the setting state of the variable parameters stored as external information related to the optical system, a uniform LPF suitable for the state is determined, and an appropriate degree is set for an appropriate color component using the uniform LPF. Can be smoothed.

  FIG. 16 is a flowchart illustrating the operation of the image processing apparatus 110 according to the fourth embodiment. The process illustrated in FIG. 16 is executed by the control device 111 as a program that starts when an image captured by the image sensor 102 via the lens 101 is stored in the image memory 112. In step S210, an image is read from the image memory 112, and the process proceeds to step S220. In step S220, the focus position is specified from the image data as described above, and the process proceeds to step S230.

  In step S230, the setting state of the variable parameter at the time of imaging is determined from the image data, and the focus position, that is, the distance to the main subject is associated with the table shown in FIG. Select the uniform LPF. Thereafter, the process proceeds to step S240, the image read using the selected uniform LPF is blurred, and the process proceeds to step S250. In step S250, it is determined whether the input image, that is, the image read from the image memory 112 is a single-plate color image sensor output image such as a Bayer array or an interpolated color image.

  If it is determined that the input image is a single-plate color image sensor output image such as a Bayer array, the process proceeds to step S260, and the processes in steps S70 to S100 of FIG. 8 in the first embodiment described above. Is executed to correct the axial chromatic aberration of the image and output it to the monitor 113, and then the process ends. On the other hand, if it is determined that the input image is an interpolated color image, the process proceeds to step S270, and the processes in steps S170 to S190 in FIG. 13 in the third embodiment described above are executed. Then, the axial chromatic aberration of the color image is corrected and output to the monitor 113, and then the process is terminated.

According to the fourth embodiment described above, the following operational effects can be obtained in addition to the operational effects of the first or third embodiment.
(1) The main subject is extracted based on the focus position information included in the image data, the distance to the extracted main subject is determined, and the table shown in FIG. The selected uniform LPF was selected to blur the image in the range where the main subject exists. Accordingly, since it is only necessary to select a preset uniform LPF based on the distance to the main subject and blur the image, various processes for determining the blur amount are not necessary, and the process can be speeded up.
(2) When determining the distance to the main subject, a predetermined range around the focus position included in the image data is set as a target range, edge detection is performed within the range, and the detected edges are grouped. The object is extracted by the above and the object is regarded as the main subject. This makes it possible to accurately extract the main subject in consideration of the fact that the main subject generally exists at or near the focus position.

-Modification-
The image processing apparatus according to the above-described embodiment can be modified as follows.
(1) In the first and second embodiments described above, in the image sensor 102, for example, the most representative R (red), G (green), and B (blue) color filters of a single-plate color image sensor. Are arranged in a Bayer array, and the image data captured by the image sensor 102 is represented by the RGB color system, and each pixel constituting the image data has color information of any one of RGB color components. An example was given assuming that exists. However, the present invention is not limited to this, and a two-plate color image sensor may be used as the image sensor 102, and color information of any two color components of RGB may exist in each pixel constituting the image data. . That is, the image sensor 102 only needs to capture an image in which at least one color component is missing in each pixel having a plurality of color components having different MTF characteristics on the imaging surface.

(2) In the first to third embodiments described above, when determining the blurring amount in (1-6) and (2-6), the discrete blurring amount used for image generation is finally used. An example of determination has been described. However, the present invention is not limited to this, and a continuous blur amount may be determined. For example, Cr1 and Cr2 are collectively expressed as Cr_blur, and linear combination is performed as shown in the following equations (55) and (56). Here, an example in which Cr_blur and Cb_blur are generated using a linear combination state (weighting coefficients (s, t)) without blurring G ″ will be described, but G ′ is added in the same manner. It can also be a linear combination.
Cr_blur = R- {s * <G "> + (1-s) * <G>} (55)
Cb_blur = B- {t * <G "> + (1-t) * <G>} (56)
However, 0 = <(s, t) = <1.

Based on Cr_blur and Cb_blur generated in Expressions (55) and (56), a color index Cdiff_blue corresponding to the blur side is generated according to Expression (57) below.
Cdiff_blur = | Cr_blur [i, j] | + | Cb_blur [i, j] | + | Cr_blur [i, j] -Cb_blur [i, j] | (57)
Then, a combination of (s, t) that gives the minimum value of Cdiff_blur is two-dimensionally searched as shown in FIG. Next, the minimum value of Cdiff_blur and the non-blurring side color index Cdiff_r0b0 are compared to determine whether to use the blurring amount (s, t) or not. This comparison is performed only when only Cdiff_r0b0 is specially handled.

When the image is blurred by the continuous blur amount determined in this way, the color difference component used for the final output image in the actual color difference plane generation process described in (1-7) of the first embodiment. Cr [i, j] and Cb [i, j] may be obtained by the following equations (58) and (59) instead of equations (28) and (29).
Cr [i, j] = one of {Cr0, Cr_blur (s; 0 = <s = <1)} R position (58)
Cb [i, j] = one of {Cb0, Cb_blur (t; 0 = <t = <1)} B position (59)

(3) In the first embodiment described above, an example is described in which normal interpolation processing is performed by determining a blurring amount after comparing a plurality of combinations of color difference surfaces immediately after being generated from a Bayer surface. did. However, the present invention is not limited to this, and the amount of blur may be determined after performing normal color interpolation processing and adaptive color difference surface correction processing for color difference planes corresponding to a plurality of types of blurring.

(4) In the first to third embodiments, the blurring method is determined in pixel units by looking at the color responses of the generated color components for a plurality of blurring methods in pixel units. However, the present invention is not limited to this, and it can be assumed that there will be little sudden change at the pixel level due to the cause of axial chromatic aberration, so that a uniform method of blurring is selected in a slightly larger range. The processing may be performed in units of blocks extending over a plurality of pixels.

  For example, a Bayer image read from the image memory 112 is divided into a plurality of blocks each having a size of 64 pixels × 64 pixels, and a color index in units of pixels is calculated as described above in the first embodiment, and each block is calculated. The average value of the color index is calculated. Based on the average value of the color index in the block, the color index is compared in units of blocks, and a set of blurring methods that give the minimum value is applied in common to the block, and axial chromatic aberration is applied to each block. Correction may be performed.

  In this case, since axial chromatic aberration correction is performed in units of blocks, an unnatural color difference may occur on the boundary between the blocks. In order to avoid this, the following modifications may be made. That is, after calculating a color index, an average value of a plurality of pixels of the color index (for example, 16 × 16 or 64 × 64) is taken, and these are calculated in the same manner as in (1-6) or (2-6). Try to evaluate the units. As a result, a uniform blur amount is determined in the average range, and unnaturalness at the block boundary can be eliminated.

(5) In the above-described third embodiment, an example in which axial chromatic aberration correction is performed after a color image represented by the RGB color system is converted to image data in the YCbCr format has been described. However, the present invention is not limited to this, and axial chromatic aberration correction may be performed with the data in the RGB format as it is. That is, the MTF matching processing (Equations (1) and (2)) in FIG. 4 may be directly performed for each of R, G, and B.

(6) In the above-described fourth embodiment, the example in which the main subject is extracted based on the focus position information included in the image data and only the range where the main subject exists has been described. However, the present invention is not limited to this, and axial chromatic aberration correction may be performed on the entire image.

(7) In the above-described fourth embodiment, the example in which the distance to the subject is specified based on the focus position included in the image data has been described. However, the present invention is not limited to this, and the distance to the subject may be specified based on the camera shooting modes such as “close-up shooting”, “portrait shooting”, and “landscape shooting” included in the image data. For example, in the close-up shooting mode, it may be determined that the subject is in the vicinity and the correspondence table shown in FIG. 15 is referred to.

(8) In the first to fourth embodiments described above, various image processing is performed by the control device 111 mounted on the digital camera 100, and axial chromatic aberration correction of an image captured by the image sensor 102 via the lens 101 is performed. An example of performing is described. However, the present invention is not limited to this, for example, after installing a program for executing the above-described image processing in an electronic device such as a personal computer, and capturing an image captured by a digital camera into the electronic device via various interfaces. On-axis chromatic aberration correction may be performed. Such a program is provided to an electronic device such as a personal computer via a recording medium such as a CD or an electric communication line such as the Internet.

(9) In the above-described first to fourth embodiments, the example in which the MTF is matched and the axial chromatic aberration is corrected according to the principle shown in FIG. 4 is described. However, the present invention is not limited to this, and FIG. As shown, a luminance plane Y (FIG. 18D) may be generated from each RGB color component, and the luminance plane Y may be smoothed to match the MTF.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired.

It is a block diagram which shows the structure of one Embodiment at the time of mounting an image processing apparatus in a digital camera. It is a 1st figure explaining the principle which the color bleed by axial chromatic aberration generate | occur | produces. It is a 2nd figure explaining the principle which the color bleed by axial chromatic aberration generate | occur | produces. It is a figure which shows the principle which correct | amends the axial chromatic aberration of each component of R and B using another color component (G component). It is a figure which shows the specific example of a Bayer image. It is a figure which shows the specific example of the smoothing filter used in 1st Embodiment. It is a figure which shows the specific example of the "non-blurred image" in a 1st embodiment, the "blurred weak image", and the "blur strong image". It is a flowchart figure which shows operation | movement of the image processing apparatus 110 in 1st Embodiment. It is a figure which shows the specific example of the Bayer image which matched MTF. It is a flowchart figure which shows operation | movement of the image processing apparatus 110 in 2nd Embodiment. It is a figure which shows the specific example of the smoothing filter used in 3rd Embodiment. It is a figure which shows the specific example of the "image without a blur" and the "image with a blur" in 3rd Embodiment. It is a flowchart figure which shows operation | movement of the image processing apparatus 110 in 3rd Embodiment. It is a figure which shows the specific example of the imaging | photography chart for evaluation in 4th Embodiment. It is a figure which shows the correspondence of the distance to the to-be-photographed object and LPF to be used in 4th Embodiment. It is a flowchart figure which shows operation | movement of the image processing apparatus 110 in 4th Embodiment. It is a figure which shows the specific example of the determination method of the continuous blurring amount. It is a figure which shows the principle which adjusts an axial chromatic aberration by matching the MTF characteristic in a modification (9).

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Digital camera 101 Lens 102 Image pick-up element 110 Image processing apparatus 111 Control apparatus 112 Image memory 113 Monitor

Claims (13)

A first image captured through the optical system, in which at least one of a plurality of color components is missing in one pixel and the MTF characteristics are different between the reference color component and at least one other missing color component on the imaging surface, is MTF An image processing apparatus for converting to a second image having matched characteristics,
In a pixel having a missing color component of the first image, information regarding a difference in MTF characteristics between the missing color component and the reference color component is obtained, and the second image is obtained using the obtained information. An image processing apparatus comprising image generating means for generating. The image processing apparatus according to claim 1.
The image generation means interpolates at least one missing color component for each pixel while correcting the missing color component to match the MTF characteristic of the reference color component using the obtained information. An image processing apparatus that generates a second image. The image processing apparatus according to claim 1.
The image generation unit corrects the MTF characteristic of the missing color component of the first image to match the MTF characteristic of the reference color component using the obtained information, and An image processing apparatus that generates the second image having the same color component arrangement. The image processing apparatus according to claim 3.
An image processing apparatus, further comprising interpolation means for interpolating a missing color component of the second image. In the image processing apparatus according to any one of claims 2 to 4,
The image generation means includes smoothing means for smoothing pixels having the reference color component so that the reference color component approaches the MTF characteristic of at least one missing color component having a different MTF characteristic of the first image. ,
An image processing apparatus that matches the MTF characteristic of the missing color component of the first image with the MTF characteristic of the reference color component based on the result smoothed by the smoothing unit. The image processing apparatus according to claim 5.
The smoothing means performs the smoothing process on a reference color component of the first image to generate a color difference component used for generating the second image, and generates the second image. An image processing apparatus, wherein the smoothing process is not performed on a reference color component of the first image for generating a luminance component to be used. The image processing apparatus according to claim 1.
An image processing apparatus, wherein a color component bearing luminance information of the first image is associated as a reference color component of the first image. The image processing apparatus according to claim 1.
The image processing apparatus according to claim 1, wherein a color component arranged at a highest density of the first image is associated as a reference color component of the first image. The image processing apparatus according to claim 1.
An image processing apparatus, wherein information relating to the difference in the MTF characteristics is determined based on external information relating to a state of an optical system at the time of imaging. The image processing apparatus according to claim 9.
The external information relating to the state of the optical system at the time of photographing includes at least one information of a lens type used at the time of photographing, a zoom position, a focus position, an aperture value, and a focus position in the screen. apparatus. The image processing apparatus according to claim 5 or 6,
The image processing apparatus according to claim 1, wherein the smoothing means determines a degree of smoothing for each pixel. The image processing apparatus according to claim 5 or 6,
The image processing apparatus according to claim 1, wherein the smoothing unit determines a common smoothing level across a plurality of pixels. The image processing apparatus according to claim 6.
When generating a plurality of types of color difference components representing different color information for one pixel, the smoothing means performs the smoothing for each color component of the first image constituting each type of color difference component. An image processing apparatus that determines a degree of smoothing of processing.

Images