JP2017183813A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect with high accuracy an area where color blurring occurs due to axial chromatic aberration.SOLUTION: A color blurring removal unit (105) calculates a signal slope at a target pixel of a color component (B plane) being a detection target of a first photographed image as a first signal slope (B_leap), calculates a signal slope at a target position of the color component (B plane) of a second photographed image which is photographed by making a diaphragm smaller than the diaphragm of the first photographed image as a second signal slope (B2_leap), and detects blue color blurring of the color component (B plane) at the target position of the first photographed image based on the first signal slope (B_leap) and the second signal slope (B2_leap).SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for performing signal processing on a captured image.

  2. Description of the Related Art Conventionally, in an imaging apparatus capable of color imaging, a color that does not originally exist around a bright portion of an image may be generated as a color blur due to chromatic aberration of an imaging optical system that forms an optical image of a subject on an imaging element. In color imaging of visible light by an imaging device, color blur tends to occur in a portion away from green, which is the central wavelength of the imaging optical system, and blue, red, or a purple artifact mixed with both blurs. It is called color blur or purple fringe.

  The chromatic aberration of the imaging optical system of the imaging apparatus can be suppressed to some extent optically by combining a plurality of lenses having different dispersions. However, in recent years, downsizing of imaging devices has progressed, and the demand for miniaturization of optical systems has increased along with higher resolution of imaging sensors used for imaging devices, and it has become difficult to sufficiently suppress chromatic aberration with only optical systems. . For this reason, suppression of color bleeding by image processing is required. Note that chromatic aberration is roughly classified into lateral chromatic aberration (also referred to as lateral chromatic aberration) and axial chromatic aberration (also referred to as longitudinal chromatic aberration). The chromatic aberration of magnification is a phenomenon in which the imaging position is shifted in the direction along the image plane depending on the wavelength of each color light. In addition, axial chromatic aberration is a phenomenon in which the imaging position is shifted in the direction along the optical axis depending on the wavelength of each color light. Here, in the case of R (red), G (green), and B (blue) primary color digital imaging systems, with respect to lateral chromatic aberration, geometry that adds different distortions to the R, G, and B color planes. Correction by conversion image processing is possible.

  On the other hand, the axial chromatic aberration is, for example, in an image focused on the G (green) plane that bears the center wavelength of the visible light region, and is subject to the subject in the R (red) plane and the B (blue) plane that are the ends of the visible light region. The image is out of focus and is unclear. This cannot be corrected by geometric transformation image processing such as correction for lateral chromatic aberration. Patent Document 1 uses a characteristic that color blur is mainly generated in the vicinity of whiteout (a preset signal saturation region), and searches for a saturated region of the G plane, and pixels in the surrounding region A technique for calculating a color blur correction value from the integrated value of the above is disclosed. In Patent Document 2, a region where the luminance difference between adjacent pixels is a certain level or more is detected as a region where color blur may occur, and the color blur is noticeable by lowering the saturation of the region. Techniques for eliminating are disclosed.

JP 2007-133582 A JP 2001-145117 A

  The color blur of the image picked up by the image pickup apparatus often occurs mainly in the periphery of the overexposure, but the color blur that is uncomfortable for the observer also occurs in an area where the overexposure does not occur. For example, in a shooting scene such as a sunbeam between trees, a noticeable color blur occurs at the boundary between a background of a blue sky that is not overexposed and branches of trees. However, in Patent Document 1, a whiteout region is searched as a region in which color blur occurs, and therefore, a region that does not have whiteout as described above cannot be detected even if color blur has occurred. Further, in Patent Document 2, when there is a luminance difference between adjacent pixels, an area that may cause color blur may be erroneously detected even though color blur does not actually occur. As described above, the techniques described in Patent Document 1 and Patent Document 2 cannot accurately detect an area in which color blur occurs, and thus cannot perform highly effective color blur correction.

  Accordingly, an object of the present invention is to detect a color blur due to axial chromatic aberration with high accuracy and enable highly effective color blur correction.

  The present invention relates to the inclination of the signal at the target position of the color component that is the detection target of the first captured image, and the color of the second captured image that has been captured with a smaller aperture than the first captured image. The color blur of the color component at the position of interest of the first captured image is detected based on the slope of the signal at the position of interest of the component.

  According to the present invention, it is possible to detect a region where color bleeding due to axial chromatic aberration is generated with high accuracy, and it is possible to perform highly effective color bleeding correction.

It is a figure which shows schematic structure of the imaging device of 1st Embodiment. It is a figure used for description of R, G, B plane. It is a figure used for description of an interpolation signal. 3 is a flowchart from color blur detection to removal according to the first embodiment. It is a figure which shows a chromaticity coordinate surface. 10 is a flowchart from color blur detection to removal according to the second embodiment. It is a figure which shows schematic structure of the imaging device of 3rd Embodiment.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
<Configuration of First Embodiment / Imaging Device>
FIG. 1A is a diagram illustrating a schematic configuration of an imaging apparatus as an application example of an image processing apparatus.
In FIG. 1A, a photographing lens 130 is a lens optical system for forming an optical image of a subject or the like on the imaging surface of the imaging unit 101, and includes a focus lens and a zoom lens. The mechanical mechanism 131 includes a mechanical shutter, a diaphragm, and the like. The photographing lens 130 is driven and controlled by the control unit 120 to perform autofocus (AF) control for focusing on a subject or the like. In the mechanical mechanism 131, the shutter speed and the aperture value are controlled by the control unit 120, and automatic exposure (AE) control is performed to match the appropriate exposure. The shutter speed and the aperture value may be arbitrarily set (manually set) by the user. Further, in the following description, shooting with the AE control or manually set shutter speed and aperture value by the user is referred to as “main shooting”, and an image shot by the main shooting is recorded or displayed as described later. The

  The image pickup unit 101 includes an image pickup element made of CCD, CMOS, or the like and its peripheral circuits, and the image pickup surface of the image pickup element is covered with a Bayer array R (red) G (green) B (blue) color filter. FIG. 2A shows an array of pixels corresponding to R, G, and B of the Bayer array color filter in the image sensor. The light transmitted through the R (red) color filter is incident on the R pixel in the diagram of FIG. 2A, and the light transmitted through the G (green) color filter is transmitted to the B pixel in the diagram. Light that has passed through the B (blue) color filter enters the pixel. From the imaging unit 101, an image signal of an image 250 in which each pixel is arranged corresponding to R, G, B in the Bayer array is output. In the imaging unit 101, the imaging operation is controlled by the control unit 120.

  Here, in the case of the present embodiment, the control unit 120 is configured so that immediately after the main shooting (or just before the main shooting), the aperture is set to be smaller than the aperture value of the main shooting (the f value of the aperture is increased). The mechanism 131 is controlled, and the imaging unit 101 and the imaging lens 130 are controlled to perform imaging. In the present embodiment, a captured image obtained by actual photographing is referred to as a first captured image. On the other hand, an image captured with an aperture smaller than the aperture value at the time of actual shooting is a correction assisting image for correcting the color blur of the first shot image, as will be described later. Then, it will call the 2nd picked-up image. Since the second captured image is captured with a smaller aperture than the first captured image (with a larger f-number), the first captured image and the second captured image are captured. For example, when the shutter speeds are the same, the second captured image is darker than the first captured image.

  The analog image signal of the first captured image and the analog image signal of the second captured image captured by the imaging unit 101 are converted into digital image signals by the A / D (analog / digital) conversion unit 102, respectively. In the following description, the digital image signal of the first captured image is referred to as a first image signal, and the digital image signal of the second captured image is referred to as a second image signal. The first image signal and the second image signal output from the A / D conversion unit 102 are sent to the WB processing unit 103. The WB processing unit 103 performs white balance (WB) processing on each of the first image signal and the second image signal under the control of the control unit 120. The first image signal and the second image signal after the WB processing by the WB processing unit 103 are sent to the interpolation processing unit 104.

  The interpolation processing unit 104 performs interpolation processing on each of the first image signal and the second image signal under the control of the control unit 120. Details of the interpolation processing by the interpolation processing unit 104 will be described later. The first image signal after the interpolation processing by the interpolation processing unit 104 is sent to the memory unit 110 as a first interpolation signal, and the second image signal after the interpolation processing is sent to the memory unit 110 as a second interpolation signal.

  Under the control of the control unit 120, the memory unit 110 stores the first interpolation signal in the first memory 110a, and stores the second interpolation signal in the second memory 110b. Note that the first memory 110a and the second memory 110b of the memory unit 110 may be separate memories or may be a memory area in which one memory is divided. The first interpolation signal stored in the first memory 110 a is read from the first memory 110 a and sent to the alignment unit 115 and the color blur removal unit 105 under the control of the control unit 120. Also, the second interpolation signal stored in the second memory 110 b is read from the second memory 110 b and sent to the alignment unit 115 under the control of the control unit 120.

  The alignment unit 115 sets the pixel position of the second interpolation signal to the first position when each corresponding pixel position of the second interpolation image is shifted from each pixel position of the first interpolation signal. The second interpolation signal after the alignment processing is output so as to match the pixel position of the interpolation signal. That is, in the present embodiment, the alignment unit 115 is an example of a correction unit that aligns the second captured image with the first captured image. When there is no deviation between the pixel positions of the first interpolation signal and the second interpolation signal, the second interpolation signal is output from the alignment unit 115 as it is. Details of the alignment processing by the alignment unit 115 will be described later. The second interpolation signal output from the alignment unit 115 is sent to the color fringing removal unit 105.

  The color blur removal unit 105 is an example of a generation unit and a detection unit, and the first interpolation is performed based on the first interpolation signal from the first memory 110a and the second interpolation signal from the alignment unit 115. A color blur area is detected from the signal to remove the color blur. Details of the color blur area detection and color blur removal processing by the color blur removal unit 105 will be described later. The signal after the color blur is removed from the first interpolation signal by the color blur removal unit 105 is sent to the signal processing unit 106.

  The signal processing unit 106 performs known luminance signal processing, color signal processing, and the like using the first interpolation signal after the color blur is removed to generate an image signal. The image signal generated by the signal processing by the signal processing unit 106 is sent to a recording unit (not shown) and recorded on a recording medium as an image signal by actual photographing, or sent to a display unit (not shown) and displayed. To do.

  FIG. 1A described above shows an example of an imaging apparatus in which the photographing lens 130 to the interpolation processing unit 104 are one system, but the imaging apparatus of the present embodiment is as shown in FIG. In addition, an imaging apparatus having two systems from the photographing lens to the interpolation processing unit may be used.

  In FIG. 1B, photographing lenses 130a and 130b have a focus lens and a zoom lens in the same manner as the photographing lens 130 in FIG. The photographing lens 130a forms an optical image of a subject or the like on the imaging surface of the imaging unit 101a, and the photographing lens 130b forms an optical image of a subject or the like on the imaging surface of the imaging unit 101b. In the photographing lenses 130a and 130b, the focus lens is driven and controlled by the control unit 120, and AF control for focusing on a subject or the like is performed. The mechanical mechanisms 131a and 131b are configured to have a mechanical shutter, a diaphragm, and the like, similarly to the mechanical mechanism 131 in FIG. The imaging units 101a and 101b include an imaging element and its peripheral circuits as in the imaging unit 101 of FIG. 1A, and the imaging surface of the imaging element is covered with an RGB color filter in a Bayer array.

  In the configuration example of FIG. 1B, an image captured by the image capturing unit 101a is a first captured image, and an image captured by the image capturing unit 101b is a second captured image. Therefore, the control unit 120 controls the shutter speed and the aperture value of the mechanical mechanism 131a to the AE control in the actual photographing or the shutter speed and the aperture value set manually by the user. On the other hand, the control unit 120 controls the mechanical mechanism 131b so that the aperture value is smaller than the aperture value set for the mechanical mechanism 131a (the f value of the aperture is increased). The imaging device having a two-system configuration illustrated in FIG. 1B can simultaneously perform the imaging of the first captured image by the imaging unit 101a and the imaging of the second captured image by the imaging unit 101b. For this reason, in the case of the imaging apparatus of FIG. 1B, the first captured image and the second captured image are images captured at the same time without a shooting time lag.

  The analog image signal of the first captured image from the imaging unit 101a is converted into a digital image signal by the A / D conversion unit 102a, and sent to the WB processing unit 103a as the first image signal. Further, the analog image signal of the second photographed image from the imaging unit 101b is converted into a digital image signal by the A / D conversion unit 102b and sent to the WB processing unit 103b as the second image signal. The WB processing unit 103a performs WB processing on the first image signal under the control of the control unit 120, and sends the first image signal after the WB processing to the interpolation processing unit 104a. The WB processing unit 103b performs WB processing on the second image signal under the control of the control unit 120, and sends the second image signal after the WB processing to the interpolation processing unit 104b.

  The interpolation processing unit 104a performs interpolation processing on the first image signal under the control of the control unit 120. Further, the interpolation processing unit 104b performs an interpolation process on the second image signal under the control of the control unit 120. The interpolation processing in these interpolation processing units 104a and 104b is the same processing as the interpolation processing unit 104 in FIG. The first interpolation signal obtained by the interpolation processing of the interpolation processing unit 104 a is sent to the alignment unit 115 and the color blur removal unit 105. The second interpolation signal obtained by the interpolation processing of the interpolation processing unit 104b is sent to the alignment unit 115. The alignment unit 115, the color blur removal unit 105, and the signal processing unit 106 are the same as those shown in FIG.

<Details of interpolation processing>
Interpolation processing performed by the interpolation processing unit 104 in FIG. 1A and the interpolation processing units 104a and 104b in FIG. 1B will be described with reference to FIGS. 2A to 2C. Interpolation processing performed in the interpolation processing unit 104 in FIG. 1A and the interpolation processing units 104a and 104b in FIG. 1B is basically the same processing, so here the interpolation processing unit in FIG. Only the interpolation processing that 104 performs on the first image signal will be described as an example.

  The first image signal is an image 250 having a pixel array corresponding to R, G, and B in the Bayer array shown in FIG. As shown in FIG. 2B, the interpolation processing unit 104 converts the primary image components of the red component (R component), the green component (G component), and the blue component (B component) into each pixel of the image 250 of the first image signal. And a pixel value of zero (0) is inserted at a pixel position where there is no R, G, B primary color component. An image 251 in FIG. 2B is an image in which R component pixel values and zero pixel values are arranged, an image 252 is an image in which G component pixel values and zero pixel values are arranged, and an image 253 is B It is an image in which pixel values of components and zero pixel values are arranged. Further, the interpolation processing unit 104 inserts pixel values interpolated from surrounding pixels at pixel positions where the pixel value is zero for each of the images 251 to 253 in FIG. 2B, as shown in FIG. R, G, and B color planes (R plane 254, G plane 255, and B plane 256) are generated. Hereinafter, in order to simplify the description, the R plane 254, the G plane 255, and the B plane 256 are expressed as only an R plane, a G plane, and a B plane, respectively.

<Details of alignment processing>
An alignment process performed by the alignment unit 115 will be described. As described above, the second interpolation signal is a signal after the alignment with respect to the first interpolation signal, or a pixel position shift has occurred with respect to the first interpolation signal. There is no signal.

  For this reason, the alignment unit 115 first detects the amount of positional deviation between the image of the first interpolation signal and the image of the second interpolation signal. Then, the alignment unit 115 aligns the first interpolation signal and the second interpolation signal based on the detected positional deviation amount. It should be noted that the method of detecting the amount of misalignment necessary for alignment and the method of alignment are used in, for example, image-sharing camera shake correction processing, etc., and many techniques have been disclosed, and the description thereof will be omitted.

  Here, as shown in FIG. 2A, in the image 250 in which pixels corresponding to R, G, and B in the Bayer array are arranged, the G component pixel is larger than the R component pixel and the B component pixel. , The resolution of the G component is higher than the resolution of the R component and the B component. For this reason, in the case of the present embodiment, the alignment unit 115 detects the positional deviation amount of the image between the first interpolation signal and the second interpolation signal, and the first interpolation signal and the second interpolation signal. Among the color planes of the interpolated signal, a G-plane having a high resolution is used to detect a positional shift amount. Thereby, the alignment unit 115 can perform alignment processing with high accuracy.

<Color blur removal process>
Hereinafter, the detection (determination) process and the color blur removal process of the color blur area by the color blur removal unit 105 will be described with reference to FIGS. 3 (A) to 3 (C), FIG. 4, and FIG. It should be noted that the color blur area detection process and the color blur removal process performed by the color blur removal unit 105 are the same in any of the configurations in FIGS. 1A and 1B, and therefore FIG. Description will be made without distinguishing the configurations of A) and FIG.

  In the present embodiment, among the color planes (R plane, G plane, B plane) shown in FIG. 2C, green light (G), which is the central wavelength light in the design of the lens optical system (shooting lens 130), is used. The corresponding G plane is set as a reference plane. Since color blur is likely to occur in any of the R plane and the B plane that are separated from the G plane that is the reference plane, in this embodiment, an image of a color component that is a target of color blur detection in the R plane and the B plane. And In the following description, a B plane is given as an example of an image of a color component to be detected for color blur, and an image having a color component different from the color component to be detected (B plane) is a G plane.

  In FIG. 3A, the horizontal axis indicates the pixel position (pos), and the vertical axis indicates the pixel value (pixel signal intensity). FIG. 3A shows pixels of the B and G planes of the first interpolation signal and the B and G planes of the second interpolation signal when shooting is performed under a certain light source light. This represents the relationship between the position pos and the pixel value. Specifically, the curve 201 in FIG. 3A is the B plane of the first interpolation signal, the curve 202 is the G plane of the first interpolation signal, the curve 203 is the B plane of the second interpolation signal, and the curve 204 is The relationship between the pixel position pos and the pixel value of the G plane of the second interpolation signal is shown.

  Here, an image taken with the aperture stopped produces less axial chromatic aberration than an image taken with the aperture opened. In the case of the present embodiment, the second captured image is an image that has been captured with a reduced aperture (with a larger f value) than in the case of the first captured image. For this reason, the second interpolation signal corresponding to the second photographed image is less likely to cause color blur, while the first interpolation signal corresponding to the first photographed image is likely to cause much color blur. Further, as described above, the color blur is generated in the color plane (B plane in this embodiment) that is distant from the reference plane (G plane). In the example of FIG. 3A, in the first interpolation signal, when the pixel position pos exceeds “1”, the difference between the pixel values of the B plane (curve 201) and the G plane (curve 202) is already zero. Since it is larger (BG> 0), it can be seen that color blur (blue blur) occurs. On the other hand, in the second interpolation signal, the color difference between the B plane (curve 203) and the G plane (curve 204) is zero (B−G = 0) until the pixel position pos reaches “5”. It can be seen that no blur (blue blur) has occurred. In the case of the second interpolation signal, when the pixel position pos is “5” or later, the difference between the pixel values of the B plane (curve 203) and the G plane (curve 204) is greater than zero (B−G> 0) and the color is changed. It can be seen that blur (blue blur) occurs.

  In FIG. 3B, the horizontal axis indicates the pixel position (pos), and the vertical axis indicates the inclination around the pixel value. The inclination around the pixel value is an inclination of a change in signal intensity between the pixel at the target position (hereinafter referred to as the target pixel) and the peripheral pixel. In the present embodiment, the following expressions (1) and (1) It is calculated | required by (2). Since the slope of the change in signal intensity between the target pixel and its surrounding pixels is specifically the slope of the change in pixel value, in the following description, the value representing the slope of the change in signal intensity is referred to as the signal slope. write. A curve 211 in FIG. 3B indicates a signal inclination of the first interpolation signal in the B plane, and a curve 212 indicates a signal inclination of the first interpolation signal in the G plane. A curve 213 in FIG. 3B indicates a signal inclination in the B plane of the second interpolation signal, and a curve 214 indicates a signal inclination in the G plane of the second interpolation image.

  In the case of the present embodiment, the color fringing removal unit 105 calculates the signal slope of the first interpolation signal in the B plane as the signal slope B_leap according to the following equation (1), and calculates the signal slope of the first interpolation signal in the G plane. It calculates as signal inclination G_leap by following formula (2). In the case of the present embodiment, the signal slope B_leap in the B plane of the first interpolation signal corresponds to the first slope generated by the generation unit, and the signal slope G_leap in the G plane of the first interpolation signal is the generation unit. This corresponds to the third slope generated by.

  Note that the variables B (pos + 1) and B (pos−1) in Expression (1) are pixel values of peripheral pixels of the pixel of interest in the B plane of the first interpolation signal. In addition, variables G (pos + 1) and G (pos−1) in Expression (2) are pixel values of peripheral pixels of the pixel of interest in the G plane of the first interpolation signal.

  Then, the color blur removal unit 105 obtains a ratio between the signal slope B_leap in the B plane of the first interpolation signal and the signal slope G_leap in the G plane of the first interpolation signal, and the pixel position pos having a large value of the ratio. It is determined that color blur has occurred in the pixels of. More specifically, the color blur removal unit 105 causes color blur in the pixel at the pixel position pos when the ratio of B_leap to G_leap is greater than a predetermined first threshold value TH1 (B_leap / G_leap> TH1). It is determined that

  In FIG. 3C, the horizontal axis indicates the pixel position (pos), and the vertical axis indicates the signal inclination ratio. A curve 221 in FIG. 3C represents a ratio between the signal gradient B_leap in the B plane of the first interpolation signal and the signal gradient G_leap in the G plane of the first interpolation signal. Note that description of the curve 222 in FIG.

  Here, in the example of FIG. 3C, when the value of the first threshold TH1 is “2”, for example, the condition that the ratio of B_leap to G_leap is greater than the first threshold TH1 (B_leap / G_leap> TH1) The pixel positions pos1 to pos5 are satisfied. In this case, it is determined that the color blur occurs from the pixel positions pos1 to pos5, and it is determined that the color blur does not occur after the pixel position pos6. However, as can be seen from the curve 221 in FIG. 3C, in the vicinity of the pixel position pos6, although the ratio of B_leap to G_leap is smaller than “2” of the first threshold value TH1, color bleeding actually occurs. However, only by comparing the ratio of the signal gradients B_leap and G_leap obtained from the first correction signal with the first threshold value TH1, the color blur occurring near the pixel position pos6 cannot be detected. In this case, in the color blur removal process performed later, the color blur in the vicinity of the pixel position pos6 is not removed and the color blur remains.

  For this reason, the color blur removal unit 105 further reduces the occurrence of color blur in the area where it is determined that the color blur is not generated by comparing the ratio between the signal gradients B_leap and G_leap and the first threshold value TH1. The presence or absence of color blur is determined using the second interpolation signal. In the present embodiment, the color blur removal unit 105 has a large ratio between the signal slope B_leap of the B plane of the first interpolation signal and the signal slope of the B plane of the second correction signal (referred to as B2_leap). It is determined that a color blur has occurred in the pixel at the position pos. More specifically, when the ratio of B_leap and B2_leap is larger than a predetermined second threshold value TH2 (B_leap / B2_leap> TH2), the color blur removing unit 105 performs color blur (blue blurring) on the pixel at the pixel position. ) Has occurred. Note that the signal slope B2_leap in the B plane of the second interpolation signal is obtained by replacing the variables B (pos + 1) and B (pos-1) in the above-described equation (1) with the periphery of the pixel of interest in the B plane of the second interpolation signal. It can be calculated by setting the pixel value of the pixel. In the present embodiment, the signal gradient B2_leap in the B plane of the second interpolation signal corresponds to the second gradient generated by the generation unit.

  A curve 222 in FIG. 3C represents a ratio between the signal inclination B_leap in the B plane of the first correction signal and the signal inclination B2_leap in the B plane of the second correction signal. As can be seen from the curve 222 in the example of FIG. 3C, for example, when the value of the second threshold value TH2 is “2”, the curve 222 falls below “2” of the second threshold value TH2. It is after pos9. Therefore, in this case, the color blur removing unit 105 can determine that color blur is generated not only in the pixel positions pos1 to pos5 but also in the pixel positions pos6 to pos8. As a result, in the color blur removal process performed later, it is possible to remove color blur up to the pixel position pos8.

  FIG. 4 is a flowchart showing a flow from determination of color blur to removal by the color blur removal unit 105. The process of the flowchart in FIG. 4 is performed by the color fringing removal unit 105, but may be realized by, for example, the CPU executing the program according to the present embodiment. The program according to the present embodiment may be prepared in advance in a ROM (not shown), for example, or read from an external storage medium (not shown) or downloaded from a network such as the Internet (not shown). You may load to RAM etc. which are not illustrated. In the following description, steps S301 to S307 of each process in FIG. 4 are abbreviated as S301 to S307, and this is the same for other flowcharts.

  The process of the flowchart in FIG. 4 starts when, for example, the user gives an instruction to start shooting. When the process of the flowchart of FIG. 4 starts, the color fringing removal unit 105 performs a first spatial calculation process as the process of S301. In the present embodiment, the first spatial calculation process is a calculation process as shown in the following formulas (3) and (4). In addition, Formula (3) and Formula (4) are formulas that represent Formula (1) and Formula (2) described above in more detail. In the first spatial calculation process, the signal inclination B_leap in the B plane of the first interpolation signal is calculated by Expression (3), and the signal inclination G_leap of the G plane of the first interpolation signal is calculated by Expression (4). .

  Here, G (x + 1, y) in the expressions (3) and (4) is a pixel value on the right side of the pixel of interest in the G plane of the first interpolation signal, and B (x + 1, y) is This is the pixel value on the right side of the pixel of interest in the B plane of the first interpolation signal. G (x-1, y) is the pixel value on the left of the pixel of interest in the G plane of the first interpolation signal, and B (x-1, y) is the pixel of interest in the B plane of the first interpolation signal. This is the pixel value on the left side. G (x, y + 1) is a pixel value below the target pixel in the G plane of the first interpolation signal, and B (x, y + 1) is a pixel below the target pixel in the B plane of the first interpolation signal. Value. G (x, y−1) is a pixel value adjacent to the pixel of interest in the G plane of the first interpolation signal, and B (x, y−1) is a pixel value of the pixel of interest in the B plane of the first interpolation signal. This is the upper adjacent pixel value. In the example of FIG. 3A described above, the B plane of the first interpolation signal corresponds to the curve 201, and the G plane of the first interpolation signal corresponds to the curve 202. After S301, the color fringing removal unit 105 advances the process to S302.

  In step S <b> 302, the color blur removal unit 105 determines whether the pixel of interest on the B plane of the first interpolation signal is a pixel in a region where color blur occurs (hereinafter referred to as a color blur region). To do. Specifically, the color blur removal unit 105 calculates a ratio between the signal slopes B_leap and G_leap obtained by the equations (3) and (4), and when the ratio is larger than the first threshold value TH1, It is determined that the pixel of interest is a pixel in the color blur area, and the process proceeds to S305. On the other hand, the color blur removal unit 105 determines that the pixel of interest is not a pixel in the color blur area when the ratio of the signal gradient B_leap to G_leap is equal to or smaller than the first threshold (B_leap / G_leap ≦ TH1). , The process proceeds to S303.

  In step S303, the color fringing removal unit 105 performs a second spatial calculation process. The second spatial calculation process is a process of calculating a signal slope B2_leap for the B plane of the second interpolation signal by the following equation (5).

  In the equation (5), B2 (x + 1, y) is the pixel value on the right side of the pixel of interest in the B plane of the second interpolation signal, and B2 (x-1, y) is the second interpolation signal. This is the pixel value on the left side of the pixel of interest in the B plane. B2 (x, y + 1) is the pixel value below the pixel of interest in the B plane of the second interpolation signal, and B2 (x, y-1) is the pixel value above the pixel of interest in the B plane of the second interpolation signal. Pixel value. In the case of the example of FIG. 3A described above, the B plane of the second interpolation signal corresponds to the curve 203. After S303, the color fringing removal unit 105 advances the process to S304.

  In S <b> 304, the color blur removal unit 105 further determines the second interpolation signal that is less likely to cause color blur for the target pixel in the B plane of the first interpolation signal that is determined not to be a pixel in the color blur area in S <b> 302. Is used to determine whether or not a color blur has occurred. Specifically, in S304, the color fringing removal unit 105 determines the signal slope B_leap for the B plane of the first interpolation signal obtained by Expression (3) and B of the second interpolation signal obtained by Expression (5). The ratio of the signal slope B2_leap to the plane is calculated. Then, when the ratio is larger than the second threshold value TH2, the color blur removal unit 105 determines that the pixel of interest is a pixel in the color blur area, and proceeds to S305. On the other hand, the color blur removal unit 105 determines that the pixel of interest is not a pixel in the color blur area when the ratio of the signal gradient B_leap to B2_leap is equal to or smaller than the second threshold (B_leap / B2_leap ≦ TH2). Then, the process of FIG. 4 for the target pixel is ended.

When the process proceeds to S305, the color blur removal unit 105 calculates (estimates) the color blur amount for the pixels determined to be the color blur area in S302 and S304. In the present embodiment, the color blur removal unit 105 calculates the amount of color blur based on the signal slope B_leap of the B plane of the first interpolation signal that is the target of color blur removal. Specifically, the color blur removal unit 105 multiplies the absolute value of the signal slope B_leap in the B plane of the first interpolation signal by a coefficient k (k> 0) as shown in the following equation (6). Is calculated as an estimated color blur amount E. After S305, the color fringing removal unit 105 advances the process to S306.
E = k × | B_leap | (6)

  In step S306, the color fringing removal unit 105 performs an excessive removal suppression process. The excessive removal suppression process is a process for correcting the estimated color bleeding amount E and obtaining an actual removal amount E ′. That is, the estimated color blur amount E calculated in S305 is in line with a certain model and does not necessarily match the actual color blur amount. For example, even in the case of blue light detected as the same B-plane, for example, the blurring method varies between light having a wavelength of 450 nm and light having a wavelength of 400 nm. The expression (6) in S305 does not take into account such a change in color blur due to the wavelength. Therefore, when the estimated color blur amount E is smaller than the actual color blur, for example, the image after the blue blur is removed. Some blueness remains. On the other hand, when the estimated color blur amount E is larger than the actual color, for example, the image after the blue blur is removed is slightly yellowish green. In particular, when the estimated color blur amount E is larger than the actual color as in the latter example, the image becomes unnatural and uncomfortable for the user.

  For this reason, the color fringing removal unit 105 limits the color fringing removal so that the color fringing removal is performed only within a certain hue range as the excessive removal suppressing process of S306. Specifically, the color fringing removal unit 105 first calculates the chromaticity of the pixels of the B plane that is the target of color fringing removal of the first interpolation signal, using Expressions (7) to (9).

a = 5 (x−y) (8)
b = 5 (yz) Formula (9)

Here, FIG. 5 is a diagram showing the chromaticity coordinate (a, b) planes of the equations (8) and (9). In the chromaticity coordinate (a, b) plane of FIG. 5, blue is in the fourth quadrant, and when the estimated color blur amount E is removed from the pixel value (signal intensity) of the B plane, the chromaticity is indicated by a dotted arrow in the figure. Move in the upper left direction. The starting point of the dotted arrow in the figure is the chromaticity before removal, and the tip of the dotted arrow is the chromaticity after the estimated color blur amount E is removed. Therefore, the color blur removal unit 105 limits the hue range that is affected by the estimated color blur amount E to a ′> 0 and b ′ <0. Specifically, the hue range is limited by Expression (10).
B> 0.22R + 0.68G and B> −1.84R + 3.30G Formula (10)

  For this reason, in step S306, the color blur removal unit 105 first sets the removal amount E ′ = 0 for pixels that do not satisfy the condition of the hue range, and excludes them from the removal target in the color blur removal process in step S307. As a result, the pixels excluded from the removal target are not removed by the color blur removal process in S307, and the pixel value is not affected by the color blur removal process in S307. As described above, in step S307, the color blur removal unit 105 performs color blur removal using only pixels that satisfy the condition of the hue range as color blur removal targets.

Further, the color blur removal unit 105 may correct the removal amount E ′ by, for example, the equation (11) and perform the color blur removal process in S307 even for pixels that satisfy the hue range.
E ′ = min (E, B− (0.22R + 0.68G), B − (− 1.84R + 3.30G)) (11)

  When the color blur removal is performed by the removal amount E ′ of the equation (11), the chromaticity change due to the color blur removal stays in the fourth quadrant as indicated by the solid line arrow in FIG. Note that the starting point of the solid line arrow in FIG. 5 is the chromaticity before removal, and the tip of the solid line arrow is the chromaticity after the color blur removal by the removal amount E ′. Thereby, it is possible to prevent over-removal of the B plane exceeding the hue range by the removal processing in S307.

The color fringing removal unit 105 removes the color fringing in S307, and generates a new B plane by subtracting the above-described removal amount E ′ from the signal intensity of the B plane as shown in Expression (12). Also good.
B = B−E ′ (12)

  In the above description, the color blur region determination is performed by taking the ratio of the signal gradient B_leap and the signal gradient B2_leap. For example, the color blur region determination is performed using the difference between the signal gradient B_leap and the signal gradient B2_leap. May be.

  In the above description, the color blur removal target is the B plane. However, even if the color blur removal target is the R plane, color blur removal can be performed in the same manner as described above. Since the process when the color blur removal target is the R plane can be realized by the same process as the case where the B plane is the color blur removal target, the description thereof is omitted.

  As described above, the imaging apparatus determines whether or not color blur due to axial chromatic aberration has occurred in the first interpolation signal obtained from the first captured image captured in the actual shooting. It is carried out. At the same time, the imaging apparatus according to the present embodiment uses the second image obtained from the second photographed image with a smaller aperture than that at the time of the main photographing with respect to the pixel determined to have no color blur in the first interpolation signal. The color blur determination using the interpolation signal is also performed. That is, in consideration of the fact that the amount of axial chromatic aberration generated in an image shot with a reduced aperture is reduced, the imaging apparatus of the present embodiment performs the second shooting with a reduced aperture than during actual shooting. Color blur determination is performed based on the image. As a result, the imaging apparatus according to the present embodiment can accurately detect color blur caused by axial chromatic aberration. Then, the imaging apparatus according to the present embodiment performs a color blur removal process on a color blur area that is determined to have color blur in the first captured image obtained by the main shooting. Therefore, according to the imaging apparatus of the present embodiment, it is possible to accurately and effectively correct the color blur caused by the axial chromatic aberration.

<Second Embodiment>
An imaging apparatus according to the second embodiment will be described. Since the configuration of the imaging apparatus according to the second embodiment is the same as that of FIGS. 1A and 1B described above, the illustration is omitted. Hereinafter, the description of the same configuration and processing as those in the first embodiment will be omitted, and differences from the first embodiment will be described. In the case of the imaging apparatus according to the second embodiment, the processing in the color fringing removal unit 105 is different from that in the first embodiment.

  FIG. 6 is a flowchart illustrating a flow from detection (determination) to removal of a color blur area in the color blur removal unit 105 according to the second embodiment. The process of the flowchart in FIG. 6 is performed by the color fringing removal unit 105, but may be realized by, for example, a CPU executing a program according to the second embodiment. As in the case of the first embodiment, the program according to the second embodiment is downloaded from a ROM (not shown), an external storage medium, a network such as the Internet, and loaded into a RAM (not shown).

  In the flowchart of FIG. 6, S601 is the same process as S301 of FIG. Similarly, S602 is S302, S604 is S303, S605 is S304, S606 is S305, S607 is S306, and S608 is almost the same as S307. In the case of the second embodiment, when it is determined in S602 that the area is not a color blur area, the color blur removal unit 105 advances the process to S603.

  In step S <b> 603, the color fringing removal unit 105 is a pixel in the moving object region corresponding to the moving subject between the first interpolation signal and the second interpolation signal as the local moving object region determination process. It is determined whether or not. For the determination process of the moving object region, various known techniques such as a determination method based on a known motion vector can be used, and thus detailed description thereof is omitted here. Note that the moving object region determination process may be performed by the alignment unit 115. When the alignment unit 115 determines the moving object region, the color blur removal unit 105 acquires information indicating the determination result of the moving object region from the alignment unit 115. If the color blur removal unit 105 determines in S603 that the pixel in the target area is a pixel in the moving object area, the process of the flowchart in FIG. 6 ends for that pixel. On the other hand, if the color blur removal unit 105 determines in S603 that the pixel is not a moving object region pixel, the process proceeds to S604. The processing after S604 is the same as the processing after S303 in the flowchart of FIG.

  As described above, in the case of the second embodiment, the color fringing removal unit 105 determines whether or not the target region is a moving object region by the moving object region determination process using the first and second interpolation signals. Yes. Then, when the target region is not a moving object region, the color blur removal unit 105 performs pixel blur that is determined to have no color blur in the color blur determination using the first interpolation signal as in the first embodiment. Further, color blur determination using the second interpolation signal is also performed. On the other hand, when the target region is a moving object region, the color blur removal unit 105 performs only color blur determination using the first interpolation signal, and does not perform color blur determination using the second interpolation signal. Accordingly, in the second embodiment, it is possible to avoid erroneously determining a moving object region as a color blur region and performing color blur removal. Therefore, according to the second embodiment, it is possible to detect a color blur area and remove a color blur with high accuracy for both the moving object area and the other non-moving object areas.

<Third Embodiment>
An imaging apparatus according to a third embodiment will be described. FIG. 7A and FIG. 7B are diagrams illustrating a schematic configuration of an imaging apparatus according to the third embodiment. FIG. 7A is a configuration example of an imaging apparatus according to the third embodiment in which the photographing lens 130 to the interpolation processing unit 104 form one system as in FIG. 1A of the first embodiment. FIG. 7B illustrates a configuration example of the imaging apparatus according to the third embodiment in which the photographing lenses 130a and 130b to the interpolation processing units 104a and 104b are provided in two systems, similar to FIG. 1B of the first embodiment. It is. In the imaging apparatus of the third embodiment shown in FIGS. 7A and 7B, the same configuration as that of the imaging apparatus of the first embodiment is the same as in FIGS. 1A and 1B. The reference numerals are attached and their explanation is omitted. Hereinafter, differences of the third embodiment from the first embodiment will be described.

  The imaging apparatus of the third embodiment has a magnification position correction unit 515 instead of the alignment unit 115 of the first and second embodiments. The magnification position correction unit 515 performs image magnification correction processing and alignment processing on the image of each color plane of the second interpolation signal. That is, in the present embodiment, the magnification position correction unit 515 corresponds to a correction unit that performs image magnification correction and alignment with respect to the first captured image with respect to the second captured image.

  Here, the second captured image that is the original image of the second interpolation signal is an image that is captured in a state of being narrower than the first captured image that is the original image of the first interpolation signal. The image becomes darker than the captured image. For example, when trying to obtain a second photographed image having the same brightness as the first photographed image, photographing with a slower shutter speed is performed than when the first photographed image is photographed. However, when the shutter speed becomes slow, subject blurring and camera shake tend to cause not only blurring in the direction perpendicular to the optical axis but also image blurring in the optical axis direction (focus direction). Appears as a change in image magnification. When the image magnification changes in this way, it may be difficult to align the edge positions of the image, for example, when aligning the first interpolation signal and the second interpolation signal.

  For this reason, the magnification position correction unit 515 according to the third embodiment performs image magnification on the second interpolation signal at the time of alignment between the first interpolation signal and the second interpolation signal. To compensate for changes. For the correction processing of the image magnification change, for example, a well-known technique described in Japanese Patent Application Laid-Open No. 2007-43637 or the like can be used, and detailed description thereof is omitted. In this way, the magnification position correction unit 515 corrects the change in the image magnification with respect to the second interpolation signal, and the position for matching the image of the second interpolation signal after the image magnification correction with the image of the first interpolation signal. And the matching process. Since the alignment process is the same as the alignment process of the first and second embodiments described above, the description thereof is omitted.

  According to the third embodiment, the image magnification correction process and the alignment process are performed by the magnification position correction unit 515, so that even when the image magnification changes due to the influence of image blurring in the optical axis direction, the first It is possible to accurately match the image position between the interpolation signal and the second interpolation signal. As a result, the image pickup apparatus according to the third embodiment can perform highly accurate color blur area determination and color blur removal.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  101, 101a, 101b Imaging unit, 131, 131a, 131b Mechanical mechanism (aperture), 104, 104a, 104b Interpolation processing unit, 105 Color blur removal unit, 110 Memory unit, 115 Alignment unit, 515 Magnification position correction unit, 120 Control unit

Claims (14)

The inclination of the signal at the position of interest of the color component that is the detection target of the first captured image is defined as the first inclination, and the second captured image captured with the aperture set smaller than that of the first captured image. Generating means for generating a slope of a signal of the color component at the position of interest as a second slope;
Detecting means for detecting a color blur of the color component at the position of interest of the first captured image based on the first inclination and the second inclination;
An image processing apparatus comprising: The generation unit generates a signal gradient at the position of interest of an image having a color component different from the color component to be detected in the first captured image as a third gradient,
The detection means includes
Detecting that the color blur of the color component to be detected has occurred at the position of interest when the ratio between the first inclination and the third inclination is larger than a predetermined first threshold,
For the position of interest where the ratio of the first inclination to the third inclination is not more than a predetermined first threshold, the ratio of the first inclination to the second inclination is a predetermined first The image processing apparatus according to claim 1, wherein when the threshold value is greater than 2, the image processing apparatus detects that the color blur of the detection target color component has occurred at the target position. The generation unit generates a signal gradient at the position of interest of an image having a color component different from the color component to be detected in the first captured image as a third gradient,
The detection means includes
Detecting that the color blur of the color component to be detected has occurred at the position of interest when the difference between the first inclination and the third inclination is greater than a predetermined first threshold;
For the position of interest where the difference between the first inclination and the third inclination is less than or equal to a predetermined first threshold, the difference between the first inclination and the second inclination is a predetermined first The image processing apparatus according to claim 1, wherein when the threshold value is greater than 2, the image processing apparatus detects that the color blur of the detection target color component has occurred at the target position. The color component to be detected is a blue component or a red component among the primary color components of the red component, the green component, and the blue component included in the captured image,
The image processing apparatus according to claim 2, wherein the color component different from the color component to be detected is the green component. Correction means for aligning the second captured image with respect to the first captured image;
The generation unit generates the inclination of a signal at the target position of the color component to be detected of the second photographed image subjected to the alignment as the second inclination. 5. The image processing apparatus according to any one of 4. Correction means for performing image magnification correction and alignment with respect to the first captured image with respect to the second captured image;
The generation unit generates a signal gradient at a target position of the color component to be detected of the second photographed image subjected to the image magnification correction and the alignment as the second gradient. The image processing apparatus according to any one of claims 1 to 4.   The correction means includes an image of a green component among the primary color components of the red component, the green component, and the blue component included in the first captured image, and a red component, a green component, and a blue component included in the second captured image. 7. The image processing apparatus according to claim 5, wherein the first photographed image and the second photographed image are aligned using a green component image among the primary color components of the component. .   When the detection unit detects a moving object region in the first captured image, the target position included in the detected moving object region includes the first inclination and the second inclination. The image processing apparatus according to claim 1, wherein the color blur detection based on the color component is not performed. The detection means includes
Calculating a color blur amount of the color component to be detected based on the first inclination;
The color blur is removed from the first photographed image by subtracting a signal intensity corresponding to the color blur amount from a signal intensity at the target position of the first captured image. The image processing apparatus according to any one of 8.   The image processing apparatus according to claim 9, wherein the detection unit sets a target position in a predetermined hue range as a target to remove the color blur.   The detection unit corrects the calculated color blur amount based on a predetermined hue range, and removes the color blur from the first captured image based on the corrected color blur amount. Item 10. The image processing apparatus according to Item 9. The target position is a pixel position of the target pixel,
The image processing apparatus according to claim 1, wherein the inclination of the signal at the target position is an inclination of a change in a pixel value of a peripheral pixel with respect to a pixel value of the target pixel. The inclination of the signal at the focus position of the color component that is the detection target of the first captured image, and the color component of the second captured image that is captured with a smaller aperture than the first captured image. An image processing method for an image processing apparatus, comprising: detecting a color blur of the color component at the position of interest of the first captured image based on an inclination of a signal at the position of interest.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 12.

Images