JP2013247597A - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent occurrence of color blurring, when suppressing influence of mixed colors caused by a flare.SOLUTION: An image processing apparatus comprises: an image input unit which receives an image; a flare determination unit which determines whether a flare has occurred or not, on the basis of information on the area of a predetermined color component included in the image; a setting unit which, when the occurrence of a flare is determined, sets to a pixel included in the area of the predetermined color component caused by the flare at least, a weight based on a color component value of the pixel; and a smoothing unit which executes a smoothing process on the image on the basis of the weight set by the setting unit.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  2. Description of the Related Art Conventionally, there has been known an imaging element in which a plurality of pixels for focus detection are arranged on a part of a light receiving surface in which a plurality of imaging pixels are arranged two-dimensionally (Patent Document 1). The plurality of imaging pixels have spectral characteristics corresponding to each of the plurality of color components, and the focus detection pixels (focus detection pixels) have spectral characteristics different from the plurality of imaging pixels. A signal for generating an image is read from a plurality of imaging pixels to determine a pixel value of the imaging pixel, and a focus detection signal is read from the focus detection pixel to detect a focus. The pixel value of the working pixel is determined. When performing the pixel interpolation process, the pixel value of the missing color component among the pixel values of the imaging pixel is interpolated, and the imaging pixel value corresponding to the position of the focus detection pixel is interpolated.

  In the invention described in Patent Document 1, in order to perform the interpolation processing for the focus detection pixel, the interpolation pixel value of the focus detection pixel is generated using the pixel value of the imaging pixel in the vicinity of the focus detection pixel, An evaluation pixel value that is a pixel value when the neighboring imaging pixel has the same spectral characteristics as the focus detection pixel is calculated, and the image value is calculated using the pixel value of the focus detection pixel and the evaluation pixel value. A high frequency component is calculated, and the pixel value of the imaging pixel corresponding to the position of the focus detection pixel is calculated by adding the high frequency component to the interpolated pixel value.

JP 2009-303194 A

  By the way, when flare is generated by the reflected light from the wiring region between pixels around the light receiving portion of the image sensor, color mixing occurs due to crosstalk in which incident light between the microlenses leaks into adjacent pixels around the light receiving portion. When such a phenomenon occurs, not only the focus detection pixels but also the surrounding pixels are affected, and the focus detection pixels that have undergone color mixing by the pixel interpolation process described above and the surrounding pixels feel uncomfortable. Appears as a residual interpolation. For a pixel in which such color mixture has occurred, for example, the effect of color mixture can be reduced by performing a smoothing process. However, if smoothing processing is performed in a region where the flare is strong, the flare is reduced in the region where the flare is strong, but the region where the flare is weak or the region where the flare is not affected There is a problem that color blur occurs.

  The present invention has been made in view of such points, and an image processing apparatus, an imaging apparatus, and an image processing capable of preventing the occurrence of color bleeding when suppressing the influence of color mixing caused by flare. The purpose is to provide a program.

  In order to solve the above-described problem, an image processing apparatus according to the present invention is configured to determine whether or not flare has occurred based on information related to an image input unit that receives an image and a region of a predetermined color component included in the image. A flare determination unit that determines whether or not the flare has occurred, and at least the color component included in the predetermined color component region caused by the flare when the flare is generated. A setting unit configured to set a weight based on the value; and a smoothing unit configured to perform a smoothing process on the image based on the weight set by the setting unit.

  Further, an area detection unit that detects an area of the predetermined color component from the image, and an edge detection unit that detects an edge included in the image, wherein the flare determination unit is a detection result of the area detection unit And based on the detection result of the said edge detection part, it is determined whether the said flare has generate | occur | produced.

  Further, the setting unit divides a color gamut that the predetermined color component in the chromaticity diagram can take into a plurality of regions, and color component values of each pixel included in the region of the predetermined color component due to the flare Is set to one of the plurality of regions, the weight is set for each pixel included in the region of the predetermined color component caused by the flare.

  The setting unit is included in the predetermined color component region caused by the flare with respect to pixels included in the region excluding the predetermined color component region caused by the flare in the image. A weight different from the weight set for each pixel is set.

  Further, the image includes an imaging pixel and a focus detection pixel, and the focus detection is performed by performing an interpolation process on the image received by the image input unit or the image subjected to the smoothing process. And a pixel interpolation unit that generates an interpolation pixel value for the target pixel.

  Moreover, the imaging device of this invention is provided with the image pick-up element which acquires the said image, and the image processing apparatus mentioned above.

  The image processing program of the present invention includes an image input step for receiving an image, and a flare determination step for determining whether or not a flare has occurred based on information relating to a region of a predetermined color component included in the image. When it is determined that the flare has occurred, a weight based on a color component value of the pixel is set for at least a pixel included in the predetermined color component region caused by the flare. And a smoothing step of executing a smoothing process on the image based on the weight set in the setting step.

  According to the present invention, it is possible to prevent the occurrence of color blur when suppressing the influence of color mixing caused by flare.

It is a block diagram which shows the electric constitution of the electronic camera of this invention. It is a figure which shows the example of arrangement | positioning of the pixel for imaging, and AF pixel. It is a figure which shows a part of image data centering on the area | region where AF pixel is arrange | positioned. It is a figure which shows the AF pixel interpolation part provided with the flare determination part. It is a figure which shows the structure of a flare determination part. It is a flowchart which shows the flow of a process from a flare determination to pixel interpolation. It is a graph which shows the example which divided | segmented the magenta color gamut in a chromaticity diagram into three area | regions. 6 is a flowchart showing a flow of first pixel interpolation processing. It is a figure showing an example of the image structure where the effect of this embodiment is exhibited. It is a flowchart which shows the flow of a 2nd pixel interpolation process. It is a figure which shows the structure of a flare determination part. It is a flowchart which shows the flow of a process from a flare determination to pixel interpolation. It is a figure which shows the relationship between the threshold value used when (a) the histogram calculated | required from image data, (b) the calculated histogram, and a high-intensity pixel. It is a figure which shows the relationship between Y image based on Y data, and the object area | region from which a rectangle is extracted. It is a graph which shows the relationship between the variable a and Sb / St value. (A) A range of color components based on [Formula 51] on the chromaticity diagram, and (b) a range of color components based on [Formula 51] and [Formula 52] on the chromaticity diagram.

  As shown in FIG. 1, an electronic camera 10 to which the present invention is applied includes a CPU 11. A non-volatile memory 12 and a work memory 13 are connected to the CPU 11, and the non-volatile memory 12 stores a control program that is referred to when the CPU 11 performs various controls. Further, in the nonvolatile memory 12, data indicating the position coordinates of AF pixels of the image sensor 17, which will be described later in detail, data such as various threshold values and weighting coefficients used for an image processing program obtained in advance, and various determination tables Memorize etc.

  The CPU 11 controls each part using the work memory 13 as a temporary storage work area according to a control program stored in the non-volatile memory 12 and operates each part (circuit) function constituting the electronic camera 10.

  Subject light incident from the photographic lens 14 forms an image on a light receiving surface of an image sensor 17 such as a CCD or CMOS through a diaphragm 15 and a shutter 16. The image sensor driving circuit 18 drives the image sensor 17 based on a control signal from the CPU 11. The image sensor 17 is a Bayer array type single-plate image sensor, and a primary color transmission filter 19 is attached to the front surface.

  The primary color transmission filter 19 is, for example, a primary color Bayer in which the resolution of G (green) is N / 2 and the resolution of R (red) and B (blue) is N / 4 with respect to the total number N of pixels of the image sensor 17. It is an array.

  The subject image formed on the light receiving surface of the image sensor 17 is converted into an analog image signal. The image signal is sequentially output to CDS 21 and AMP 22 constituting an AFE (Analog Front End) circuit, subjected to predetermined analog processing by the AFE circuit, and then digitally processed by an A / D (Analog / Digital converter) 23. It is converted into image data and sent to the image processing unit 25.

  The image processing unit 25 includes a separation circuit, a white balance processing circuit, a pixel interpolation (demosaicing) circuit, a matrix processing circuit, a non-linear transformation (γ correction) processing circuit, an edge enhancement processing circuit, and the like. Thus, processing such as white balance, pixel interpolation, matrix, nonlinear conversion (γ correction), and contour enhancement is performed. The separation circuit separates a signal output from an imaging pixel, which will be described in detail later, and a signal output from the focus detection pixel. The pixel interpolation circuit converts a Bayer array signal of one color per pixel into a normal color image signal composed of three colors per pixel.

  The three-color image data output from the image processing unit 25 is stored in the SDRAM 27 through the bus 26. The image data stored in the SDRAM 27 is read out under the control of the CPU 11 and sent to the display control unit 28. The display control unit 28 converts the input image data into a predetermined display signal (for example, an NTSC color composite video signal) and outputs it as a through image to the display unit 29.

  The image data acquired in response to the shutter release is read from the SDRAM 27 and then sent to the compression / decompression processing unit 30. After the compression processing is performed here, the memory card 32 which is a recording medium via the media controller 31 is sent. To be recorded. A release button 33 and a power switch (not shown) are connected to the CPU 11, and temperature information is input from a temperature detection unit 34 that detects the temperature of the image sensor 17. This information is sent to the image processing unit 25 and used, for example, when determining noise.

  The AWB / AE / AF detector 35 detects the defocus amount and the defocus direction by the pupil division type phase difference detection method based on the signal of the focus detection pixel (AF pixel). The CPU 11 controls the driver 36 based on the defocus amount and the defocus direction obtained by the AWB / AE / AF detection unit 35 to drive the focusing motor 37 to advance and retract the focus lens in the optical axis direction. Adjust the focus.

  Further, the AWB / AE / AF detection unit 35 uses the photometric luminance value (Bv) calculated based on the image pickup pixel signal and the ISO sensitivity value (Sv) set by the photographer using the ISO sensitivity setting unit 38. The light value (Lv = Sv + Bv) is calculated. Then, the AWB / AE / AF detection unit 35 determines the aperture value and the shutter speed so that the exposure value (Ev = Av + Tv) becomes the calculated light value Lv. Based on this determination, the CPU 11 drives the aperture drive unit 39 to adjust the aperture diameter of the aperture 15 so that the obtained aperture value is obtained. At the same time, the CPU 11 drives the shutter driving unit 40 to execute an opening / closing operation of the shutter 16 so that the obtained shutter speed and the shutter 16 are opened.

  The AWB / AE / AF detection unit 35 performs thinning-out reading from the image data of one screen captured in the SDRAM 27 during auto white balance adjustment, and generates, for example, 24 × 16 AWB evaluation data. Then, the AWB / AE / AF detection unit 35 performs light source type discrimination using the generated AWB evaluation data, and corrects the signal of each color channel according to the white balance adjustment value suitable for the discriminated light source type.

  The imaging element 17 is provided with a primary color transmission filter 19 of any of R (red), G (green), and B (blue) in a Bayer array type in each of a plurality of imaging pixels on the light receiving surface. A CCD or CMOS semiconductor image sensor with a microlens array provided thereon is appropriately selected and used. Furthermore, the image sensor 17 of the present embodiment has a plurality of AF pixels 41 arranged one-dimensionally in the horizontal scanning direction in a partial region on the light receiving surface. Those AF pixels 41 are not provided with the primary color transmission filter 19. In addition, there are two types of AF pixels 41 that receive light beams that pass through the left or right side of the pupil of the optical system of the photographic lens 14. The imaging element 17 can individually read out pixel signals from the imaging pixel group and the AF pixel group.

  In the present embodiment, an image sensor in which the primary color transmission filter 19 is not installed for the AF pixel 41 is described. However, the present invention is not limited to this, and an ND filter or An image sensor in which a G primary color transmission filter is disposed may be used.

  As shown in FIG. 2, the AF pixel 41 has sensor openings 41a and 41b that are biased to one side from the cell center (the center of the microlens), and are arranged one-dimensionally along the direction of the bias. The sensor openings 41a and 41b are biased in opposite directions, and the bias distance is the same. The AF pixel 41 having the sensor opening 41a is placed in place of the G pixel in the RGB primary color Bayer array, and the AF pixel 41 having the sensor opening 41b is placed in place of the B pixel in the RGB primary color Bayer array. It is. The pupil division phase difference AF method is realized by the AF pixel 41 having such sensor openings 41a and 41b. That is, of the two light fluxes passing through the exit pupil, two partial light fluxes that are symmetrical with respect to the optical axis of the photographic lens 14 are separated by the AF pixel 41 having the sensor opening 41a and the AF pixel 41 having the sensor opening 41b. If each light is received, the direction of focus shift (moving direction of the focusing lens) and the amount of focus shift (moving amount of the focusing lens) are known from the phase difference between the signals output from the two pixels 41. This enables quick focusing.

  Therefore, each AF pixel 41 in the present embodiment outputs a detection signal obtained by dividing the left or right pupil according to the luminance of white light. FIG. 3 shows a part of the image data centered on the area where the AF pixel 41 is arranged in the image data captured by the image sensor 17. Each cell represents one pixel. The symbols R, G, and B at the head of each cell indicate imaging pixels having the primary color transmission filters 19. On the other hand, symbols X and Y indicate AF pixels that are sensitive to light beams from the left side or the right side, and they are alternately arranged one-dimensionally in the horizontal scanning direction. The two-digit number following these symbols indicates the pixel position.

  The pixel interpolation unit includes an AF pixel interpolating unit 45 that interpolates the pixel value of the AF pixel 41 using the pixel value of the imaging pixel, and a linear interpolation method from the Bayer array to RGB after interpolating the pixel value of the AF pixel. A pixel interpolation unit for performing color interpolation.

  As shown in FIGS. 4 and 5, the AF pixel interpolation unit 45 includes a flare determination unit 46, and performs different AF pixel interpolation processing based on the determination by the flare determination unit 46. For example, when the flare determination unit 46 determines that no flare has occurred, the AF pixel interpolation unit 45 executes the first pixel interpolation process. This first pixel interpolation process is a process of interpolating the pixel value of the AF pixel by predicting with a weighted sum based on the pixel value of the imaging pixel from the pixel value (white (W) component) of the AF pixel. .

  On the other hand, if the flare determination unit 46 determines that flare has occurred, the AF pixel interpolation unit 45 executes a second pixel interpolation process. The second pixel interpolation process is a process of correcting the pixel values of the imaging pixels around the AF pixel by weighting and smoothing the corrected pixel values of the imaging pixels.

  The flare determination unit 46 includes a color detection unit 47, an edge detection unit 48, an evaluation value determination unit 49, and a weight setting unit 50. The color detection unit 47 detects whether or not the captured image data includes pixels having magenta color information. For example, when a pixel having magenta color information is included, the color detection unit 48 extracts a pixel having magenta color information. The edge detection unit 47 performs edge detection on the captured image data. The evaluation value calculation unit 49 determines whether or not flare has occurred in the captured image data based on the result of edge detection in the edge detection unit 48. The weight setting unit 50 sets a weight level used for calculating a clip amount, which will be described later, for each pixel of the image data. The weight level set for pixels included in a magenta area recognized as a mixed color area caused by the occurrence of flare and the weight level set for pixels included in another area are: , Different weight levels are set.

  Next, the operation of the AF pixel interpolation unit 45 will be described with reference to FIG. In the present embodiment, since the primary color transmission filter 19 installed in each of the imaging pixels has a Bayer pattern, green (G) imaging is performed at the position of the AF pixel of symbol X shown in FIG. The pixel value of the pixel is interpolated, and the pixel value of the blue (B) imaging pixel is interpolated at the pixel position of the AF pixel of symbol Y. In the following description, a case will be described in which the pixel value of the blue imaging pixel Y44 and the pixel value of the green imaging pixel X45 are each interpolated. The procedure for interpolating the pixel values of the imaging pixels in the other AF pixels is the same.

  Hereinafter, flare determination will be described based on the flowchart of FIG.

(Color detection processing)
The AF pixel interpolation unit 45 performs color detection processing based on the image data output from the CPU 11 (S-1). For example, if the captured image data is image data (RGB data) indicated in the RGB color space, the AF pixel interpolation unit 45 converts the RGB data into image data (YCrCb data) indicated in the YCrCb color space. Here, an equation for converting RGB data into YCrCb data is expressed by [Equation 1]. If the captured image data is YCbCr data, the color space conversion process represented by [Equation 1] is omitted.

ＡＦ画素補間部４５は、ＹＣｒＣｂデータのうち、Ｃｒデータ及びＣｂデータを用いて、マゼンタ色の色情報を有する画素が含まれるか否かを判定する。マゼンタ色の色情報を有する画素が含まれると判定された場合、ＡＦ画素補間部４５は、マゼンタ色の色情報を有する画素の領域（マゼンタ領域）を抽出する。なお、マゼンタ領域に含まれる画素は、１画素であっても、複数の画素であってもよい。
（エッジ検出）
ＡＦ画素補間部４５は、ＹＣｒＣｂデータのうち、Ｙデータを用いてエッジ検出処理を実行する（Ｓ−２）。ＡＦ画素補間部４５は、［数２］を用いて隣り合う４方向のＹ成分の全体差分和ＳＡＤを求める。

The AF pixel interpolation unit 45 uses the Cr data and the Cb data in the YCrCb data to determine whether or not a pixel having magenta color information is included. When it is determined that a pixel having magenta color information is included, the AF pixel interpolation unit 45 extracts a pixel region (magenta region) having magenta color information. The pixels included in the magenta area may be one pixel or a plurality of pixels.
(Edge detection)
The AF pixel interpolating unit 45 executes edge detection processing using Y data among YCrCb data (S-2). The AF pixel interpolation unit 45 obtains the total difference sum SAD of the Y components in the four adjacent directions using [Expression 2].

  The AF pixel interpolation unit 45 determines whether or not the total difference sum SAD of the Y components in the four adjacent directions is equal to or less than the threshold value Th1. Then, when the total difference sum SAD of the Y components in the four adjacent directions is equal to or less than the threshold value Th1, the pixel is set as the target pixel.

Although the edge detection process is executed using the entire Y data, the present invention is not limited to this, and the edge detection process may be performed using the Y data of the magenta area detected by the color detection process. Is possible. Here, the edge detection processing need not be limited to the above-described method, and for example, an edge filter such as a Sobel Filter or a Prewitt Filter can be used.
(Determining whether flare has occurred)

  The AF pixel interpolation unit 45 determines whether or not flare has occurred in the captured image data (S-3). The AF pixel interpolation unit 45 determines whether or not there is a magenta region in which the total number of target pixels is equal to or greater than the threshold Th2, based on the detection result in the color detection process and the detection result in the edge detection process. As described above, when flare occurs in the reflected light from the wiring area between pixels around the light receiving portion of the image sensor 17 during photographing, color mixing occurs due to crosstalk in which incident light between microlenses leaks into adjacent pixels around the light receiving portion. Will occur. When such a phenomenon occurs, the target pixel and its peripheral pixels are also affected. Therefore, when flare occurs, there are a predetermined number of pixels that are mixed colors due to the occurrence of flare. For this reason, the threshold value Th2 used for determining whether or not flare has occurred is a value obtained in advance from experiments, statistics, and the like.

  For example, if there is no magenta area in which the number of target pixels is equal to or greater than the threshold, the AF pixel interpolation unit 45 determines that no flare has occurred. In this case, the AF pixel interpolation unit 45 performs the first pixel interpolation process (S-4). The first pixel interpolation process will be described later. On the other hand, if there is a magenta region in which the number of target pixels is equal to or greater than the threshold, the AF pixel interpolation unit 45 determines that flare has occurred.

(Weight setting)
The AF pixel interpolation unit 45 sets a weight level for each of the pixels included in the magenta area caused by the occurrence of flare (S-5). This weight level is used when calculating the clip amount executed in the second pixel interpolation process. As shown in FIG. 7, the area surrounded by the straight line L1 and the straight line L2 is a color gamut indicating a magenta color. In the color gamut indicating the magenta color, the darker the magenta color, the stronger the influence of color mixing due to the occurrence of flare, and it is necessary to perform a strong interpolation process. Further, the lighter the magenta color, the weaker the influence of the color mixture caused by the occurrence of flare, so that the interpolation process has a lower intensity. For this reason, the AF pixel interpolating unit 45 determines that the color component value (pixel value) of a pixel included in a magenta area recognized as a mixed color area caused by the occurrence of flare is a magenta color gamut based on shading. The weight level is set depending on whether it belongs to one of a plurality of divided areas. The AF pixel interpolation unit 45 sets the weight level to “0” for pixels other than the above-described region. Then, weight information in which the set weight levels are collected is generated.

  In FIG. 7, the magenta color gamut is divided into three areas, and among the divided areas, the weight level is “small” for the area 55 and the weight is set for the area 56. The case where the level is set to “medium” and the weight level is set to “large” for the region 57 is described. After setting the weight level, the AF pixel interpolation unit 45 executes a second pixel interpolation process (S-6). The magenta color gamut is divided into three areas and weight levels are provided for the respective areas. However, the present invention is not limited to this, and the magenta color gamut is divided into two or more areas. It is also possible to provide a level of weight in the region.

[First pixel interpolation processing]
When it is determined that no flare has occurred, the AF pixel interpolation unit 45 selects and executes the second pixel interpolation process. In the first pixel interpolation process, the pixel value of the imaging pixels around the AF pixel is used to determine the direction in which the variation value, which is the change rate of the pixel value, is minimized. Then, the pixel value of the AF pixel is interpolated using the pixel value of the imaging pixel in the direction of the smallest fluctuation. Hereinafter, the flow of the first pixel interpolation processing will be described based on the flowchart of FIG.

(Calculate the direction of the minimum fluctuation value)
The AF pixel interpolating unit 45 uses the pixel values of the imaging pixels around X45 and Y44 to interpolate the AF pixels X45 and Y44, and direction variations H1 to H4 that are pixel value change rates in four directions. Are obtained using [Equation 3] to [Equation 6], respectively (S-21). In this embodiment, the four directions are 45 degrees and 135 degrees with respect to the horizontal scanning direction, the vertical scanning direction, and the horizontal scanning direction.

[Equation 3]
Direction variation H1 in the horizontal scanning direction =
2 × (| G34-G36 | + | G54-G56 |) + | R33-R35 | + | R53-R55 | + | B24-B26 | + | B64-B66 |

[Equation 4]
Direction variation H2 in the vertical scanning direction =
2 × (| G34-G54 | + | G36-G56 |) + | R33-R53 | + | R35-R55 | + | B24-B64 | + | B26-B66 |

[Equation 5]
Directional variation H3 = 45 degrees with respect to the horizontal scanning direction =
2 × (| G27-G36 | + | G54-G63 |) + | R35-R53 | + | R37-R55 | + | B26-B62 | + | B28-B64 |

[Equation 6]
Directional change H4 = 135 degrees with respect to the horizontal scanning direction =
2 × (| G23-G34 | + | G56-G67 |) + | R33-R55 | + | R35-R57 | + | B22-B66 | + | B24-B68 |

(Interpolate the pixel value of the AF pixel using the pixel value of the surrounding imaging pixels according to the direction of the minimum variation value)
The AF pixel interpolation unit 45 selects the direction of the smallest direction variation among the direction variations H1 to H4 obtained in step (S-21), and uses the pixel value of the imaging pixel in that direction to perform AF. selected direction among the pixel values _{B Y44} imaging pixels of B at the position of the pixel values _{G X45} and AF pixel Y44 of imaging pixels of G at the position of the pixel X45 [Expression 7] - [number 10] (S-22). As a result, by using the pixel values of the imaging pixels in the direction of small fluctuation, it becomes possible to more accurately perform interpolation for AF pixels such as X45 and Y44.

[Equation 7]
When the direction change H1 is the minimum, B _Y44 = (B24 + B64) / 2
G _X45 = (G34 + G36 + G54 + G56) / 4

[Equation 8]
When the direction change H2 is the minimum B _Y44 = (B24 + B64) / 2
G _X45 = (G25 + G65) / 2

[Equation 9]
When direction change H3 is the minimum, B _Y44 = (B26 + B62) / 2
G _X45 = (G36 + G54) / 2

[Equation 10]
When the direction change H4 is the minimum, B _Y44 = (B22 + B66) / 2
G _X45 = (G34 + G56) / 2

  The AF pixel interpolating unit 45 changes the direction variation H5 of the pixel value of the AF pixel in the horizontal scanning direction that is the arrangement direction of the AF pixels, for example, pixel values W44 and W45 of white light of Y44 and X45 of the AF pixel, and [ It is calculated using [Formula 11].

[Equation 11]
H5 = | W44-W45 |

The AF pixel interpolation unit 45 determines whether or not the value of the direction variation H5 exceeds the threshold Th3 (S-23). If the directional fluctuation H5 is a value that exceeds the threshold Th3 (YES side), AF pixel interpolation unit 45, the step (S-22) at the determined _B of _Y44 and _{G X45} interpolated value of the imaging pixel in the Y44 and X45 Image data is updated using pixel values. The image processing unit 25 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 27 via the bus 13 (S-24). ).

  On the other hand, when the direction variation H5 is equal to or less than the threshold Th3 (NO side), the image processing unit 25 proceeds to (S-25). Note that the threshold value Th3 may be a value of about 512 when processing a 12-bit image, for example.

The AF pixel interpolation unit 45 determines whether or not the direction variation H2 obtained in step (S-21) exceeds the threshold Th4 (S-25). If the values directional fluctuation H2 is greater than the threshold value Th4 (YES side), AF pixel interpolation unit 45, the step (S-22) at the determined _B of _Y44 and _{G X45} interpolated value of the imaging pixel in the Y44 and X45 Image data is updated using pixel values. The image processing unit 25 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 27 via the bus 13 (S-24). ).

  On the other hand, when the direction variation H2 is equal to or less than the threshold Th4 (NO side), the image processing unit 25 proceeds to (S-26). Note that the threshold Th4 may be a value of about 64 when processing a 12-bit image, for example.

  Thereafter, the AF pixel interpolating unit 45 calculates the average pixel value <W44> of white light in the AF pixel such as Y44 having sensitivity to the light flux from the right side, and the like for the imaging pixels of the color components R, G, and B in the vicinity. Calculation is performed using the pixel value (S-26). Specifically, in step (S-21), for example, when the image processing unit 25 determines that the direction variation H2 is the minimum, the pixel value of the B imaging pixel is expressed by the equation described in [Equation 8]. Some B24 and B64 are used. On the other hand, for the R and G pixel values, the R and G pixel values at the positions of the B imaging pixels B24 and B64 are interpolated using the four equations described in [Equation 12].

[Equation 12]
(1) R _B24 = (R13 + R15 + R33 + R35) / 4
(2) G _B24 = (G14 + G23 + G25 + G34) / 4
(3) R _B64 = (R53 + R55 + R73 + R75) / 4
(4) G _B64 = (G54 + G63 + G65 + G74) / 4

  Then, the AF pixel interpolating unit 45 uses the white light pixel values W24 and W64 at the positions of the imaging pixels B24 and B64 by using the R, G, and B weighting factors WR, WG, and WB transferred from the CPU 11. , [Equation 13]. Note that how to obtain the weighting factors WR, WG and WB will be described later.

[Equation 13]
W24 = WR × R _B24 + WG × G _B24 + WB × B24
W64 = WR × R _B64 + WG × G _B64 + WB × B64
Then, the image processing unit 25 calculates the average pixel value <W44> = (W24 + W64) / 2 of white light in Y44.

  The AF pixel interpolating unit 45 calculates the average pixel value <W45> of white light in the AF pixel such as X45 having sensitivity to the light beam from the left side, as in the case of step (S-26), and the adjacent color components. Calculation is performed using the pixel values of the R, G, and B imaging pixels (S-27). In step (S-21), when the image processing unit 25 determines that the direction variation H2 is the minimum, the pixel value of the G imaging pixel is expressed by G25 and G65 in the equation described in [Equation 8]. Use. On the other hand, for the R and B pixel values, the R and B pixel values at the positions of the G imaging pixels G25 and G65 are interpolated using the four equations described in [Equation 14].

[Formula 14]
(1) R _G25 = (R15 + R35) / 2
(2) B _G25 = (B24 + B26) / 2
(3) R _G65 = (R55 + R75) / 2
(4) B _G65 = (B64 + G66) / 2

  Then, the AF pixel interpolation unit 45 calculates the pixel values W25 and W65 of white light at the positions of the imaging pixels G25 and G65 by using the weighted sum of the formula described in [Equation 15].

[Equation 15]
W25 = WR × R _G25 + WG × G25 + WB × B _G25
W65 = WR × R _G64 + WG × G25 + WB × B _G65
Then, the image processing unit 25 calculates the average pixel value <W45> = (W25 + W65) / 2 of white light at X45.

  The AF pixel interpolation unit 45 obtains the high-frequency component of the pixel value of white light in each AF pixel of the image sensor 17 using the average pixel value of white light obtained in (S-26) and (S-27) ( S-28). The AF pixel interpolation unit 45 first obtains an average pixel value of white light at the pixel position of each AF pixel from the pixel value of each AF pixel of the image sensor 17. That is, the pixel value of each AF pixel is a value obtained by dividing the light flux from the left side or the right side into pupils. Therefore, in order to obtain the pixel value of the white light at the position of each AF pixel, it is necessary to add the pixel values of the light beams from the left side and the right side to each other. Therefore, the AF pixel interpolation unit 45 of the present embodiment uses the pixel value of each AF pixel and the pixel value of the adjacent AF pixel, for example, to calculate the average pixel value of white light at the position of the AF pixel Y44 or X45 [several 16].

[Equation 16]
<W44>'= W44 + (W43 + W45) / 2
<W45>'= W45 + (W44 + W46) / 2

  In [Expression 16] described in step (S-27), the pixel value of the white light at the position of each AF pixel is calculated using the pixel value of the AF pixel adjacent in the AF pixel arrangement direction. When there is a strong variation in the arrangement direction, the calculation of the high frequency component becomes inaccurate, and the resolution of the white light pixel values in the arrangement direction may be lost. Therefore, in the above-described step (S-23), when there is a strong variation in the arrangement direction, the addition of the high frequency component is stopped.

Thereafter, the AF pixel interpolation unit 45 obtains high-frequency components HF _Y44 and HF _X45 of white light at the positions of Y44 and X45 from the equation described in [Equation 17].

[Equation 17]
HF _Y44 = <W44>'-<W44>
HF _X45 = <W45>'-<W45>

The AF pixel interpolating unit 45 determines that the ratio of the high-frequency component HF of the white light pixel value at the position of each AF pixel obtained in step (S-28) to the white light pixel value is the threshold Th5 (in this embodiment). For example, about 10%) is determined (S-29). If the threshold Th5 high-frequency component HF is smaller (YES side), AF pixel interpolation unit 45 the interpolated values of _{B Y44} and _{G X45} calculated in step S26 as the pixel values of the image pickup pixels in the Y44 and X45, the image data Update. The image processing unit 25 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 27 via the bus 13 (S-24). .

  On the other hand, when the high frequency component HF exceeds the threshold Th5 (NO side), the AF pixel interpolating unit 45 proceeds to step (S-30). Note that the value of the threshold Th5 will be described together with the description of the weighting factors WR, WG, and WB later.

  The AF pixel interpolation unit 45 calculates the color variations VR, VGr, VB, and VGb of the pixel values of the image pickup pixels of the respective color components R, G, or B in the vicinity of Y44 and X45 (S-30). Here, the color variations VGr and VGb indicate the G color variation at the position of the R or B imaging pixel. The AF pixel interpolation unit 45 obtains the color fluctuations VR and VGr based on the two equations described in [Equation 18].

[Equation 18]
(1) VR = | R33-R53 | + | R35-R55 | + | R37-R57 |
(2) VGr = | (G32 + G34) / 2− (G52 + G54) / 2 | + | (G34 + G36) / 2− (G54 + G56) / 2 | + | (G36 + G38) / 2− (G56 + G58) / 2 |

  Note that the AF pixel interpolation unit 45 of the present embodiment calculates the value of VGr after obtaining the average value of the G pixel values at the R imaging pixel positions R33, R35, R37, R53, R55, and R57.

  On the other hand, the AF pixel interpolation unit 45 obtains the color fluctuations VB and VGb based on the two equations described in [Equation 19].

[Equation 19]
(1) VB = | B22−B62 | + | B24−B64 | + | B26−B66 |
(2) VGb = | (G21 + G23) / 2− (G61 + G63) / 2 | + | (G23 + G25) / 2− (G63 + G65) / 2 | + | (G25 + G27) / 2− (G65 + G67) / 2 |

  Note that the AF pixel interpolation unit 45 of the present embodiment calculates the value of VGb after obtaining the average value of the G pixel values at the positions B22, B24, B26, B62, B64, and B66 of the B imaging pixels.

  The AF pixel interpolation unit 45 calculates the color variation rates KWG and KWB for the white light of the color components G and B using the color variations VR, VGr, VB, and VGb calculated in step (S-30) (S- 31). First, the AF pixel interpolation unit 45 obtains the color variations VR2, VG2, and VB2 from the three formulas described in [Equation 20] from the color variations VR, VGr, VB, and VGb.

[Equation 20]
(1) VR2 = (VR + α) × (VGb + α)
(2) VB2 = (VB + α) × (VGr + α)
(3) VG2 = (VGb + α) × (VGr + α)

  Here, α is an appropriate constant for stabilizing the value of the color variation rate. For example, when processing a 12-bit image, α may be set to about 256.

  Then, the image processing unit 25 calculates the color variation VW in white light using the color variations VR2, VG2, and VB2 according to the equation described in [Equation 21].

[Equation 21]
VW = VR2 + VG2 + VB2
Therefore, the AF pixel interpolation unit 45 calculates the color variation rates K _WG and K _WB from [Equation 22].

[Equation 22]
K _WB = VG2 / VW
K _WB = VB2 / VW

The AF pixel interpolation unit 45 uses the high-frequency component HF of the white light pixel value at the position of each AF pixel obtained in step (S-28), and the color variation rates K _WG and K _WB calculated in step (S-31). Are used to calculate the high-frequency components of the pixel values of the color components G and B at the position of each AF pixel from the equation described in [Equation 23] (S-32).

[Equation 23]
HFB _Y44 = HF _Y44 × K _WB
HFG _X45 = HF _X45 × K _WG

  The AF pixel interpolation unit 45 adds the high-frequency component of each color component in each AF pixel obtained in step (S-32) to the pixel value of the imaging pixel obtained by interpolation in step (S-22) (S -18). For example, the CPU 11 calculates the imaging pixel values B ′ and G ′ for Y44 and X45 based on the equation described in [Equation 24], respectively.

[Equation 24]
B ′ _Y44 = B _Y44 + HFB _Y44
G ′ _X45 = G _X45 + HFG _X45

The AF pixel interpolating unit 45 uses the pixel values such as B ′ _Y44 and G ′ _X45 obtained by interpolation at the positions of AF pixels such as Y44 and X45 as the pixel values of the imaging pixels at the respective positions. Update. The image processing unit 25 converts the updated image data into image data of three colors per pixel and stores it in the SDRAM 27 (S-24).

  Even if there is no change in the arrangement direction of the AF pixels, the high-frequency component of the pixel value of the white light is caused by the deviation between the weighted sum of the spectral characteristics of the imaging pixels of each color component and the spectral characteristics of the AF pixels. Has a slight error. When there is no large fluctuation of the image in the vertical scanning direction (direction intersecting the AF pixel arrangement direction), the accuracy of the interpolation value is sufficient without adding a high-frequency component. There is a risk that a false structure resulting from Therefore, in step (S-25), in such a case, addition of a high frequency component is suppressed. In addition, when the calculated high frequency component is sufficiently small, the accuracy of the interpolation value is sufficient even if it is not added, and there is a possibility that a false structure due to an error may be caused by adding the high frequency component. For this reason, in (S-25), addition of a high frequency component is suppressed in such a case.

  Next, how to obtain the weighting factors WR, WG and WB will be described together with the threshold value Th5. In obtaining such weighting factors and threshold values, an image sensor 17 having the same performance as the image sensor 17 incorporated in the product or the image sensor 2 is prepared. The imaging element 17 is irradiated with illumination with substantially uniform illuminance while changing the wavelength band in various ways, and captured image data for each wavelength band is acquired. Then, for the captured image data n in each wavelength band, the pixel value Wn of white light is calculated by adding the pixel values of AF pixels with different pupil divisions as in the equation described in [Equation 16]. At the same time, the pixel values Rn, Gn, and Bn of the image pickup pixels of the respective color components in the vicinity of the AF pixel are also extracted.

  Then, a square error E is defined as a function of unknown weighting factors WR, WG and WB as shown in [Equation 25].

[Equation 25]
E = Σn (WR × Rn + WG × Gn + WB × Bn−Wn) ²

  Then, weighting coefficients WR, WG, and WB that minimize E are obtained (weighting coefficients WR, WG, and WB that obtain a value obtained by partially differentiating E from WR, WG, or WB are set to 0). In this way, by obtaining the weighting factors WR, WG, and WB, a weighting factor that represents the spectral characteristic of the AF pixel by the weighted sum of the spectral characteristics of the imaging pixels of the respective color components R, G, and B is obtained. The weighting factors WR, WG and WB thus obtained are recorded in the nonvolatile memory of the electronic camera 10.

  Further, an error rate Kn is obtained for each captured image data n based on the obtained weighting factors WR, WG, and WB by the equation described in [Equation 26].

[Equation 26]
Kn = | WR × Rn + WG × Gn + WB × Bn−Wn | / Wn

  Then, the maximum value of Kn is obtained and recorded in the nonvolatile memory 12 as the threshold Th5.

  FIG. 9 shows an example of an image structure in which the effect of this embodiment is exhibited. FIG. 13 is a diagram in which an image structure of five vertical pixels including a convex structure (bright lines or dots) is vertically cut, the horizontal axis is the vertical scanning direction (y coordinate), and the vertical axis is the light quantity or the pixel value. The convex structure is just on the AF pixel row arranged in the horizontal scanning direction.

  The circles in FIG. 9 are pixel values captured by the G imaging pixels. However, since there is no G imaging pixel at the position of the AF pixel, the G pixel value at that position cannot be obtained. Therefore, when there is a convex structure at the position of the AF pixel, the convex structure in FIG. 13 cannot be reproduced only from the pixel values of the G imaging pixels in the vicinity of the AF pixel. In fact, in (S-22), the G pixel value (marked with ● in FIG. 9) obtained by interpolation at the position of the AF pixel using the pixel value of the G imaging pixel in the vicinity of the AF pixel is a convex structure. Is not reproduced.

  On the other hand, the pixel value of white light is obtained at the position of the AF pixel. However, normal pixels receive light that passes through the entire pupil region, whereas AF pixels receive only light that passes through the right or left side of the pupil, so adjacent AF pixels with different pupil divisions are added. Thus, the pixel value of the normal white light (of the light that has passed through the entire area of the pupil) is calculated ([Equation 16]).

  In addition, it is sufficient in many cases by interpolating and generating other color components R and B at the position of the G imaging pixel in the vicinity of the AF pixel to obtain a weighted sum of the color components R, G and B. The pixel value of white light can be obtained with high accuracy ([Equation 13] and [Equation 15]).

  The squares in FIG. 9 are the distribution of pixel values of white light determined in this way. In many cases, since the high-frequency component of the white light pixel value is proportional to the high-frequency component of the color component G pixel value, the high-frequency component calculated from the white light pixel value is the convex structure component of the G pixel value. Have information. Therefore, the high-frequency component of the G pixel value is obtained based on the high-frequency component of the white light pixel value, and the value is added to the data of the ● mark to obtain the G pixel value of the ☆ mark and reproduce the convex structure ([Equation 23]).

[Second pixel interpolation processing]
When it is determined that the flare has occurred, the AF pixel interpolation unit 45 selects and executes the second pixel interpolation process. In the second pixel interpolation process, the pixel value of the imaging pixels around the AF pixel is corrected by the weighting coefficient, and the pixel value of the corrected imaging pixel is smoothed, and then the first pixel interpolation process described above is performed. Is a process of executing. Hereinafter, the second pixel interpolation process for the two columns of the AF pixel X43 and the AF pixel Y44 in FIG. 3 will be described. Hereinafter, the flow of the first pixel interpolation process will be described based on the flowchart of FIG.

(The pixel values of the imaging pixels around the AF pixel row are corrected by the weighting coefficient)
The AF pixel interpolation unit 45 determines whether or not the pixel value of the imaging pixels arranged around the AF pixel row is equal to or greater than the threshold value MAX_RAW, and uses the set weight coefficient based on the determination result. (S-41). Here, the threshold value MAX_RAW is a threshold value for determining whether or not the pixel value is saturated.

  The AF pixel interpolation unit 45 does not correct the pixel value of the imaging pixel when the pixel value of the imaging pixel is equal to or greater than the threshold value MAX_RAW. On the other hand, when the pixel value of the imaging pixel is less than the threshold value MAX_RAW, the AF pixel interpolating unit 45 subtracts the weighted sum value using the weighting coefficient from the original pixel value to obtain the pixel of the imaging pixel. Correct the value.

  The AF pixel interpolating unit 45 corrects the pixel value of the R color component imaging pixel using [Equation 27] to [Equation 30].

[Equation 27]
R13 ′ = R13− (R3U — 0 × R33 + R3U — 1 × G34 + R3U — 2 × B24)

[Equation 28]
R33 ′ = R33− (R1U — 0 × R33 + R1U_1 × G34 + R1U_2 × B24)

[Equation 29]
R53 ′ = R53− (R1S_0 × R53 + R1S_1 × G54 + R1S_2 × B64)

[Equation 30]
R73 ′ = R73− (R3S — 0 × R53 + R3S — 1 × G54 + R3S — 2 × B64)

  Here, R1U_0, R1U_1, R1U_2, R1S_0, R1S_1, R1S_2, R3U_0, R3U_1, R3U_2, R3S_0, R3S_1, and R3S_2 are weighting coefficients. In the weighting factor, the character S indicates that it is positioned above the AF pixel, and the character U indicates that it is positioned below the AF pixel.

  The AF pixel interpolating unit 45 corrects the pixel values of the G color component imaging pixels using [Equation 31] to [Equation 36].

[Equation 31]
G14 ′ = G14− (G3U_0 × R33 + G3U_1 × G34 + G3U_2 × B24)

[Formula 32]
G23 ′ = G23− (G2U — 0 × R33 + G2U_1 × G34 + G2U_2 × B24)

[Equation 33]
G34 ′ = G34− (G1U — 0 × R33 + G1U_1 × G34 + G1U_2 × B24)

[Formula 34]
G54 ′ = G54− (G1S_0 × R53 + G1S_1 × G54 + G1S_2 × B64)

[Equation 35]
G63 ′ = G63− (G2S — 0 × R53 + G2S_1 × G54 + G2S_2 × B64)

[Equation 36]
G74 ′ = G74− (G3S_0 × R53 + G3S_1 × G54 + G3S_2 × B64)

  Here, G1U_0, G1U_1, G1U_2, G1S_0, G1S_1, G1S_2, G2U_0, G2U_1, G2U_2, G2S_0, G2S_1, G2S_2, G3U_0, G3U_1, G3U_2, G3S_1, G3S_1, G3S_1, and G3S_1 are weights.

  Further, the AF pixel interpolation unit 45 corrects the pixel value of the image pickup pixel of the B color component using [Equation 37] and [Equation 38].

[Equation 37]
B24 ′ = B24− (B2U — 0 × R33 + B2U_1 × G34 + B2U_2 × B24)

[Equation 38]
B64 ′ = B64− (B2S_0 × R53 + B2S_1 × G54 + B2S_2 × B64)

  Here, B2U_0, B2U_1, B2U_2, B2S_0, B2S_1, and B2S_2 are weighting coefficients.

(Calculation of clip amount using pixel values of adjacent AF pixels)
The AF pixel interpolation unit 45 reads the pixel values X43 and Y44 of the adjacent AF pixels, and obtains the clip amount Th_LPF using [Equation 39] (S-42).

[Equation 39]
Th_LPF = (X43 + Y44) × K_Th_LPF

  Here, K_Th_LPF is a coefficient. As the coefficient K_Th_LPF is larger, the effect of the smoothing process is higher, that is, the strength of the smoothing process is increased.

  In the flare determination described above, when it is determined that a flare has occurred, weight information is generated. The AF pixel interpolation unit 45 sets the value of the coefficient K_Th_LPF based on the weight information. For example, when the weight level is “0”, the AF pixel interpolation unit 45 sets the coefficient K_Th_LPF = 5. When the weight level is “small”, the AF pixel interpolating unit 45 sets the coefficient K_Th_LPF = 20. When the weight level is “medium”, the AF pixel interpolation unit 45 sets the coefficient K_Th_LPF = 70. When the weight level is “large”, the AF pixel interpolating unit 45 sets the coefficient K_Th_LPF = 128. As described above, the effect of the smoothing process is changed by changing the value of the coefficient K_Th_LPF in accordance with the weight level.

(Calculate prediction error for each color component)
The AF pixel interpolation unit 45 uses [Equation 40] and [Equation 41], and among the imaging pixels having the same color component arranged in the same column, the imaging pixels (far from the AF pixel 41 ( The difference between the pixel value of the distal imaging pixel) and the pixel value of the imaging pixel (proximal imaging pixel) located near the AF pixel 41 is calculated as a prediction error (S-43).

[Equation 40]
deltaRU = R13′−R33 ′
deltaRS = R73′−R53 ′

[Equation 41]
deltaGU = G14'-G34 '
deltaGS = G74'-R54 '

(Determines whether the prediction error exceeds the clip range)
The AF pixel interpolating unit 45 calculates the clip range (−) based on the clip amount obtained by [Equation 39] in which each value of the prediction error deltaRU, deltaRS, deltaGU, and deltaGS obtained by [Equation 40] and [Equation 41]. It is determined whether or not included in (Th_LPF to Th_LPF) (S-44).

(Clip processing)
The AF pixel interpolation unit 45 performs a clipping process on a prediction error out of the clip range among the prediction errors deltaRU, deltaRS, deltaGU, and deltaGS (S-45). Here, the clipping process is to perform clipping so that a prediction error value outside the clip range is included in the clip range.

(Add prediction error to pixel value of proximal imaging pixel)
The AF pixel interpolation unit 45 adds the prediction error to the pixel value of the proximal imaging pixel in each column by [Equation 42] (S-46). Here, the prediction error is a value obtained by [Equation 40] and [Equation 41] or a clipped value.

[Formula 42]
R33 "= R33 '+ deltaRU
R53 "= R53 '+ deltaRS
G34 "= G34 '+ deltaGU
G54 "= G54 '+ deltaGS

  Thereby, the pixel value of the distal imaging pixel and the pixel value of the proximal imaging pixel, which are the pixel values of the imaging pixels around the AF pixel row, are respectively corrected, and further, a smoothing process using a prediction error Thus, the pixel value of the proximal imaging pixel is corrected.

(The corrected pixel value of the imaging pixel is stored in the SDRAM)
The AF pixel interpolation unit 45 stores the pixel value of the distal imaging pixel corrected by the weighting factor and the pixel value of the proximal imaging pixel corrected by the prediction error in the SDRAM 27 (S-47).

  When the above-described processing (the processing from S-41 to S-47) is completed, the first pixel interpolation processing is executed.

(First pixel interpolation processing)
The AF pixel interpolation unit 45 executes the second pixel interpolation process described above using the pixel values of the imaging pixels stored in the SDRAM 27 (S-48). Thereby, the pixel value of the imaging pixel corresponding to the AF pixel is calculated. That is, the pixel value of the AF pixel is interpolated.

(The interpolated AF pixel value is stored in SDRAM)
The AF pixel interpolation unit 45 stores the pixel value of the AF pixel interpolated by the first pixel interpolation process (S-49) in the SDRAM 27.

  In the second pixel interpolation process, a smoothing process based on the weight level set by the density of magenta is performed in the correction process. By performing this smoothing process, it is possible to reduce the influence of color mixing due to the flare that occurs on the imaging pixels adjacent to the AF pixel, and to suppress the occurrence of color blur in areas where no flare occurs can do. In addition, since the interpolation processing for the AF pixel is performed using the pixel value of the imaging pixel in which the influence of the color mixture or the color blur is reduced, the pixel value in which the influence of the color mixture due to the generated flare is reduced can be obtained also in the AF pixel. It is done. That is, an image with reduced flare effects can be obtained.

  As described above, according to the embodiment, when flare occurs, the pixel value of the imaging pixels around the focus detection pixel among the pixels constituting the flare-determined image is corrected by the weighting factor, Since the pixel value of the corrected imaging pixel is smoothed using a threshold value based on the density of the mixed color (magenta color) caused by flare, color mixing is performed on the imaging pixels around the focus detection pixel. It is possible to suppress the occurrence of color bleeding that occurs when the influence of the above is reduced. Thereby, it is possible to suppress the influence of color mixing and the effect of color blurring even in the subsequent generation of the interpolation pixel value of the focus detection pixel.

  In the above embodiment, the arrangement direction of the AF pixels is the horizontal scanning direction. However, the present invention is not limited to this, and the AF pixels may be arranged in the vertical scanning direction or other directions.

  In each of the above embodiments, each AF pixel is a focus detection pixel that divides the light beam from the left or right side into pupils, but the present invention is not limited to this, and each AF pixel has a light beam from the left and right sides. Focus detection pixels having pixels that divide the pupil may be used.

  In the present embodiment, an example of the flare determination unit 46 including the color detection unit 47, the edge detection unit 48, the evaluation value determination unit 49, and the weight setting unit 50 is taken up. However, the present invention is not limited to this, and FIG. The flare determination unit 60 shown in FIG.

  Hereinafter, the configuration of the flare determination unit 60 shown in FIG. 11 will be described. The flare determination unit 60 includes a histogram generation unit 61, a threshold calculation unit 62, a target region extraction unit 63, a matrix determination unit 64, a magenta region detection unit 65, an evaluation value determination unit 66, and a weight setting unit 67. The flare determination unit 47 acquires a gain value for white balance from the CPU 11 in addition to image data (YRGB data).

  The histogram generation unit 61 generates a histogram using the Y data of the image data. The threshold calculation unit 62 calculates a threshold (high luminance determination threshold) used when extracting a region with high luminance (high luminance region) from the histogram generated by the histogram generation unit 61. The target area extraction unit 63 uses the Y data and the high luminance determination threshold value to extract a rectangular area having high luminance as the target area. The matrix determination unit 64 determines a matrix used when converting RGB data of image data into CrCb data. The magenta region detection unit 65 detects a region (magenta region) in which the magenta color included in the target region extracted by the target region extraction unit 63 exists. In this detection, the magenta area detection unit 65 calculates the total area of the magenta area, the total edge amount of the Y component, and the like. The evaluation value determination unit 66 obtains the total area of the magenta region, the dispersion value / the total area of the magenta region, and the average edge amount of the Y component, and determines whether or not flare has occurred by determining each of the obtained values as a threshold value. To do. The weight setting unit 67 sets the weight level based on the color components (Cr value and Cb value) of each pixel included in the magenta area recognized as a mixed color area caused by flare.

  Hereinafter, the case where the flare determination is performed in the AF pixel interpolation unit including the flare determination unit 60 having the configuration illustrated in FIG. 11 will be described based on the flowchart illustrated in FIG. 12. The AF pixel interpolation unit will be described with the same reference numeral (symbol 45) as in the present embodiment.

(Generate histogram)
The AF pixel interpolation unit 45 generates a histogram using the Y data of the image data (S-61). If the Y data is 16-bit data, the AF pixel interpolation unit 45 converts the Y data from 16-bit data to 8-bit data, and generates a histogram. FIG. 13A is an example of a histogram using Y data.

(Calculate threshold for high brightness determination using histogram)
The AF pixel interpolation unit 45 uses the generated histogram to calculate a threshold (high brightness determination threshold) Th_high used when extracting a region (target region) used when detecting a magenta region, which will be described later ( S-62). As shown in FIG. 13B, the AF pixel interpolating unit 45 uses Y data and uses a threshold value (saturation determination threshold value) Th <b> 11 to determine whether the pixel value is saturated. Identify the value. The saturation determination threshold Th11 is preferably provided as 8-bit data because the histogram is generated using 8-bit data. However, in the case of 16-bit data, a value converted to 8-bit data is used. Use it.

  After this specification, the AF pixel interpolation unit 45 sequentially adds values obtained by multiplying the histogram by a pixel value equal to or less than the threshold value Th11 and the number of pixels from the highest pixel value. Then, in the process of adding, the pixel value when [Equation 43] is satisfied is set as a high-luminance determination threshold Th_high.

[Equation 43]
Histogram addition value (pixel value × number of pixels)> (256-Th12) × total number of pixels

  Note that Th12 is a threshold value indicating the ratio of the high luminance component. The total number of pixels is 24 × 16 (= 384) pixels because Y data of 24 × 16 pixels is used.

  The AF pixel interpolation unit 45 converts the obtained high-luminance determination threshold Th_high from 8-bit data to 16-bit data.

(Rectangle extraction of the area (target area) used when detecting the magenta area using the threshold for high brightness determination)
The AF pixel interpolation unit 45 uses the high-luminance determination threshold Th_high converted to 16 bits and the Y data for AWB evaluation to extract a rectangle of the target area (S-63). As shown in FIG. 14, the AF pixel interpolating unit 45 shifts pixels one by one in the horizontal scanning direction among the pixels of the image (Y image) 70 based on the Y data, and determines the pixel value of the target pixel. Then, a pixel in which both the pixel value of the pixel immediately before the target pixel is equal to or higher than the high luminance determination threshold Th_high is detected. When the processing for one line in the horizontal scanning direction is completed, the AF pixel interpolation unit 45 shifts one pixel in the vertical scanning direction and executes the same processing. As a result, the AF pixel interpolation unit 45 determines that the pixel value of the target pixel and the pixel value of the pixel immediately before the target pixel are both high-luminance determination threshold Th_high for all pixels of Y data. Pixels having the above pixel values are detected.

  In FIG. 13, the area indicated by reference numeral 71 is the area of the detected pixel. The AF pixel interpolation unit 45 extracts a rectangular area (reference numeral 72) smaller by one pixel from the detected area (reference numeral 71) as a target area. This is because if the detected pixel area is extracted as a target area as it is, the pixel value of the low-brightness pixel will be used when obtaining the total difference sum of the edge amounts of Y components in the four adjacent directions described later. To prevent that. The AF pixel interpolation unit 45 generates the position data of the pixels located at the four corners of the extracted target area as the position information of the target area to be extracted as a rectangle.

  Here, the AF pixel interpolating unit 45 performs the above-described processing on all the pixels of the Y data. However, the present invention is not limited to this, and the luminance value is less than the above-described saturation determination threshold Th11. It is also possible to perform the above-described processing on the pixel.

(Matrix calculation)
The AF pixel interpolation unit 45 calculates a matrix used when converting the RGB data of the image data into CrCb data (S-64).

  It is known that a color mixture caused by flare occurring at the time of photographing appears as a magenta color in an image. The reason for this magenta color is that when the R, G, and B gain values used for white balance processing are multiplied by the pixel values of the respective color components of the respective pixels, the values of the R color component and the B color component are changed. For example, it becomes larger than the value of the G color component.

  Here, when the same subject is photographed by changing the type of the light source, the R, G, and B gain values used in the white balance process use values corresponding to the type of the light source. For this reason, even when a subject of the same color is photographed, the output value varies greatly depending on the type of light source. Due to the fluctuation of the output value, it is difficult to reliably detect the flare.

  Therefore, in the present invention, a color space conversion matrix used when converting RGB data to CrCb data is multiplied by a correction coefficient matrix obtained by using a gain value used in white balance processing, and a new matrix is obtained. A color space conversion matrix is calculated. Then, the RGB data is converted into CrCb data using the newly obtained color space matrix.

  Here, a new gain value for the R color component is represented by [Equation 44] when a white balance R gain value and a reference value (512 × 2) are set by a weighted sum using a weighting coefficient.

[Equation 44]
α _R = a × Rgain + (1−a) × 512 × 2

  Here, the reference value (512 × 2) is a value set in consideration of the fact that the value of Rgain is approximately twice that of Ggain even if the type of the light source changes.

  Here, the Sb / St value obtained when the value of a is changed in each of shooting under clear sky and indoor shooting is considered. Sb is the variance between classes (shooting under clear sky and indoor shooting), and St is the variance in all samples. Note that the smaller the Sb / St value, the smaller the distance between classes (between shooting under clear sky and indoor shooting), and the smaller the variation in output value between different light sources. Here, as shown in FIG. 15, when the value of a is varied, it can be seen that the Sb / St value is the smallest when a = 0.5.

Considering this, when the signal values of the R color component, the G color component, and the B color component are LSB = 512, the corrected R gain value α _R , G gain value α _G , and B gain value α _B are respectively [Expression 45] Rgain, Ggain, and Bgain are gain values of the respective color components for white balance processing.

[Equation 45]
α _R = 0.5 × Rgain + 512
α _G = Ggain = 512
α _B = 0.5 × Bgain + 512

Further, when the signal values of the R color component and the B color component are LSB = 256 and the signal value of the G color component is LSB = 512, the corrected R gain value α _R , G gain value α _G , and B gain value α _B Is [Equation 46].

[Equation 46]
α _R = Rgain + 512
α _G = Ggain = 512
α _B = Bgain + 512

Using these α _R , α _G , and α _B , RGB data is converted into CrCb data. Here, general RGB → YCrCb conversion is expressed by [Equation 47].

As described above, in the present invention, the R gain value α _R , the G gain value α _G , and the B gain value α _B corrected at the time of color space conversion are multiplied. Therefore, the equation used when converting the color space from RGB to CrCb is [Equation 48].

  That is, the matrix C used when obtaining the CrCb data from the RGB data is represented by [Equation 49] from [Equation 47] and [Equation 48].

なお、マトリクスの各値は、［数５０］のように、整数化した値を用いてもよい。

As each value of the matrix, an integerized value such as [Equation 50] may be used.

[Equation 50]
C ₀₀ = (50 × α _R ) / 100
C ₀₁ = (− 42 × α _G ) / 100
C ₀₂ = (− 8 × α _B ) / 100
C ₁₀ = (− 17 × α _R ) / 100
C ₁₁ = (− 33 × α _G ) / 100
C ₁₂ = (50 × α _B ) / 100

(Convert RGB data to CrCb data)
The AF pixel interpolation unit 45 converts the RGB data of the pixels included in the extracted target area into CrCb data using the obtained matrix (S-65). The AF pixel interpolation unit 45 refers to the position information of the target area, and performs matrix conversion of RGB data of the pixels included in the target area into CrCb data using [Formula 6] and [Formula 7].

  When the AWB evaluation data is not YRGB data but YCrCb data, it is not necessary to perform the color space conversion process described above. In this case, the Cr value and the Cb value may be corrected using a value obtained by correcting each color gain value used in the white balance process with respect to the Cr value and the Cb value.

  Moreover, although the case where the RGB data of the pixels included in the extracted target area is converted into CrCb data has been described, it is also possible to convert the entire RGB data into CrCb data.

(Detection of magenta area included in target area)
The AF pixel interpolation unit 45 determines whether or not the pixel value of each pixel converted into CrCb data satisfies [Equation 51] to [Equation 53] (S-66).

[Formula 51]
Cr <Th13 × Cb and Cr> (Th14 / 256) × Cb

[Formula 52]
Cb> Th15 or Cr> Th15

[Formula 53]
R <Th11 and G <Th11 and B <Th11

  Here, [Equation 9] is a conditional expression for determining whether or not the pixel value (Cr, Cb) of the pixel included in the target region falls within the magenta color gamut. As shown in FIG. 16A, in the chromaticity diagram, the range indicated by diagonal lines is the range indicated by [Equation 51]. [Formula 52] is a conditional expression for determining whether or not the pixel value (Cr, Cb) of the pixel has a predetermined intensity value. Here, when each of the Cr value and the Cb value is “0” or a value close to 0, the intensity of the magenta color is weak. Therefore, a case where any one of the Cr value and the Cb value exceeds a predetermined value (here, the threshold Th5) may be considered. Therefore, the range obtained from [Equation 52] and [Equation 53] (the range indicated by hatching in FIG. 16B) is the magenta color gamut, and the pixel values of the pixels included in the high luminance region are included in this color gamut. Whether or not is included is determined.

  Furthermore, [Equation 53] is a conditional expression for determining whether or not the pixel values of the R, G, and B color components of the pixels included in the target region are saturated.

  The AF pixel interpolation unit 45 extracts pixels satisfying the above-described [Equation 51] to [Equation 53] from the pixels included in the target region. At this time, the AF pixel interpolation unit 45 obtains the absolute difference sum SAD of the edge amounts of the Y components in the four adjacent directions by using [Expression 54].

  The AF pixel interpolation unit 45 divides the absolute difference sum SAD of the edge amounts of the Y component obtained by [Equation 54] by the normalization factor value (= 4) of the edge amount to obtain the average value of the edge amounts of the Y component. Ask. The AF pixel interpolation unit 45 determines whether the average value of the Y component edge amounts is equal to or less than the threshold Th16. Here, the AF pixel interpolation unit 45 calculates the average value of the edge amount of the Y component and then compares it with the threshold Th16. However, the present invention is not limited to this, and the edge amount of the Y component and the threshold are calculated. You may compare.

  The AF pixel interpolation unit 45 calculates the number of specified pixels every time a pixel that satisfies [Equation 51] to [Equation 53] and the average value of the edge amount of the Y component is equal to or less than the threshold Th16 is specified. to add. Further, the AF pixel interpolating unit 45 calculates the total edge amount edge_sum by adding the absolute difference sum SAD of the obtained Y component edge amounts, and used to obtain the average value of the edge amounts in the Y direction. The normalization factor value edge_count is added.

  Further, the AF pixel interpolation unit 45 obtains the x coordinate of the specified pixel, the value obtained by squaring the x coordinate (square value of the x coordinate), the y coordinate, and the value obtained by squaring the y coordinate (square value of the y coordinate), respectively. to add. Hereinafter, the value obtained by adding the x coordinate is the x coordinate added value ave_x, the value obtained by adding the square value of the x coordinate is the x coordinate square added value sqr_x, the value obtained by adding the y coordinate is the y coordinate added value ave_y, and the square value of the y coordinate. A value obtained by adding is described as a y-coordinate square addition value sqr_y.

(Evaluation value judgment)
The AF pixel interpolation unit 45 obtains first to third evaluation values described below and determines these evaluation values (S-67). The AF pixel interpolating unit 45 determines whether or not flare has occurred by determining each of these evaluation values.

  The first evaluation value is [total area of the magenta area], which corresponds to the total number of pixels specified as the magenta area. The AF pixel interpolation unit 45 determines whether the first evaluation value exceeds the threshold Th17 (whether the first evaluation value> the threshold Th17).

  The second evaluation value is [dispersion value / total area of magenta region]. In general, the variance value is obtained by [Equation 55].

  When the dispersion value represented by [Expression 55] is replaced with an expression corresponding to the spatial dispersion value (spatial moment), [Expression 56] is obtained.

  Using this equation, the AF pixel interpolation unit 45 calculates the total area of the dispersion value / magenta region using [Equation 57].

  The AF pixel interpolation unit 45 sets [dispersion value / total area of magenta area] obtained in [Equation 57] as the second evaluation value, and whether the second evaluation value is less than the threshold Th18 (second evaluation value < Whether the threshold value is Th18). Although the second evaluation value <threshold Th18 is set, the present invention is not limited to this, and the second evaluation value <threshold Th18 / 32 may be used. In this case, 32 is a normalization factor value assigned to the second evaluation value.

  The third evaluation value is [average value of edge amounts of Y component]. The AF pixel interpolation unit 45 calculates the average value of the edge amounts of the Y component using [Equation 58].

[Formula 58]
Yedge = edge_sum / edge_count

  The AF pixel interpolating unit 45 sets the [average value of the edge amount of the Y component] obtained in [Equation 58] as the third evaluation value, and whether the third evaluation value is less than the threshold value TH19 (third evaluation value < Whether the threshold value is Th19). In the evaluation value determination, [total area of magenta area] is the first evaluation value, [dispersion value / total area of magenta area] is the second evaluation value, and [average value of edge amount of Y component] is the third evaluation value. However, the present invention is not limited to this, and [total area of magenta area], [dispersion value / total area of magenta area], [Y component] It is also possible to perform the evaluation value determination using at least two of the average values of the edge amounts] as evaluation values.

(Determining whether flare has occurred)
By performing the evaluation value determination process described above, it is determined whether or not flare has occurred in the captured image data (S-68). In the evaluation value determination described above, the AF pixel interpolation unit 45 at least one of the conditions of [first evaluation value> threshold Th17], [second evaluation value <threshold Th18], and [third evaluation value <threshold Th19]. When one condition is not satisfied, it is determined that no flare has occurred. In this case, the AF pixel interpolation unit 45 executes the first pixel interpolation process (S-69). On the other hand, if all these conditions are satisfied, the AF pixel interpolation unit 45 determines that flare has occurred.

(Weight level setting)
The AF pixel interpolating unit 45 sets the weight level for the pixels included in the magenta area recognized as a color mixture caused by flare (S-68). The setting of the weight level is the same as that in the present embodiment, and thus the description thereof is omitted here.

  In this case, the same effect as that of the present embodiment can be obtained.

  In the present embodiment, the first pixel interpolation process is performed after the process of suppressing flare (the process from S-41 to S-47) in the second pixel interpolation process, but this is not limitative. It is not necessary, and it is also possible to execute the first pixel interpolation process after performing a smoothing process (S-41 to S-47) for suppressing flare a plurality of times. When the smoothing process for suppressing flare is performed a plurality of times, the value of the coefficient K_Th_LPF used when calculating the clip amount may be different between the value used for the first calculation and the value used for the second and subsequent calculations. It is also possible to use the same value.

  In the present embodiment, the AF pixel interpolation unit 45 that performs the flare determination process is described. However, the present invention is not limited to this, and the AF pixel interpolation unit that performs the noise determination process in addition to the flare determination process may be used. Good. In this case, noise determination processing is performed on the image data. In this noise determination process, when it is determined that there is little noise, the flare determination process in the present embodiment may be executed. As noise determination in this case, for example, noise determination based on information on the temperature, ISO sensitivity, and shutter speed of the image sensor at the time of shooting may be performed.

  In this embodiment, the case where flare determination is performed on an image having imaging pixels and AF pixels is described. However, the present invention is not limited to this, and an image including only imaging pixels is described. Also, the present invention can be used.

  In the present embodiment, the electronic camera 10 is described as an example, but the present invention can also be used for a portable terminal such as a camera-equipped mobile phone or a smartphone.

  In addition to this, the functions of FIGS. 4 and 5 and the image processing program for realizing the processing in the flowcharts of FIGS. 6, 8 and 10 by a computer, the functions of FIG. 11, and the flowchart of FIG. The present invention can also be applied to any one of the image processing programs for realizing the above. The image processing program is preferably stored in a computer-readable storage medium such as a memory card, an optical disk, or a magnetic disk.

  Further, the present invention may be an invention of an image processing program for capturing image data such as RAW data in which AF pixel values are not interpolated and determining flare and performing AF pixel interpolation, and a storage medium storing the image processing program.

  DESCRIPTION OF SYMBOLS 10 ... Electronic camera, 17 ... Imaging device, 25 ... Image processing part, 27 ... SDRAM, 41 ... AF pixel, 45 ... AF pixel interpolation part, 46, 60 ... Flare determination part, 47 ... Color detection part, 48 ... Edge detection 49, evaluation value determination unit, 50, 67 ... weight setting unit, 61 ... histogram generation unit, 62 ... threshold calculation unit, 63 ... target region extraction unit, 64 ... matrix determination unit, 65 ... magenta region detection unit, 66 ... Evaluation value judgment unit

Claims (7)

An image input unit for receiving images;
A flare determination unit that determines whether or not flare has occurred based on information related to a region of a predetermined color component included in the image;
Setting that sets a weight based on a color component value of the pixel at least for a pixel included in the predetermined color component region caused by the flare when it is determined that the flare has occurred And
A smoothing unit that performs a smoothing process on the image based on the weight set by the setting unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
An area detector for detecting an area of the predetermined color component from the image;
An edge detection unit for detecting an edge included in the image;
With
The flare determination unit determines whether or not the flare is generated based on a detection result of the region detection unit and a detection result of the edge detection unit. The image processing apparatus according to claim 1.
The setting unit divides a color gamut that the predetermined color component in the chromaticity diagram can take into a plurality of regions, and a color component value of each pixel included in the region of the predetermined color component caused by the flare is An image processing apparatus, wherein the weight is set for each pixel included in the region of the predetermined color component caused by the flare by determining which of the plurality of regions belongs. The image processing apparatus according to claim 3.
The setting unit includes pixels that are included in a region of the predetermined color component caused by the flare and pixels included in a region excluding the region of the predetermined color component caused by the flare in the image. An image processing apparatus characterized in that a weight different from the weight set in the above is set. The image processing apparatus according to claim 1.
The image has imaging pixels and focus detection pixels,
The image processing apparatus further includes a pixel interpolation unit that generates an interpolation pixel value for the focus detection pixel by performing an interpolation process on the image received by the image input unit or the image subjected to the smoothing process. A featured image processing apparatus. An image sensor for acquiring the image;
The image processing apparatus according to any one of claims 1 to 5,
An imaging apparatus comprising: An image input process for receiving images;
A flare determination step for determining whether or not flare has occurred based on information relating to a region of a predetermined color component included in the image;
Setting that sets a weight based on a color component value of the pixel at least for a pixel included in the predetermined color component region caused by the flare when it is determined that the flare has occurred Process,
A smoothing step of executing a smoothing process on the image based on the weight set in the setting step;
An image processing program capable of causing a computer to execute.

Images