JP2014036262A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To optimally perform pixel interpolation for focus detection even if lens information cannot be obtained.SOLUTION: An imaging element 14 has a phase difference AF pixel and obtains a Bayer array image by imaging a subject image formed by a photographic lens 24. A non-volatile memory 12 stores the position of a pixel for focus detection and also stores a parameter group determined per lens condition. A lens communication detection unit 17 detects whether data communication with an interchangeable lens 15 is possible or not. An AF pixel interpolation unit 50 interpolates, if communication is not possible, a pixel value for focus detection with a pixel value for interpolation imaging corresponding to a pixel value for imaging for each of a plurality of parameters selected from the parameter group relative to reference image data imaging a surface light source with uniform luminance. A parameter estimation unit 19 estimates as an optimum a parameter corresponding to a pixel value for interpolation imaging with the minimum error between them by comparing a pixel value for interpolation imaging and a pixel value for peripheral imaging per parameter. The AF pixel interpolation unit 50 interpolates the pixel value for focus detection with the pixel value for interpolation imaging by use of the estimated parameter at the time of real photographing.

Description

  The present invention relates to an imaging apparatus.

  2. Description of the Related Art Conventionally, there has been known an imaging element in which a plurality of pixels for focus detection are arranged on a part of a light receiving surface in which a plurality of imaging pixels are arranged in a two-dimensional manner (Patent Document 1). The plurality of imaging pixels have spectral characteristics corresponding to each of the plurality of color components, and the focus detection pixels have spectral characteristics different from the plurality of imaging pixels. In the image processing apparatus described in Patent Literature 1, an image pickup pixel value corresponding to the position of a focus detection pixel is interpolated, and then a pixel value of a missing color component among the image pickup pixel values is interpolated to obtain a three-color image. Create data.

  When interpolating an imaging pixel value corresponding to the position of the focus detection pixel, an interpolation pixel of the focus detection pixel using the pixel value of the imaging pixel near the focus detection pixel in the image generated by the imaging element A value is generated, and an evaluation pixel value, which is a pixel value when a nearby imaging pixel has the same spectral characteristics as the focus detection pixel, is calculated.

  The evaluation pixel value is calculated by obtaining a weighted sum using a plurality of imaging pixel values in the vicinity of the focus detection pixel together with the weighting coefficient. The weighting coefficient is a parameter for representing the spectral characteristic of the focus detection pixel by the weighted sum of the spectral characteristics of the plurality of imaging pixels, and is stored in the storage unit in advance.

  The weighting coefficient differs depending on the spectral sensitivity of the focus detection imaging pixel and the spectral sensitivity of the surrounding imaging pixels. In addition, the weighting coefficient varies depending on the lens conditions because the light collection changes according to the lens conditions obtained at the time of imaging, that is, the aperture (F) value and the exit pupil position (PO value). It is desirable. Therefore, the interchangeable lens is provided with a CPU, a memory, a communication contact, and the like, and lens information such as an F value set at the time of shooting and a PO value stored in the memory in advance is sent to the camera body.

JP 2009-303194 A

  However, in the case of an interchangeable lens that does not incorporate a CPU, memory, or the like, lens information cannot be recognized on the camera body side. In this case, for example, processing is performed using parameters that are fixed values prepared in advance, and interpolation is not necessarily performed with optimal parameters. It was.

  An object of the present invention is to provide an imaging apparatus capable of optimally performing AF pixel interpolation processing even when an interchangeable lens that cannot acquire lens information is used.

  One embodiment of an imaging device illustrating the present invention includes an imaging element that has a plurality of imaging pixels and a plurality of focus detection pixels and that captures a subject image formed by a photographic lens, and positions of the focus detection pixels. Peripheral imaging obtained from the positions of the surrounding imaging pixels, the AF pixel position storage means for storing each, and the focus detection pixel value obtained from the position of the focus detection pixel for the image data captured by the imaging device And interpolating to the interpolated imaging pixel value corresponding to the imaging pixel value using a pixel value and a parameter determined based on a lens condition acquired at the time of imaging from among a predetermined parameter group for each lens condition When the lens condition cannot be acquired at the time of shooting, the focus detection image included in the reference image data obtained by imaging the subject having a uniform luminance distribution by the imaging device. A pixel interpolating means for interpolating the interpolated imaging pixel value for each of a plurality of parameters selected from the parameter group; an interpolated imaging pixel value interpolated for each of the parameters output from the pixel interpolating means; Parameter estimation means for comparing the peripheral imaging pixel values included in the reference image data and estimating an optimum parameter from the plurality of parameters based on a comparison result.

  According to the present invention, it is possible to set an optimum parameter for focus detection pixel interpolation even when an interchangeable lens that cannot acquire lens information is attached.

It is a block diagram which shows the outline of the electronic camera which is one aspect | mode of the imaging device of this invention. It is explanatory drawing which shows the pixel row for phase difference focus detection provided in the light-receiving surface of an image pick-up element. It is explanatory drawing which shows the principal part of the pixel for focus detection. It is a flowchart which shows the outline of the operation | movement procedure of an electronic camera. It is a flowchart which shows the operation | movement procedure of a parameter estimation process. FIG. 3 is a perspective view illustrating one aspect of a document, illumination, and the like used when an object with uniform luminance distribution is imaged with an electronic camera. It is a flowchart which shows the operation | movement procedure of a pixel interpolation process. It is explanatory drawing which represented the pixel value obtained from AF pixel and the pixel for imaging at each pixel position. It is explanatory drawing which shows roughly the process which estimates the parameter corresponding to the pixel value data for imaging containing the least error amount. It is explanatory drawing which shows a pixel value when the vertical 5 pixel image structure containing a convex structure is cut | disconnected vertically. It is explanatory drawing which shows another example which provided the peripheral imaging pixel row | line | column in the position distant from AF pixel row | line | column.

  An electronic camera 10 to which the present invention is applied includes a CPU 11 as shown in FIG. A non-volatile memory 12 and a work memory 13 are connected to the CPU 11, and the non-volatile memory 12 stores a control program that is referred to when the CPU 11 performs various controls. Further, in the nonvolatile memory 12, position information of focus detection pixels (AF pixels) for each type of the image sensor 14, which will be described in detail later, various threshold data used for an image processing program obtained in advance, and focus detection A parameter table describing parameters (weighting coefficients) for interpolating the pixel values for imaging to the imaging pixel values, various determination tables, and the like are stored.

  The CPU 11 controls each part using the work memory 13 as a temporary storage work area according to a control program stored in the non-volatile memory 12 and operates each part (circuit) function constituting the electronic camera 10.

  As the parameters, parameters corresponding to lens conditions at the time of photographing are used when performing pixel interpolation processing for interpolating pixel values obtained from AF pixels to imaging pixel values. The lens conditions at the time of shooting are an exit pupil position (PO value) determined for each type of the interchangeable lens 15 and an aperture (F) value determined at the time of shooting.

  The interchangeable lens 15 is provided with a plurality of different types such as a focal length and an open aperture (F) value, and is detachably attached to the mount portion of the electronic camera 10. In addition to the single-focus imaging lens 24, the interchangeable lens 15 includes a CPU (not shown), a memory, a communication contact, and the like, and can perform data communication with the CPU 11 of the camera body. When using the interchangeable lens 15 that can communicate in this way, the CPU 11 can automatically acquire the lens conditions obtained at the time of imaging. Note that wireless communication may be used instead of wired communication such as communication contacts.

  Some interchangeable lenses 15 are not compatible with the mount portion of the camera body. Therefore, a conversion adapter 16 is provided. The conversion adapter 16 includes first and second mount parts, an input part, and an output part. The first mount portion is detachably attached to the interchangeable lens 15. The second mount part is detachably attached to the camera body. Lens information is input from the interchangeable lens 15 to the input unit. The output unit outputs the lens information input to the input unit to the CPU 11 of the camera body. As described above, the interchangeable lens 15 having incompatible mount portions includes those that can perform data communication and those that cannot.

  For this reason, the electronic camera 10 includes a lens communication detection unit 17 and a lens condition acquisition unit 18. The lens communication detection unit 17 detects whether or not the conversion adapter 16 is attached, and determines that the interchangeable lens 15 is attached when detecting that the conversion adapter 16 is not attached.

  Thereafter, it is detected whether or not the interchangeable lens 15 attached can communicate. If communication is possible, the lens condition acquisition unit 18 communicates with the CPU of the interchangeable lens 15 to acquire lens conditions and sends the information to the CPU 11. The CPU 11 includes a parameter determination unit 20, and the parameter determination unit 20 stores one parameter that is optimal at the time of shooting based on the lens information automatically acquired from the interchangeable lens 15 at the time of shooting. 12 is automatically determined with reference to a parameter table stored in advance in the table 12.

  When the interchangeable lens 15 that cannot communicate is used, the lens condition cannot be acquired. Therefore, the CPU 11 includes a parameter estimation unit 19. The parameter estimation unit 19, based on the reference image data obtained by imaging the surface light source with uniform brightness, details the imaging pixel value interpolated for each of a plurality of parameters by the AF pixel interpolation unit 50 described later, and the focus detection pixel. The peripheral imaging pixel values obtained from the peripheral imaging pixels are compared, and the optimum parameter is determined based on the comparison result based on the lens conditions at that time.

  The operation unit 37 includes an adjustment mode selection operation unit. The adjustment mode selection operation unit is an operation unit for selecting an adjustment mode. The adjustment mode is a mode that is selected when the interchangeable lens 15 that cannot communicate is used, and executes processing for estimating parameters when the lens conditions are unknown. In the case of using the interchangeable lens 15 that cannot communicate, the CPU 11 responds to the selection of the adjustment mode, for example, an instruction to shoot a reference subject such as “shoot a subject with a uniform luminance distribution”. The display unit 21 is controlled to display a message for prompting to image a subject (reference subject) having a uniform luminance distribution, and based on the reference image data acquired by the imaging, the lens conditions at the time of the next main photographing Estimate the optimal parameters. Note that the CPU 11 may control to automatically display the message in response to detecting that communication with the interchangeable lens 15 is not possible.

Subject light incident from the photographing lens 24 forms an image on the light receiving surface of the image sensor 14 such as a CCD or CMOS. The image sensor drive circuit 25 drives the image sensor 14 based on a control signal from the CPU 11. The image sensor 14 is a Bayer array type single-plate image sensor, and a primary color transmission filter 26 is attached to the front surface.
The primary color transmission filter 26 is, for example, a primary color Bayer in which the resolution of G (green) is N / 2 and the resolution of R (red) and B (blue) is N / 4 with respect to the total number N of pixels of the image sensor 14. It is an array.

  The subject image formed on the light receiving surface of the image sensor 14 is converted into an analog image signal. The image signal is sequentially sent to a CDS 27 and an AMP 28 constituting an AFE (Analog Front End) circuit, subjected to predetermined analog processing, and then converted into digital image data in an A / D (Analog / Digital converter) 29. After being converted, it is sent to the image processing unit 30.

  The image processing unit 30 includes a separation circuit, a white balance processing circuit, a pixel interpolation (demosaicing) circuit, a matrix processing circuit, a non-linear transformation (γ correction) processing circuit, an edge enhancement processing circuit, and the like. Thus, processing such as white balance, pixel interpolation, matrix, nonlinear conversion (γ correction), and contour enhancement is performed. The separation circuit separates a signal output from an imaging pixel, which will be described in detail later, and a signal output from the focus detection pixel. The pixel interpolation circuit converts Bayer array data of one color per pixel into normal color image data composed of three colors per pixel.

  The three-color image data output from the image processing unit 30 is stored in the SDRAM 32 through the bus 31. The image data stored in the SDRAM 32 is read out under the control of the CPU 11 and sent to the display control unit 33. The display control unit 33 converts the input image data into a predetermined signal for display (for example, an NTSC color composite video signal), and outputs the converted signal to the display unit 21 as a through image.

  The image data acquired in response to the shutter release is read from the SDRAM 32 and then sent to the compression / decompression processing unit 34. After the compression processing is performed here, the card memory 36 which is a recording medium via the media controller 35. To be recorded. An operation unit 37 is connected to the CPU 11. The operation unit 37 includes a shutter button, a power switch, a shutter speed setting unit, an aperture setting unit, an ISO sensitivity setting unit, and the like in addition to the adjustment mode selection operation unit described above.

  The AE / AF detection unit 39 detects the defocus amount and the defocus direction by the pupil division type phase difference detection method based on the focus detection pixel signal. The CPU 11 controls the driver 40 based on the defocus amount and the defocus direction obtained by the AE / AF detection unit 39 to drive the lens driving unit 41 to move the focus lens of the photographing lens 24 in the optical axis direction. Adjust the focus by moving forward and backward.

  Further, the AE / AF detection unit 39 calculates a light value (Lv = Sv + Bv) by using the luminance value (Bv) calculated based on the image pickup pixel signal and the Sv value set by the ISO sensitivity setting unit. . Then, the AE / AF detection unit 39 determines the aperture value and the shutter speed so that the exposure value (Ev = Av + Tv) becomes the light value (Lv). In accordance with this, the CPU 11 drives the diaphragm drive unit 43 via the electronic shutter of the image sensor 14 and the driver 42 to control the aperture of the diaphragm 44 provided on the interchangeable lens 15 to obtain an appropriate exposure amount.

  The imaging device 14 is provided with a primary color transmission filter 26 of R (red), G (green), or B (blue) in a Bayer array type in each of a plurality of imaging pixels on the light receiving surface. A CCD or CMOS semiconductor image sensor with a microlens array provided thereon is appropriately selected and used. Furthermore, the image sensor 14 of the present embodiment has a plurality of focus detection pixels (AF pixels) arranged one-dimensionally in the horizontal scanning direction in a partial region on the light receiving surface. In the present embodiment, an example in which the primary color transmission filter 26 is not provided in each AF pixel will be described.

  The AF pixels are two types of phase difference AF pixels that receive light beams that pass through the left or right side of the pupil of the optical system of the imaging lens 24. The image sensor 14 can individually read out pixel signals from the imaging pixel group and the AF pixel group.

  As shown in FIG. 2, in the AF pixel group, AF pixel rows (straight lines shown in the figure) 47 closely arranged in the horizontal scanning direction on the light receiving surface 14a are separated by a predetermined pitch in the vertical scanning direction. It is arranged and configured. A plurality of imaging pixel columns are arranged between the AF pixel columns 47. Among the plurality of AF pixel rows 47, the uppermost and lower AF pixel rows 47a and 47z are reduced in length by omitting the portions included in the four corners of the light receiving surface 14a where the necessity of focus detection is low.

  As shown in FIG. 3, each AF pixel 46 has sensor openings 46a and 46b that are biased to one side from the cell center (the center of the microlens), and are arranged one-dimensionally along the direction of the bias. . The adjacent sensor openings 46a and 46b are biased in opposite directions, and the bias distance is the same. The AF pixel 46 having the sensor opening 46a is placed in place of the G pixel in the RGB primary color Bayer array, and the AF pixel 46 having the sensor opening 46b is placed in place of the B pixel in the RGB primary color Bayer array. It is. The pupil division phase difference AF method is realized by the AF pixel 46 having such sensor openings 46a and 46b. That is, of the two light fluxes passing through the exit pupil, two partial light fluxes that are symmetric with respect to the optical axis of the photographic lens 24 are separated by the AF pixel 46 having the sensor aperture 46a and the AF pixel 46 having the sensor aperture 46b. If each light is received, the defocus direction and the defocus amount can be known from the phase difference between the signals output from these two AF pixels 46. This enables quick focusing.

  Therefore, each AF pixel 46 in the present embodiment outputs a detection signal obtained by dividing the left or right pupil according to the luminance of white light. FIG. 3 shows a part of the image data centered on the area where the AF pixel 46 is arranged in the image data captured by the image sensor 14. Each cell represents one pixel. Symbols R, G, and B at the head of each cell indicate an imaging pixel having each primary color transmission filter 26. On the other hand, symbols X and Y indicate AF pixels that are sensitive to light beams from the left side or the right side, and they are alternately arranged one-dimensionally in the horizontal scanning direction. The two-digit number following these symbols indicates the pixel position.

  The image processing unit 30 interpolates the pixel value of the AF pixel 46 (AF pixel value (focus detection pixel value)) to the pixel value of the imaging pixel (interpolated imaging pixel value) (pixel interpolation means). 50, and after interpolating the AF pixel value to the interpolation imaging pixel value, color interpolation is performed by the linear interpolation method from the Bayer array to RGB.

  When the lens communication with the interchangeable lens 15 is impossible or when the adjustment mode is selected, the AF pixel interpolation unit 50 performs a pixel interpolation process that performs pixel interpolation using the AF pixel value. In the pixel interpolation process, the AF pixel value is interpolated by predicting with a weighted sum based on the surrounding imaging pixel value (peripheral imaging pixel value) from the AF pixel value (white (W) component). At this time, one parameter corresponding to the lens condition at the time of shooting, that is, a weighting coefficient described later in detail, is selected and used for each color component of the pixel among a plurality of predetermined parameters.

  Next, the operation of the above configuration will be described with reference to FIG. In response to turning on the power switch (S-1), the CPU 11 monitors the mode selected by the mode selection operation unit (S-2). When the adjustment mode is selected, parameter estimation processing is executed (“Y” side of S-2). In a shooting mode other than the adjustment mode, it is determined whether or not data communication can be performed with the interchangeable lens 15 attached at that time (“N” side of S-2). Here, in this embodiment, the interchangeable lens 15 compatible with the mount that can be mounted without using the conversion adapter 16 is capable of data communication and is compatible with the mount using the conversion adapter 16. Description will be made on the assumption that there are some interchangeable lenses 15 that are not capable of data communication and those that are not.

  The lens communication detection unit 17 detects whether or not the conversion adapter 16 is attached in response to the instruction to execute the parameter determination process (S-3), and then detects that the conversion adapter 16 is attached. It is detected whether data communication is possible (S-4). When it is detected that the conversion adapter 16 is attached and data communication with the interchangeable lens 15 is possible ("Y" side of S-4), the lens condition acquisition unit 18 is operated. A parameter determination process is executed (S-5). If the attachment of the conversion adapter 16 is not detected (“N” side of S-3), it is assumed that the interchangeable lens 15 capable of data communication is attached, and the parameter determination process is executed in the same manner as described above. (S-5).

[Parameter determination processing]
The parameter determination process is a process of acquiring the lens condition at the time of shooting from the interchangeable lens 15 and automatically determining the parameter based on the acquired lens condition.
Specifically, the lens condition acquisition unit 18 automatically reads and acquires lens condition information from the memory via the CPU of the interchangeable lens 15 in response to a half-pressing operation or full-pressing operation of the shutter release. . The acquired lens condition information includes a PO value and an F value (aperture value) corresponding to the type of the interchangeable lens 15. The parameter determination unit 20 obtains an optimum parameter at the time of shooting based on the PO value and the F value. The parameters are described in a parameter table stored in advance in the nonvolatile memory 12. In the parameter table, for example, as shown in [Table 1], the horizontal axis (row) has the PO value, the vertical axis (column) has the F value, and different parameter groups are described at the intersection of the matrix (matrix). It has a form. The parameter table is prepared for each color component of the pixel. The parameters obtained from the parameter table are temporarily stored in the work memory 13, and are used when pixel interpolation processing (S-8) is performed on image data acquired by actual photographing, which will be described in detail later.

  When it is detected that the conversion adapter 16 is attached and data communication is not possible, the parameter estimation unit 19 executes a parameter estimation process for estimating parameters according to the lens conditions at the time of shooting (S-6). ). The parameter estimation process is executed by the parameter estimation unit 19 of the CPU 11 when the lens condition cannot be obtained from the interchangeable lens 15, and for the reference image data obtained by imaging the surface light source having a uniform luminance with the imaging device 14. Pixel interpolation processing is performed for each of the plurality of parameters, and the pixel value for imaging (interpolation imaging pixel value) interpolated for each parameter and the pixel values of the imaging pixels around the AF pixel (peripheral imaging pixel value) , And a parameter that is optimal for the lens condition at the time of shooting is estimated from a plurality of parameters based on the comparison result. Note that when data communication with the interchangeable lens 15 is not possible, the focus of the photographing lens 24 and the aperture 44 cannot be controlled automatically, so the focus position, shutter speed, and aperture value are determined in advance. Manual shooting is performed.

  After the parameter is determined or estimated, actual photographing is performed (S-7). Image data acquired from the image sensor 14 by the main photographing is taken into the image processing unit 30. The AF pixel interpolation unit 50 generates image data for recording or reproduction by interpolating the AF pixel value included in the captured image data into the imaging pixel value using the determined or estimated parameter (S -8). The generated image data is compressed by the compression / decompression processor 34 and then recorded in the card memory 36 (S-9).

[Parameter estimation processing]
The parameter estimation unit 19 executes parameter estimation processing when data communication with the interchangeable lens 15 is not possible or when the adjustment mode is selected. In the parameter estimation process, it is assumed that an aperture value is set in advance by manual shooting. As shown in FIG. 5, the parameter estimation unit 19 displays a message prompting an instruction to shoot a reference subject, such as “Please shoot a subject with a uniform luminance distribution” on the outside, for example, the display unit 21 ( S-10). Note that a speaker may be provided and externally displayed with sound. As a specific method for photographing a subject having a uniform luminance distribution, as shown in detail in FIG. 6, a plain transparency chart 48 without a pattern is illuminated from the back with a uniform luminance by a diffusion plate 51. Then, the image of the chart 48 (image of the reference subject) is picked up by the electronic camera 10. The diffusion plate 51 uniformly diffuses the light emitted from the light source 49.

  After displaying the message externally, the CPU 11 monitors the shutter button in order to determine whether shooting has been performed (S-11). If it is determined that shooting has been performed, the CPU 11 determines whether or not the reference image data captured by the image processing unit 30 has a uniform luminance in response to the shutter release. If it is determined that the brightness is not uniform (“N” side of S-12), a message is displayed outside so as to capture again (S-10). When it is determined that the luminance is uniform (“Y” side in S-12), the parameter estimation unit 19 controls the AF pixel interpolation unit 50 to change the parameter with respect to the reference image data and execute pixel interpolation processing. (S-13). In addition, as the determination of whether or not the brightness is uniform, for example, the average value of the brightness value of each area arbitrarily divided with respect to the reference image data is calculated, and the maximum value and the minimum value of the average value of each area are calculated. It is desirable to determine uniformity based on this.

  The AF pixel interpolation unit 50 selects a plurality of parameters from a parameter group (parameter group shown in [Table 1]) determined based on the lens condition. As a method for selecting a plurality of parameters, for example, parameters described in a diagonal line from the upper left to the lower right of the parameter group shown in a matrix form in [Table 1] may be selected in a jumping manner. The matrix-like parameter group may be divided into a plurality of regions, and any one of a plurality of parameters included in the divided regions may be selected for each region.

  After selecting a plurality of parameters, the AF pixel interpolation unit 50 performs pixel interpolation processing for interpolating the AF pixel values included in the reference image data into the interpolation imaging pixel values for each of the selected parameters (S-13). ). The parameter estimation unit 19 acquires interpolation imaging pixel value data including interpolation imaging pixel values obtained by interpolating a plurality of AF pixel values for each parameter, and includes interpolation imaging pixel values included in the interpolation imaging pixel value data; The pixel value (peripheral imaging pixel value) obtained from the imaging pixels around the AF pixel to be interpolated is compared (S-14), and the difference between the peripheral imaging pixel value and the interpolation imaging pixel value is compared. The parameter corresponding to the pixel value data for interpolation imaging with the least number is selected, and the selected parameter is estimated as a suitable parameter at the time of imaging and stored in the work memory 13 (S-15).

[Pixel interpolation processing in parameter estimation processing]
The pixel interpolation processing in the AF pixel interpolation unit 50 will be described with reference to FIG. Note that the pixel interpolation processing shown in FIG. 7 is the same as the pixel interpolation processing performed on the image data acquired at the time of actual photographing. The difference is the threshold value th1, threshold value th2, and threshold value described later in detail in the case of parameter estimation processing. The value of th3 is set to an extremely large value, and the processes of “S-21” to “S-24” and “S-26” to “S-29” are necessarily executed.

  Here, in this embodiment, since the primary color transmission filter 26 installed in each of the imaging pixels is a Bayer array pattern, the position of the AF pixel of the symbol X in the AF pixel row shown in FIG. The green (G) imaging pixel value is interpolated, and the blue (B) imaging pixel value is interpolated at the pixel position of the AF pixel of symbol Y. In the following description of the pixel interpolation processing, a case will be described in which interpolation is performed for the Y44 blue imaging pixel value and the X45 green imaging pixel value shown in FIG. The procedure for interpolating to the imaging pixel values in other AF pixels is the same.

  The CPU 11 reads out AF pixel position information based on information about the image sensor 14 from the nonvolatile memory 12 and sends the AF pixel position information to the AF pixel interpolation unit 50. After specifying the position of the AF pixel, the AF pixel interpolation unit 50 uses the imaging pixel values around the specified AF pixel to determine the direction in which the variation value, which is the change rate of the pixel value, is minimized. Then, the AF pixel value is interpolated using the imaging pixel value in the direction of the smallest fluctuation.

(Calculate the direction of the minimum fluctuation value)
The AF pixel interpolation unit 50 uses four pixel values of the surrounding pixels of X45 and Y44 to interpolate the pixel values of the X45 and Y44 AF pixels shown in FIG. The values of the direction fluctuations H1 to H4, which are the rate of change of the values, are obtained using the equations described in [Equation 1] to [Equation 4] (S-16). In this embodiment, the four directions are 45 degrees and 135 degrees with respect to the horizontal scanning direction, the vertical scanning direction, and the horizontal scanning direction.

[Equation 1]
Direction variation H1 in the horizontal scanning direction =
2 × (| G34-G36 | + | G54-G56 |) + | R33-R35 | + | R53-R55 | + | B24-B26 | + | B64-B66 |

[Equation 2]
Direction variation H2 in the vertical scanning direction =
2 × (| G34-G54 | + | G36-G56 |) + | R33-R53 | + | R35-R55 | + | B24-B64 | + | B26-B66 |

[Equation 3]
Directional variation H3 = 45 degrees with respect to the horizontal scanning direction =
2 × (| G27-G36 | + | G54-G63 |) + | R35-R53 | + | R37-R55 | + | B26-B62 | + | B28-B64 |

[Equation 4]
Directional change H4 = 135 degrees with respect to the horizontal scanning direction =
2 × (| G23-G34 | + | G56-G67 |) + | R33-R55 | + | R35-R57 | + | B22-B66 | + | B24-B68 |

(Interpolate the pixel value of the AF pixel using the surrounding imaging pixel value according to the direction of the minimum variation value)
The AF pixel interpolation unit 50 selects the direction of the smallest direction variation among the direction variations H1 to H4 obtained in step (S-16), and uses the pixel value of the imaging pixel in that direction to perform AF. expression corresponding to the selected direction of the imaging pixel value _{B Y44} of B at the position of the imaging pixel value _{G X45} and AF pixel Y44 of G at the position of the pixel X45 equation 5] to [expression 8] (S-17). Thereby, by using the pixel value of the imaging pixel in the direction of small fluctuation, it becomes possible to more accurately interpolate the imaging pixel value at the position of the AF pixel such as X45 and Y44.

[Equation 5]
When the direction change H1 is the minimum, B _Y44 = (B24 + B64) / 2
G _X45 = (G34 + G36 + G54 + G56) / 4

[Equation 6]
When the direction change H2 is the minimum B _Y44 = (B24 + B64) / 2
G _X45 = (G25 + G65) / 2

[Equation 7]
When direction change H3 is the minimum, B _Y44 = (B26 + B62) / 2
G _X45 = (G36 + G54) / 2

[Equation 8]
When the direction change H4 is the minimum, B _Y44 = (B22 + B66) / 2
G _X45 = (G34 + G56) / 2

  The AF pixel interpolating unit 50 changes the direction variation H5 of the pixel value of the AF pixel in the horizontal scanning direction that is the arrangement direction of the AF pixels, for example, the pixel values W44 and W45 of the white light of Y44 and X45 of the AF pixel, and [ It is calculated using the equation described in [Equation 9].

[Equation 9]
H5 = | W44-W45 |

The AF pixel interpolation unit 50 determines whether or not the direction variation H5 exceeds the threshold th1 (S-18). Here, in the pixel interpolation process performed in the parameter estimation process, the threshold th1 is set to a very large value.
Therefore, since the pixel interpolation process performed in the parameter estimation process corresponds to the case where the direction variation H5 is equal to or less than the threshold th1 (NO side), the AF pixel interpolation unit 50 proceeds to (S-20).

The AF pixel interpolation unit 50 determines whether or not the direction variation H2 obtained in step (S-16) exceeds the threshold th2 (S-20). Here, in the pixel interpolation process performed in the parameter estimation process, the threshold th2 is set to an extremely large value.
Therefore, since the pixel interpolation process performed in the parameter estimation process corresponds to the case where the direction variation H2 is equal to or less than the threshold th2 (NO side), the AF pixel interpolation unit 50 proceeds to (S-21).

  The AF pixel interpolation unit 50 calculates the average pixel value <W44> and the like in AF pixels such as Y44 that are sensitive to the light flux from the right side, using the imaging pixel values of the surrounding color components R, G, and B. (S-21). Specifically, in step (S-16), for example, when the AF pixel interpolating unit 50 determines that the direction variation H2 is the minimum, the imaging pixel value of B is in the equation described in [Equation 5]. B24 and B64 are used. On the other hand, for the R and G pixel values, the R and G pixel values at the positions of the B imaging pixels B24 and B64 are interpolated using the four equations described in [Equation 10].

[Equation 10]
(1) R _B24 = (R13 + R15 + R33 + R35) / 4
(2) G _B24 = (G14 + G23 + G25 + G34) / 4
(3) R _B64 = (R53 + R55 + R73 + R75) / 4
(4) G _B64 = (G54 + G63 + G65 + G74) / 4

  The parameter estimation unit 19 refers to the parameter table [Table 1], selects a plurality of different parameters, temporarily stores the selected parameters (hereinafter “weighting coefficients”) in the work memory 13, and selects the selected weights. The coefficients are read from the work memory 13 as appropriate and sent to the AF pixel interpolation unit 50 (S-21). The read weighting coefficient is prepared for each color component of the pixel. The AF pixel interpolating unit 50 uses the read weighting coefficients, that is, the weighting coefficients WR, WG, and WB of R, G, and G, for the white light pixel values W24 and W64 at the positions of the imaging pixels B24 and B64. , [Equation 11].

[Equation 11]
W24 = WR × R _B24 + WG × G _B24 + WB × B24
W64 = WR × R _B64 + WG × G _B64 + WB × B64

Then, the AF pixel interpolation unit 50 calculates the average pixel value <W44> = (W24 + W64) / 2 of white light in Y44 (S-22).
The AF pixel interpolating unit 50 calculates the average pixel value <W45> of white light in the AF pixel such as X45 having sensitivity to the light beam from the left side, as in the case of step (S-22), and the adjacent color components. Calculation is performed using the R, G, and B imaging pixel values (S-23). In step (S-17), when the AF pixel interpolating unit 50 determines that the direction variation H2 is the minimum, the G imaging pixel value uses G25 and G65 in the equation described in [Equation 6]. . On the other hand, for the R and B pixel values, the R and B pixel values at the positions of the G imaging pixels G25 and G65 are interpolated using the four equations described in [Equation 12].

[Equation 12]
(1) R _G25 = (R15 + R35) / 2
(2) B _G25 = (B24 + B26) / 2
(3) R _G65 = (R55 + R75) / 2
(4) B _G65 = (B64 + G66) / 2

  Then, the AF pixel interpolation unit 50 calculates the pixel values W25 and W65 at the positions of the imaging pixels G25 and G65 by using the weighted sum of the formula described in [Equation 13].

[Equation 13]
W25 = WR × R _G25 + WG × G25 + WB × B _G25
W65 = WR × R _G64 + WG × G25 + WB × B _G65
Then, the image processing unit 30 calculates the average pixel value <W45> = (W25 + W65) / 2 of white light at X45 (S-23).

  The AF pixel interpolation unit 50 obtains the high-frequency component of the pixel value of white light in each AF pixel of the image sensor 2 using the average pixel value of white light obtained in (S-22) and (S-23) ( S-24). The AF pixel interpolation unit 50 first obtains an average pixel value of white light at the position of each AF pixel from each AF pixel value of the image sensor 14. That is, each AF pixel value is a value obtained by dividing the light flux from the left side or the right side into pupils. Therefore, in order to obtain each AF pixel value, it is necessary to add pixel values of light beams from the left side and the right side to each other. Therefore, the AF pixel interpolation unit 50 of the present embodiment uses the pixel value of the AF pixel and the pixel value of the adjacent AF pixel, for example, to calculate the average pixel value of white light at the position of the AF pixel Y44 or X45 [several 14].

[Formula 14]
<W44>'= W44 + (W43 + W45) / 2
<W45>'= W45 + (W44 + W46) / 2

  In [Expression 14] described in step (S-24), the pixel value of the white light at the position of each AF pixel is calculated using the pixel value of the AF pixel adjacent in the arrangement direction of the AF pixels. When there is a strong variation in the arrangement direction, the calculation of the high frequency component becomes inaccurate, and the resolution of the white light pixel values in the arrangement direction may be lost. Therefore, in step (S-18) described above, when there is a strong variation in the arrangement direction, the addition of the high frequency component is stopped.

  Thereafter, the AF pixel interpolation unit 50 obtains high-frequency components HFY44 and HFX45 of white light at the positions of Y44 and X45 from the equation described in [Equation 15].

[Equation 15]
HFY44 = <W44>'-<W44>
HFX45 = <W45>'-<W45>

  The AF pixel interpolation unit 50 determines whether or not the ratio of the high-frequency component HF of the pixel value of white light at the position of each AF pixel obtained in step (S-24) to the pixel value is smaller than the threshold th3. (S-25). Here, in the pixel interpolation process performed in the parameter estimation process, the threshold th3 is set to a very large value. Therefore, the pixel interpolation process performed in the parameter estimation process corresponds to the case where the high frequency component HF is equal to or greater than the threshold th3 (NO side), and the AF pixel interpolation unit 50 proceeds to step (S-26).

  The AF pixel interpolation unit 50 calculates the color fluctuations VR, VGr, VB, and VGb of the imaging pixel values of each color component R, G, or B in the vicinity of Y44 and X45 (S-26). Here, the color variations VGr and VGb indicate the G color variation at the position of the R or B imaging pixel. The AF pixel interpolation unit 50 obtains the color fluctuations VR and VGr based on the two equations described in [Equation 16].

[Equation 16]
(1) VR = | R33-R53 | + | R35-R55 | + | R37-R57 |
(2) VGr = | (G32 + G34) / 2− (G52 + G54) / 2 | + | (G34 + G36) / 2− (G54 + G56) / 2 | + | (G36 + G38) / 2− (G56 + G58) / 2 |

  The AF pixel interpolation unit 50 of the present embodiment calculates the value of VGr after obtaining the average value of the G pixel values at the R imaging pixel positions R33, R35, R37, R53, R55, and R57.

  On the other hand, the AF pixel interpolation unit 50 obtains the color fluctuations VB and VGb based on the two expressions described in [Equation 17].

[Equation 17]
(1) VB = | B22−B62 | + | B24−B64 | + | B26−B66 |
(2) VGb = | (G21 + G23) / 2− (G61 + G63) / 2 | + | (G23 + G25) / 2− (G63 + G65) / 2 | + | (G25 + G27) / 2− (G65 + G67) / 2 |

  Note that the AF pixel interpolation unit 50 of the present embodiment calculates the value of VGb after obtaining the average value of the G pixel values at the positions B22, B24, B26, B62, B64, and B66 of the B imaging pixels.

  The AF pixel interpolation unit 50 calculates the color variation rates KWG and KWB of the color components G and B using the color variations VR, VGr, VB, and VGb calculated in step (S-26) (S-27). . First, the AF pixel interpolation unit 50 obtains the color variations VR2, VG2, and VB2 from the three formulas described in [Equation 18] from the color variations VR, VGr, VB, and VGb.

[Equation 18]
(1) VR2 = (VR + α) × (VGb + α)
(2) VB2 = (VB + α) × (VGr + α)
(3) VG2 = (VGb + α) × (VGr + α)

Here, α is an appropriate constant for stabilizing the value of the color variation rate. For example, when processing a 12-bit image, α may be set to about 256.
Then, the image processing unit 30 calculates the color variation VW in white light using the color variations VR2, VG2, and VB2 according to the equation described in [Equation 19].

[Equation 19]
VW = VR2 + VG2 + VB2
Therefore, the AF pixel interpolation unit 50 calculates the color variation rates K _WG and K _WB from [Equation 20].

[Equation 20]
K _WB = VG2 / VW
K _WB = VB2 / VW

The AF pixel interpolation unit 50 uses the high frequency component HF of the pixel value of white light at the position of each AF pixel obtained in step (S-24), and the color variation rates K _WG and K _WB calculated in step (S-27). Are used to calculate the high-frequency components of the pixel values of the color components G and B at the position of each AF pixel from the equation described in [Equation 21] (S-28).

[Equation 21]
HFB _Y44 = HF _Y44 × K _WB
HFG _X45 = HF _X45 × K _WG

  The AF pixel interpolation unit 50 adds the high-frequency component of each color component in each AF pixel obtained in step (S-28) to the imaging pixel value obtained by interpolation in step (S-17) (S-29). ).

  Even if there is no change in the arrangement direction of the AF pixels, the high-frequency component of the pixel value has a slight error due to a shift between the weighted sum of the spectral characteristics of the imaging pixels of each color component and the spectral characteristics of the AF pixels. have. When there is no large fluctuation of the image in the vertical scanning direction (direction intersecting the AF pixel arrangement direction), the accuracy of the interpolation value is sufficient without adding a high-frequency component. There is a risk that a false structure resulting from Therefore, in step (S-20), in such a case, addition of a high frequency component is suppressed. In addition, when the calculated high frequency component is sufficiently small, the accuracy of the interpolation value is sufficient even if it is not added, and there is a possibility that a false structure due to an error may be caused by adding the high frequency component. For this reason, in (S-20), addition of a high frequency component is suppressed in such a case.

  For example, the CPU 11 calculates the imaging pixel values B ′ and G ′ of Y44 and X45 based on the formula described in [Equation 22], respectively.

[Equation 22]
B ′ _Y44 = B _Y44 + HFB _Y44
G ′ _X45 = G _X45 + HFG _X45

The AF pixel interpolating unit 50 _calculates pixel values such as B ′ _Y44 and G ′ _X45 obtained by interpolation at the positions of AF pixels such as Y44 and X45 based on the formulas described in [Equation 22] at the respective positions. The pixel value data for interpolation imaging as the pixel value for interpolation imaging in is stored in the work memory 13 in association with the first parameter. Similarly, pixel interpolation processing is performed for each of the remaining plurality of parameters, and pixel value data for interpolation imaging is derived for each parameter and stored in the work memory 13.

(Parameter estimation)
The parameter estimation unit 19 reads out the interpolation imaging pixel value data derived for each parameter from the work memory 13, and compares the interpolation imaging pixel value included in the interpolation imaging pixel value data with the peripheral imaging pixel value. . For example, the interpolated imaging pixel value B ′ _Y44 obtained by interpolation at the position of the AF pixel Y44 described in FIG. 8 is compared with the pixel values of the peripheral imaging pixel positions B24 and B64 with respect to the position Y44 of the AF pixel. . Here, the pixel values of the peripheral imaging pixel positions B24 and B64 are average values. The interpolated imaging pixel value G ′ _X45 obtained by interpolation at the position of the AF pixel _X45 and the peripheral imaging pixel value for the AF pixel position X45 (the average of the peripheral imaging pixel value G25 and the peripheral imaging pixel value G65) Value) and an error amount (difference) is obtained. The remaining interpolation imaging pixel values are also compared with the peripheral imaging pixel values to obtain an error amount. The parameter corresponding to the pixel value data for interpolation imaging having the smallest amount of error compared to each pixel value data for interpolation imaging is selected. Here, the error amount of the pixel value data for interpolation imaging may be the total error amount of the AF pixels or may be an average value. In the example shown in FIG. 9, pixel value data for interpolated images is derived for each parameter A, B, C..., And the smallest error amount (X, Z, Y... The parameter corresponding to the pixel value data for interpolation imaging including (difference) is estimated as the optimum parameter at the time of imaging. The estimated parameters are stored in the work memory 13.

[Pixel interpolation processing during actual shooting]
In the pixel interpolation processing at the time of actual photographing, the pixel interpolation processing is performed on the image data acquired at the time of actual photographing using the parameters estimated by the parameter estimating unit 19 or the parameters determined by the parameter determining unit 20 as described above. I do.
The difference is that in the determination at “S-18” described in FIG. 7, when the direction variation H5 exceeds the threshold th1 (YES side), the AF pixel interpolating unit 50 determines the B obtained in step (S-17). The interpolated value of _Y44 and G _X45 is used as the pixel value for imaging in Y44 and X45, and the image data is updated. The image processing unit 30 interpolates the updated image data into three color pixel values per pixel to generate three color image data, and records the three color image data in the SDRAM 32 via the bus 31. (S-19). Here, in the pixel interpolation processing performed at the time of actual photographing, for example, when processing a 12-bit image, the threshold value th1 may be set to a value of about 512.

Further, if the directional fluctuation H2 is greater than the threshold value th2 (YES side), AF pixel interpolation unit 50, the step (S-17) at the determined _{B Y44} and imaging pixel values at Y44 and X45 interpolated values of _{G X45} The image data is updated. The image processing unit 30 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 32 via the bus 31 (S-19). ). Here, in the pixel interpolation processing performed at the time of actual photographing, the threshold th2 may be set to a value of about 64 when processing a 12-bit image, for example.

Further, if the threshold th3 high-frequency component HF is smaller (YES side), the AF pixel interpolation unit 50, for imaging in Y44 and X45 interpolated values of _{B Y44} and _{G X45} calculated in step S-22 and S-23 Image data is updated using pixel values. The image processing unit 30 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 32 via the bus 31 (S-19). . In this embodiment, the threshold th3 is preferably about 10%, for example. The description of the threshold value th3 will be made together with the description of the weighting factors WR, WG and WB later.

The AF pixel interpolating unit 50 _calculates pixel values such as B ′ _Y44 and G ′ _X45 obtained by interpolation at the positions of AF pixels such as Y44 and X45 based on the formulas described in [Equation 22] at the respective positions. The image data acquired at the time of the actual photographing is updated as the pixel value for interpolation imaging at. The image processing unit 30 performs pixel interpolation or the like for the three colors on the updated image data, generates final image data for each of the three colors, and stores the image data in the SDRAM 32.

  FIG. 10 shows an example of an image structure in which the effect of the present embodiment is exhibited. This figure is a vertical section of an image structure of five vertical pixels including a convex structure (bright lines or dots), the horizontal axis is the vertical scanning direction (y coordinate), and the vertical axis is the light quantity or the pixel value. The convex structure is just on the AF pixel row arranged in the horizontal scanning direction.

  The circles shown in FIG. 10 are pixel values captured by the G imaging pixels. However, since there is no G imaging pixel at the position of the AF pixel, the G pixel value at that position cannot be obtained. Therefore, when there is a convex structure at the position of the AF pixel, the convex structure shown in the figure cannot be reproduced only from the pixel values of the G imaging pixels in the vicinity of the AF pixel. Actually, in the above-described step (S-17), the G pixel value (marked with ● in FIG. 10) obtained by interpolating at the position of the AF pixel using the pixel value of the G imaging pixel in the vicinity of the AF pixel is The convex structure is not reproduced.

  On the other hand, the pixel value of white light is obtained at the position of the AF pixel. However, normal pixels receive light that passes through the entire pupil region, whereas AF pixels receive only light that passes through the right or left side of the pupil, so adjacent AF pixels with different pupil divisions are added. Thus, the pixel value of the normal white light (of the light that has passed through the entire area of the pupil) is calculated ([Equation 14]).

  In addition, it is sufficient in many cases by interpolating and generating other color components R and B at the position of the G imaging pixel in the vicinity of the AF pixel to obtain a weighted sum of the color components R, G and B. The pixel value of white light can be obtained with high accuracy ([Equation 11] and [Equation 13]).

  The squares shown in FIG. 10 are distributions of pixel values of white light obtained in this way. In many cases, since the high-frequency component of the white light pixel value is proportional to the high-frequency component of the color component G pixel value, the high-frequency component calculated from the white light pixel value is the convex structure component of the G pixel value. Have information. Therefore, the high-frequency component of the G pixel value is obtained based on the high-frequency component of the white light pixel value, and the value is added to the data of the ● mark to obtain the G pixel value of the ☆ mark and reproduce the convex structure ([Equation 21]).

In each of the above embodiments, the arrangement direction of the AF pixels is the horizontal scanning direction. However, the present invention is not limited to this, and the AF pixels may be arranged in the vertical scanning direction or other directions.
In each of the above embodiments, each AF pixel is a focus detection pixel that divides the light beam from the left or right side into pupils, but the present invention is not limited to this, and each AF pixel has a light beam from the left and right sides. Focus detection pixels having pixels that divide the pupil may be used.

  In each of the embodiments described above, the AF pixel 46 described with reference to FIG. 2 is described as an example in which the primary color transmission filter 26 is not installed. However, the present invention is not limited to this. For example, a neutral density (ND) filter or a green (G) filter may be installed in the AF pixel 46. When a green filter is installed in the AF pixel 46, a detection signal corresponding to the luminance of the transmitted light that has passed through the green filter is output from the AF pixel, and the signal may be used as the pixel value.

  In each of the above-described embodiments, the peripheral imaging pixel columns are described as the peripheral imaging pixel columns 52 and 53 with respect to the AF pixel column 47 described in FIG. As shown in FIG. 5, the peripheral imaging pixel rows 54 and 55 which are G / B pixel rows far away from the AF pixel row 47 in the vertical scanning direction may be used. Further, the AF pixel column is not limited to the column corresponding to the G / B pixel column, and may be provided in the column corresponding to the G / R pixel column.

  Furthermore, in each said embodiment, although this invention is demonstrated as the electronic camera 10, it is good also as imaging devices, such as a mobile telephone with a camera, and a smart phone. The storage means of the present invention may be a card memory exchangeable from the outside, instead of the nonvolatile memory 12.

DESCRIPTION OF SYMBOLS 10 Electronic camera 14 Image pick-up element 19 Parameter estimation part 20 Parameter determination part 50 AF pixel interpolation part

Claims (7)

An image sensor that has a plurality of imaging pixels and a plurality of focus detection pixels and images a subject image formed by a photographic lens;
AF pixel position storage means for storing the position of each focus detection pixel;
The focus detection pixel value obtained from the position of the focus detection pixel, the peripheral imaging pixel value obtained from the surrounding imaging pixel position, and the lens condition for the image data captured by the imaging element. A pixel interpolation means for interpolating into an interpolation imaging pixel value corresponding to the imaging pixel value using a parameter determined based on a lens condition acquired at the time of shooting from among a predetermined parameter group;
In an imaging apparatus comprising:
If the lens condition cannot be acquired at the time of shooting, the pixel interpolation means uses the focus detection pixel value included in the reference image data obtained by capturing an image of a subject having a uniform luminance distribution by the image sensor as the parameter group. And interpolating to the interpolation imaging pixel value for each of a plurality of parameters selected from,
The interpolated imaging pixel value interpolated for each parameter output from the pixel interpolating means is compared with the peripheral imaging pixel value included in the reference image data, and based on the comparison result, the parameter value is selected from the parameter group. An imaging apparatus comprising: parameter estimation means for estimating an optimum parameter. The imaging device according to claim 1,
The parameter estimation means sets a parameter corresponding to the interpolation imaging pixel value data having the smallest difference between the interpolation imaging pixel value and the surrounding imaging pixel value among the interpolation imaging pixel value data interpolated for each parameter. An imaging apparatus characterized by estimating. The imaging device according to claim 1 or 2,
Mode selection means for selecting an adjustment mode to be used when the lens conditions cannot be obtained;
An instruction means for outputting a message to the outside to image a subject having a uniform luminance distribution when the adjustment mode is selected by the mode selection means;
Control means for operating the pixel interpolation means and the parameter estimation means after outputting the instructions by the instruction means;
An imaging apparatus comprising: The imaging device according to any one of claims 1 to 3,
An imaging apparatus comprising storage means for storing parameters estimated by the parameter estimation means. In the imaging device according to any one of claims 1 to 4,
The imaging apparatus, wherein the parameter is a weighting coefficient determined by an exit pupil distance and an aperture value. The imaging apparatus according to claim 5,
Storage means for storing a table in which the weighting factor is associated with an exit pupil distance and an aperture value;
A selection unit that selects a plurality of weighting factors used by the pixel interpolation unit when lens conditions cannot be acquired from the weighting factor group described in the table;
An imaging apparatus comprising: The imaging device according to any one of claims 1 to 6,
The image pickup apparatus, wherein the parameter is determined for each color component of a pixel.

Images