JP5163733B2 - Imaging apparatus and focus determination program - Google Patents

Description

  The present invention relates to an imaging apparatus and a focus determination program.

  The following autofocus devices are known. This autofocus device detects the focal position shift of a lens using chromatic aberration of light and corrects the blur of each color light by performing focus adjustment based on the detection result (for example, Patent Document 1).

JP-A-6-138362

  However, the conventional autofocus device cannot detect in which direction of the optical axis the position of the AF lens is deviated from the in-focus position, and the accuracy of determining the in-focus state is low.

An image pickup apparatus according to the present invention includes an image input unit that inputs a subject image formed by an optical system having axial chromatic aberration as an input image, and edge detection that detects an edge for each color component from the input image input by the image input unit. And a correction for correcting a change in resolution between the color components caused by the edge direction during pixel addition reading from the image sensor or image processing based on the edge of each color component detected by the edge detection unit And a difference in blur width between the edges of the two color components having axial chromatic aberration among the edges of the color components detected from the image after the resolution change between the color components is corrected by the correction means. And a focusing state determining unit that determines a focusing state based on a difference in blur width calculated by the calculating unit.
The focus determination program according to the present invention includes an image input procedure for inputting a subject image formed by an optical system having axial chromatic aberration as an input image, and an edge for detecting an edge for each color component from the input image input in the image input procedure. Based on the detection procedure and the edge of each color component detected in the edge detection procedure, correction for correcting the change in resolution between the color components caused by the edge direction during pixel addition reading from the image sensor or image processing The difference between the blur widths of the edges of two color components having axial chromatic aberration among the edges of the color components detected from the procedure and the image after the resolution change between the color components is corrected by the correction procedure is calculated. The computer is caused to execute a calculation procedure and an in-focus state determination procedure for determining an in-focus state based on a difference in blur width calculated in the calculation procedure.

  According to the present invention, the in-focus state can be determined with high accuracy.

1 is a block diagram illustrating a configuration of an embodiment of a camera 100. FIG. It is a flowchart figure which shows the flow of the focus position detection process in 1st Embodiment. It is a figure which shows the example of approximation to a model edge. It is a figure which shows the example of a correction | amendment of the background color of an edge area | region. It is a figure which shows the example of determination of a focusing state. It is a 1st figure which shows the example of calculation of the color difference sum between two color edges. It is a 2nd figure which shows the example of calculation of the color difference sum between two color edges. It is a figure which shows the shift | offset | difference of the sample position at the time of 2x2 pixel addition. It is a flowchart figure which shows the flow of the focus position detection process in 2nd Embodiment. It is a figure which shows a Bayer arrangement | sequence. It is an example of a filter for detecting the same color pixel gradient from the Bayer array. It is a figure which shows the reference pixel range with respect to the original image at the time of 2x2 pixel addition. It is a figure which shows the use pixel with respect to the edge of the horizontal direction on a Bayer arrangement | sequence. It is a figure which shows the use pixel with respect to the edge of the perpendicular direction on a Bayer arrangement | sequence. It is a figure which shows the use pixel with respect to the edge of the diagonal direction on a Bayer arrangement | sequence. It is a flowchart figure which shows the flow of the focus position detection process in 3rd Embodiment. It is a figure which shows the chart image image | photographed for correction | amendment process determination. It is a figure which shows the change of the reference value shift | offset | difference at the time of focusing by edge direction. It is a figure which shows the smoothing filter with the directionality for correction | amendment. It is a figure which shows the change of the reference value shift | offset | difference at the time of focusing by edge direction.

-First embodiment-
FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the first embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a zoom button, a cross key, an enter button, a play button, and a delete button.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The lenses constituting the lens 102 include an AF lens (focus adjustment lens) for AF (Auto Focus / automatic focus adjustment) described later. The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and captures a subject image formed by the lens 102. Then, an image signal obtained by imaging is output to the control device 104.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  The control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. The memory card slot 105 reads an image file stored in the memory card based on an instruction from the control device 104.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, when the user sets the mode of the camera 100 to the shooting mode, the control device 104 outputs image data for display of images acquired from the image sensor 103 in time series to the monitor 106. As a result, a through image is displayed on the monitor 106.

  In the present embodiment, when input of a through image from the image sensor 103 is started, the control device 104 executes AF (Auto Focus / Automatic Focus Adjustment) processing, and continuously performs focus adjustment during the through image display. By performing this, the AF control is always performed during the through image display. Specifically, when the frame input from the image sensor 103 is an image expressed in the RGB color system, focusing on the optical axis of each color component due to the difference in axial chromatic aberration for each color component of RGB. Focusing on the fact that the magnitude relation of the blur width of each color component in the edge region is reversed before and after the in-focus position on the optical axis because the position is different, the current position based on the difference in the blur width of the edge between the two colors Determine the in-focus state. Then, it is determined in which direction of the optical axis direction the current AF lens position is shifted from the in-focus position, thereby realizing continuous AF control during live view display. A specific example in which the magnitude relationship of the blur width of each color component in the edge region is reversed before and after the in-focus position on the optical axis will be described later with reference to FIG.

  Here, in the case of the optical system (lens 102) and the image sensor 103 used in a general digital camera, the blur width of each color component described above has a spread of several pixels to several tens of pixels. The difference in blur width between the two colors is only about 1/10 to 1/100, and this difference in blur width is small with respect to the measurement error of the blur width that can be performed only in pixel units. It is difficult to accurately determine the difference in blur width from the measurement results. For this reason, in the present embodiment, as described below, the control device 104 indirectly calculates a difference in blur width between two colors based on a set classification of pixel values and a statistic for each set. The in-focus state is determined by determining in which direction of the optical axis the current AF lens position is shifted from the in-focus position.

  FIG. 2 is a flowchart showing a flow of a focus position detection process for constantly performing AF control during live view display in the first embodiment. The processing illustrated in FIG. 2 is executed by the control device 104 as a program that is activated when input of a through image from the image sensor 103 is started. In the present embodiment, there is axial chromatic aberration between R and G, and the focus is shifted toward the image sensor 103 from the in-focus position on the optical axis, that is, in the case of the rear pin, R Assuming that G is more blurred than G, and conversely, the lens 102 is out of focus with respect to the in-focus position on the optical axis, that is, G is always more blurred than R in the case of the front pin. Processing will be described.

  In step S <b> 1, the control device 104 reads an image in the evaluation region (evaluation region image) from the image input from the image sensor 103. Here, the evaluation area image means, for example, an image in the AF evaluation area arranged in the shooting screen. Thereafter, the process proceeds to step S <b> 2, and the control device 104 performs edge detection on the evaluation area image and inspects the presence or absence of an edge that can be used for determining the in-focus state in the evaluation area image. Thereafter, the process proceeds to step S3, and the control device 104 determines whether or not an edge is detected from the evaluation area image based on the edge detection result in step S3. If a negative determination is made in step S3, the process proceeds to step S14, where the control device 104 determines that determination of the in-focus state is impossible and ends the process. On the other hand, if a positive determination is made in step S3, the process proceeds to step S4.

In step S4, the control device 104 separates the evaluation region image into a flat region (flat portion) F and an edge region (edge portion) E based on the presence or absence of a gradient for each pixel in the evaluation region image. Specifically, the control device 104 represents each pixel of the evaluation region image as p (x, y), and represents each pixel with R, G, and B values of p (x, y) as R (x, y). ), G (x, y), and B (x, y), the gradient amount absolute values gR (x, y) and gG (x, y) are expressed by the following equations (1) and (2). Is calculated. As described above, in this embodiment, since it is assumed that there is axial chromatic aberration between R and G, the gradient absolute value gR (x, y) and gG (x, y) are calculated. If there is axial chromatic aberration between other color components, the absolute value of the gradient amount may be calculated for that color component.

And the control apparatus 104 has the gradient which exceeds a threshold value using following Formula (3) and (4) based on the calculated gradient amount absolute value gR (x, y) and gG (x, y). Whether or not the evaluation area image is separated into the flat area F and the edge area E.

Further, the control device 104 separates the flat area F into sub-flat areas FH and FL using the average value Fmean of the pixels belonging to the flat area F as a threshold, as shown in the following equations (5) and (6). To do. Thereby, an edge having an arbitrary cross-sectional shape in the evaluation area image can be approximated by a model edge represented by two sub-flat areas FH and FL and an edge area E connecting the two sub-flat areas. .

  For example, an edge 3a having a cross-sectional shape as shown in FIG. 3A has two subflat regions FL and FH in which the contrast difference between both sides of the edge is C, as shown in FIG. It is approximated by the model edge 3b separated into the edge region E connecting the regions FL and FH. Note that, in the model edge 3b shown in FIG. 3B, the length w of the edge region E indicates the blur width of the edge. A model edge per pixel unit length in the direction along the edge is called a conversion edge.

  Thereafter, the process proceeds to step S5, and the control device 104 examines the average colors of the sub-flat regions FH and FL to determine whether the background color is a color in which axial chromatic aberration is visible. That is, when there is axial chromatic aberration between R and G and between R and B and there is no axial chromatic aberration between G and B, the control device 104 determines that the sub-flat regions FH and FL Even if there is a contrast difference between only G and B in the background color, no axial chromatic aberration is observed unless there is a contrast difference in R. Therefore, in this case, if the background color has a sufficient contrast difference in R and there is a sufficient contrast difference in at least one of G and B, an affirmative determination is made in step S5. A negative determination is made in S5. If a negative determination is made in step S5, the process proceeds to step S14 described above, and the control device 104 determines that determination of the in-focus state is impossible and ends the process. On the other hand, if an affirmative determination is made in step S5, the process proceeds to step S6.

  In step S6, control device 104 corrects the entire evaluation area image so that the colors of sub-flat areas FH and FL are gray. Specifically, the control device 104 normalizes contrast differences on both sides of the R, G, and B edges in the entire evaluation area image so that the colors of the sub-flat areas FH and FL are gray. For example, in FIG. 4A, the R component edge R (x) indicated by the solid line and the G component edge G (x) indicated by the dotted line are contrasts on both sides of the edge region E as shown in FIG. The background color is corrected so that the difference is the same. Thereafter, the process proceeds to step S7.

In step S7, the control device 104 uses the edge region E in the evaluation region image after normalizing the contrast difference between both edges of each color component in step S6, and 2 having axial chromatic aberration among R, G, and B. For the inter-color component, the index value I is calculated using the following equations (7) to (11). That is, the control device 104 first sets the average value Emean of the pixels belonging to the edge region E as a threshold, as shown in equations (7) and (8), for the evaluation region image after the background color correction is performed. The edge region E is separated into sub-edge regions EL and EH.

Next, the control device 104 calculates the color difference sum ΔAL of the sub-edge region EL using the following equation (9), and calculates the color difference sum ΔAH of the sub-edge region EH using the following equation (10). As described above, in the present embodiment, since it is assumed that there is axial chromatic aberration between R and G, in the following equations (9) and (10), the edge of the R component and the G component The color difference sums ΔAL and ΔAH are calculated for the edges of the color component, but if there is axial chromatic aberration between the other color components, the color difference sums ΔAL and ΔAH are calculated for the edges of the color components. Good.

Further, the control device 104 calculates an index value I between two colors having axial chromatic aberration, that is, between R and G, by the following equation (11). The index value I is an amount indicating how many edges indicating the converted edge blur width difference Δw exist in the evaluation region image by comparison with the equation (23) described later. Note that the evaluation value I is calculated by the following formula (11) without assuming any edge shape in the evaluation region image, and only by classification into a hierarchical pixel set based on statistics, so The evaluation value I can also be calculated for the region image by the equation (11).

  Thereafter, the process proceeds to step S8, and the control device 104 approximates the number N of converted edges (number of converted edges) N included in the evaluation area image. For example, the control device 104 can approximate the converted edge number N by binarizing the evaluation region image with the above-mentioned Emeans and counting pixels serving as boundaries. Then, it progresses to step S9.

In step S9, the control device 104 calculates a converted edge blur width difference Δw that is used to determine in which direction of the optical axis direction the current AF lens position is shifted from the in-focus position. Since the relationship of I = Δw × N is established between the evaluation value I calculated by the equation (11), the converted edge number N, and the converted edge blur width difference Δw, the control device 104 represents the following equation (12) To calculate the converted edge blur width difference Δw.
Δw = I / N (12)
Thereby, the converted edge blur width difference Δw can be indirectly calculated from the evaluation area image including the edge of an arbitrary shape. Note that the converted edge blur width difference Δw calculated here is a converted edge blur width difference between the R component and G component having axial chromatic aberration, and thus Δw can also be expressed as ΔwR−G.

  Thereafter, the process proceeds to step S10, and the control device 104 sets a threshold value used for determining the moving direction of the AF lens based on the converted edge blur width difference Δw. Note that as the AF lens is closer to the in-focus position, the calculated Δw becomes closer to 0. For this reason, it is possible to determine the in-focus position when Δw is 0, and in other cases, it is possible to determine the direction in which Δw is 0 as the movement direction of the AF lens. However, in this embodiment, in order to avoid erroneous determination before and after focusing due to a measurement error, in order to determine that the AF lens position is in the in-focus position if Δw is within a certain range centered on 0, In S10, a predetermined threshold is set. Note that the threshold may be a predetermined constant, or may be set as appropriate based on an image, shooting conditions, and the like.

  Thereafter, the process proceeds to step S11, and the control device 104 compares the converted edge blur width difference Δw calculated in step S9 with a threshold value to determine whether the current focus adjustment state is the front pin or the rear pin. That is, when Δw is between −0.5 and 0.5, when the threshold is set so that the in-focus state is determined, the controller 104 determines that Δw is greater than 0.5. It is determined that it is a rear pin, and when Δw is smaller than −0.5, it is determined that it is a front pin.

  For example, considering the normalized R-component edge R (x) and G-component edge G (x) shown in FIG. 4, in the case of the rear pin, R is more blurred than G as described above. Therefore, as shown in FIG. 5A, a relationship of wR> wG is established between the length wR of the R component edge region E and the length wG of the G component edge region E. For this reason, when Δw (= wR−wG) is larger than 0.5 which is the upper limit of the threshold value, the control device 104 determines that it is a rear pin.

  On the other hand, in the case of the front pin, since G is more blurred than R as described above, the length wR of the R component edge region E and the length of the G component edge region E as shown in FIG. A relationship of wR <wG is established with wG. Therefore, the control device 104 determines that the pin is the front pin when Δw (= wR−wG) is smaller than −0.5 which is the lower limit of the threshold.

  On the other hand, as shown in FIG. 5B, in principle, between the length wR of the R component edge region E and the length wG of the G component edge region E, as shown in FIG. A relationship is established. For this reason, the control device 104 is in a focused state when Δw (= wR−wG) is −0.5 or more which is the lower limit of the threshold value and 0.5 or less which is the upper limit of the threshold value. It is determined that the front pin or the rear pin is unknown.

  Thereafter, the process proceeds to step S12, and the control device 104 determines whether or not it is determined as a front pin or a rear pin as a result of the determination in step S11. If a negative determination is made in step S12, the process proceeds to step S14 described above, and the control device 104 determines that determination of the in-focus state is impossible and ends the process. On the other hand, when an affirmative determination is made in step S12, the process proceeds to step S13.

  In step S13, the control device 104 determines a direction in which Δw can approach 0 as the movement direction of the AF lens based on the determination result in step S11. Then, the control device 104 outputs the determined movement direction of the AF lens to a motor for driving the AF lens, thereby moving the AF lens in the focusing direction. Thereafter, the process ends.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 104 detects an edge for each color component from the evaluation area image, calculates a converted edge blur width difference Δw of the edges of two color components having axial chromatic aberration, and calculates the converted edge blur width difference Δw. Based on this, the focused state is determined. Thereby, the in-focus state can be determined with high accuracy.

(2) The control device 104 classifies the edges of the two color components having axial chromatic aberration into the flat region F and the edge region E, and calculates the evaluation value I based on the color difference sum calculated for the edge region E. I tried to do it. And the control apparatus 104 was made to calculate conversion edge blur width difference (DELTA) w by Formula (12) using the calculated evaluation value I. FIG. Thereby, even when the shape of the edge is unknown, the converted edge blur width difference Δw can be calculated.

(3) The control device 104 performs correction so as to eliminate the contrast difference between both sides of the edge of the two color components having axial chromatic aberration, and calculates the converted edge blur width difference Δw using the corrected evaluation area image. I made it. As a result, the calculation accuracy of the converted edge blur width difference Δw can be increased.

-Second embodiment-
In the first embodiment described above, the converted edge blur width difference Δw between the edges of two color components having axial chromatic aberration is calculated, and the in-focus state is determined based on the converted edge blur width difference Δw. . In this case, apart from the blur amount of each color component due to axial chromatic aberration, the blur width measured by changing the resolution between the color components due to the process of the process of generating the input image and the blur between the color components. The width difference may change.

  In such a situation, for example, in an imaging apparatus that generates an image by interpolating a RAW image read from a single-plate color image sensor, pixel addition is performed on the RAW image read by performing pixel addition readout from the image sensor. Occurs when interpolating with no interpolator. Specifically, when a focusing operation by contrast AF is performed on the output image of the same image sensor as the captured image, pixel addition reading is performed to reduce the image size for the purpose of improving the frame rate. On the other hand, if only an interpolator for captured images is provided for cost reasons, an image that has been interpolated by an interpolator that is not necessarily optimized for interpolating the pixel-added image used for the focusing operation is generated. . In this case, if the pixel arrangement of the single-plate color image sensor does not have a uniform distribution due to the color components with respect to pixel addition, each color component depends on the vertical, horizontal, and diagonal directions with respect to the pixels of the generated image. There may be a resolution change between.

  For example, in a four-color Bayer array of RGGB in which an image sensor is most often used, when pixel addition reading with an average of 2 × 2 pixels of the same color is performed, a reduced Bayer array RAW having an array similar to the original Bayer array RAW image by pixel addition reading Although an image is generated, the sample position gravity center in the optical image on the image sensor and the pixel position on the image data are shifted as shown in FIG.

  That is, in the original Bayer array RAW image shown in FIG. 8A, in the reduced Bayer array RAW image obtained by performing pixel addition reading of 2 × 2 pixels average of the same color as shown in FIG. There is a deviation between the sample position centroid on the original Bayer array RAW image, that is, the centroid position 8a of the four R pixels and the pixel position 8b of the R ′ pixel in the reduced Bayer array RAW image. Similarly, there is a difference between the sample position centroid on the original Bayer array RAW image of Gr pixels, that is, the centroid position 8c of four Gr pixels and the pixel position 8d of Gr ′ pixels in the reduced Bayer array RAW image. There is a deviation between the sample position centroid on the original Bayer array RAW image of Gb pixels, that is, the centroid position 8e of the four Gb pixels and the pixel position 8f of the Gb ′ pixel in the reduced Bayer array RAW image. There is a deviation between the sample position centroid on the original Bayer array RAW image of B pixels, that is, the centroid position 8g of the four B pixels and the pixel position 8h of the B ′ pixel in the reduced Bayer array RAW image.

  When the interpolation device that does not assume the sample position shift depending on the color of the reduced Bayer array RAW image performs interpolation by the same processing as the original Bayer array RAW image, false colors and color components depending on the direction with respect to the Bayer array An image in which a change in resolution occurs is generated. When focus position determination is performed on such an image using the edge blur width difference between the two colors caused by the axial chromatic aberration described in the first embodiment, it is measured depending on the direction of the edge included in the image. The edge blur width difference between the two colors changes. If such a change exists, it is necessary to make a condition-based determination based on the dependency of the change when the focus position is determined by comparison with the reference value in the in-focus state, and the determination accuracy deteriorates. In addition, when performing in-focus state determination due to longitudinal chromatic aberration while tracking a moving subject or shooting a moving image, the value used as the determination reference changes when the edge position rotates during the determination operation, so a stable determination result is obtained. It becomes difficult to put out.

  In order to solve such a problem, in the second embodiment, when such an interpolation process is performed, a RAW image is used for a situation in which the input image includes a change in resolution between color components other than axial chromatic aberration. Alternatively, a method for performing a highly accurate focal position determination based on axial chromatic aberration by selecting and using only combinations of pixels that are not affected by a change in resolution between color components from an interpolated image will be described.

  In the second embodiment, the flow of processing will be described by taking as an example a case where a change in resolution between color components depending on the edge direction occurs due to pixel addition reading and interpolation. FIG. 9 is a flowchart showing a flow of focus position detection processing in the second embodiment.

  In step S1, the control device 104 performs a reduced Bayer array RAW image obtained by performing the above-described 2 × 2 pixel addition from the Bayer array image sensor including four types of color filters R, Gr, Gb, and B and reading the image. get. As shown in FIG. 8B, in this reduced Bayer array RAW image, the pixel position on the image data and the sample position centroid of each color component with respect to the original image are shifted. For this reason, when the interpolation device does not assume pixel addition reading, an image in which the resolution between the color components is changed depending on the edge direction is generated. Therefore, in order to avoid this adverse effect, the process proceeds to step S2 without performing interpolation.

  In step S2, the control device 104 detects the edge directions of the four color components R, Gr, Gb, and B from the reduced Bayer array RAW image. For example, with respect to a reduced Bayer array RAW image as shown in FIG. 10, the same color pixel gradient is obtained by using four types of filters as shown in FIGS. 11 (a), (b), (c) and (d). It is calculated for each image, and the whole image is taken and compared. Then, from the sign of the edge gradient and the direction having the maximum absolute value, one of the eight directions can be detected as the edge direction for each color component.

  Thereafter, the process proceeds to step S3, and the control device 104 determines whether or not an edge exists in the reduced Bayer array RAW image based on the detection result of the edge direction in step S2. For example, if the gradient is small and no edge is detected in step S2, it is determined that no edge exists in the reduced Bayer array RAW image. If a negative determination is made in step S3, it is determined that the reduced Bayer array RAW image acquired in step S1 is an image that is inappropriate for focus position determination due to axial chromatic aberration, and the process ends. On the other hand, if a positive determination is made in step S3, the process proceeds to step S4.

  In step S4, the control device 104 determines whether or not the edge directions detected for each color component from the reduced Bayer array RAW image coincide with each other based on the detection result of the edge direction in step S2. When a negative determination is made in step S4, it is determined that the reduced Bayer array RAW image acquired in step S1 is an image that is inappropriate for the focal position determination due to the longitudinal chromatic aberration, and the process is ended as it is. On the other hand, if an affirmative determination is made in step S4, the reduced Bayer array RAW image acquired in step S1 is determined to be an image capable of determining the focal position based on axial chromatic aberration, and the process proceeds to step S5.

  In step S5, the control device 104 selects pixels having the same resolution and sample position between the respective color components in accordance with the edge direction detected in step S2. Here, the focal position determination based on axial chromatic aberration uses the relationship shown in FIG. 5 to determine the difference in blur width between two colors of R and G, G and B, and R and B at the time of focusing. The result of the difference between the three sets of two-color blur widths is calculated and weighted for each to obtain a final determination result.

  First, when interpolating a Bayer array RAW image composed of four color filters of R, Gr, Gb, and B, a G plane image is generated using both Gr and Gb. In this embodiment, Gr and Gb is distinguished and handled. When pixel addition as shown in FIG. 8 is performed, as shown in FIG. 12A, the reference range of the R ′ pixel of the reduced Bayer array RAW image to the R pixel of the original Bayer array RAW image is 3 × 3. In contrast to the pixel, as shown in FIG. 12B, the reference range for the Gr + Gb pixel of the original Bayer array RAW image of the Gr ′ + Gb ′ pixel of the reduced Bayer array RAW image is 4 × 4 pixels. This is because if the Gr ′ + Gb ′ pixel is used as a G pixel, the resolution of G with respect to R is slightly deteriorated, and there is an adverse effect when a minute difference in blur width due to axial chromatic aberration is measured.

  On the other hand, if Gr ′ and Gb ′ are distinguished from each other, as shown in FIG. 12C, for example, the reference range of the Gr ′ pixel of the reduced Bayer array RAW image to the original RAW image Gr pixel is 3 × 3 pixels. It becomes equal to the reference range of the pixel, and the resolution can be matched.

  Next, the control device 104 selects a combination of pixels to be used for each combination of two colors for performing focal position determination based on axial chromatic aberration according to the edge direction. In this embodiment, the following three cases (a) to (c) are used.

(A) In the case of a horizontal edge as shown in FIG. 13A In this case, as a combination of R and G, R and Gr pixels are selected as shown in FIG. As a combination, a Gb pixel and a B pixel are selected as shown in FIG. By selecting in this way, as shown in FIGS. 13B and 13C, the same resolution and sample position are obtained in the respective combinations of R and G and G and B in the direction crossing the edge with respect to the horizontal edge. be able to. In this case, the resolution matches the combination of R and B, but the sample position is shifted. Therefore, the result obtained with R and B is not used, or the result obtained with R and B is used with a lower weight than the result obtained with R and G, and G and B.

(B) Case of vertical edge as shown in FIG. 14A In this case, as a combination of R and G, R and Gb pixels are selected as shown in FIG. As a combination, a Gr pixel and a B pixel are selected as shown in FIG. By selecting in this way, as shown in FIGS. 14B and 14C, the same resolution and sample position are obtained in the respective combinations of R and G and G and B in the direction across the edge with respect to the vertical edge. be able to. Also in this case, as in the case of (a) described above, for the combination of R and B, the resolution is the same, but the sample position is shifted. Therefore, the result obtained with R and B is not used, or the result obtained with R and B is used with a lower weight than the result obtained with R and G, and G and B.

(C) In the case of an oblique edge as shown in FIG. 15 (a) In this case, as shown in FIG. 15 (b), R and B are set to the same resolution and sample position by a combination of R and B pixels. I can do it. In this case, the sample position is shifted with respect to the combination of R and G and G and B, but the resolution can be matched by using only one of the Gr pixel and the Gb pixel. Therefore, the results obtained from R and G and G and B are not used, or the results obtained from R and G, G and B are used with a lower weight than the results obtained from R and B.

  Thereafter, the process proceeds to step S6, where the control device 104 performs a focal position determination based on axial chromatic aberration from the pixel group selected in step S5, and ends the process. Specifically, the control device 104 executes the processes of steps S4 to S14 of FIG. 2 in the first embodiment for the pixel group selected in step S5. That is, in the first embodiment, an example in which axial chromatic aberration is present between R and G has been described. However, instead of R and G, a set of pixels selected in step S5 is a target. Steps S4 to S14 of step 2 are executed.

According to the second embodiment described above, the following effects can be obtained.
The control device 104 selects pixel combinations for matching different resolutions due to factors other than axial chromatic aberration, according to the direction of the edge included in the input image. As a result, the reference value in the focused state of the edge blur width difference between the two colors caused by the longitudinal chromatic aberration can be aligned regardless of the edge condition included in the input image, and the magnitude of the reference value in the focused state can be increased or decreased. It is possible to improve the accuracy of focal position detection performed in comparison. In addition, when performing focusing operations in subject tracking or movie shooting, the subject is in focus even if the subject moves within the input image and the edge direction rotates, or the illumination changes and the subject color changes. Since the reference value does not change at this time, it is possible to perform focus adjustment by performing focus state detection using stable axial chromatic aberration. Furthermore, by selecting and limiting the pixels to be used, the amount of pixel data to be handled is suppressed, and no interpolation processing is required, so that the calculation cost is reduced and the focus position detection processing is speeded up. I can expect.

-Third embodiment-
In the third embodiment, even when the input image includes a change in resolution between color components other than axial chromatic aberration, a simple correction process is performed on the input image, thereby providing a highly accurate on-axis. A method for performing focal position determination based on chromatic aberration will be described. In the present embodiment, a correction process for canceling the resolution change between each color component generated in the image used for the determination is obtained in advance, and the input image is corrected using this correction process before performing the focus position determination. Determine the focal position.

  In the third embodiment, the flow of processing will be described by taking as an example the case of pixel addition reading and interpolation in which a resolution change between each color component occurs depending on the edge direction. FIG. 16 is a flowchart showing the flow of the focus position detection process in the third embodiment.

  In step S <b> 1, the control device 104 reads out an image by performing pixel addition from the Bayer array image sensor. Thereafter, the process proceeds to step S2, and the control device 104 performs an interpolation process on the image read out in step S1. Here, when the interpolation device does not assume pixel addition reading, an image is generated in which the resolution between the color components changes depending on the edge direction.

  Thereafter, the process proceeds to step S3, and the control device 104 determines whether or not the input image has edge direction dependency. In this embodiment, the edge of the input image is obtained by combining the same image sensor, pixel addition read drive, interpolation device, image processing, and the like as the image input to the focal position detection process based on the longitudinal chromatic aberration targeted by the present invention. Investigate the direction dependency beforehand. For example, an image in which a black and white checkered pattern as shown in FIGS. 17A and 17B is placed vertically and horizontally and obliquely with respect to the pixel array is photographed in a focused state by contrast AF or the like, and the edge of the input image The direction dependency is investigated, and the determination in step S3 is performed based on the investigation result.

  If a negative determination is made in step S3, processing for correcting the edge direction dependency of the input image is not necessary, and the process proceeds to step S5 described later. On the other hand, if an affirmative determination is made in step S3, it is determined that processing for correcting the edge direction dependency of the input image is necessary, and the process proceeds to step S4. In step S4, the control device 104 executes a process for correcting the edge direction dependency of the resolution of the input image on the image subjected to the interpolation process in step S2. Specifically, processing is performed as follows.

  First, the control device 104 calculates the edge blur width difference Δω between two colors for each edge direction by the same method as that used in the focal position detection process based on longitudinal chromatic aberration. Here, a case where the in-focus state is determined from the longitudinal chromatic aberration generated between the R and G color components will be described. The control device 104 calculates the edge blur width difference between the R and G color components from the chart photographed image in the focused state, and sets this as ΔωAF (RG). Here, if the blur width of R in the in-focus state is ωAF_R and the blur width of G is ωAF_G, there is a relationship of ΔωAF (RG) = ωAF_R−ωAF_G.

  FIG. 18 shows the result of plotting the change of ΔωAF (RG) due to the edge direction. FIG. 18 shows an example in which the calculation result of ΔωAF (RG) is plotted with respect to edges in eight directions that are slanted vertically and horizontally. In FIG. 18, it can be seen that ΔωAF (RG) changes in the plus direction with 0.0 as a reference when the edge direction is around 45 degrees and 225 degrees with respect to the image sensor array. Both are edges extending from the upper right to the lower left on the image, and ΔωAF (RG) changes in the positive direction at the edge in this direction. It can be interpreted that the resolution of ωAF_R has decreased and ωAF_R has increased. In order to correct this, for example, the 3 × 3 smoothing filter shown in FIG. 19A is applied only to the G color component, thereby reducing the resolution of G in the direction from the upper left to the lower right. Then, smoothing may be performed so as to match the resolution of R. Thus, the change in ΔωAF (RG) due to the edge direction is corrected as shown in FIG. As a result, the variation of ΔωAF (RG) due to the edge direction is eliminated. Therefore, in step S5, ΔωAF (RG) = 0.0 is set as the reference value for the edge in any direction, and depending on the magnitude thereof. The focal position detection process can be performed.

  Another interpretation is possible from FIG. That is, even if the edge direction is around 135 ° and 315 ° (= −45 °) with respect to the image sensor array, ΔωAF (RG) changes in the negative direction with 0.3 as a reference. good. In this case, both are edges extending from the upper left to the lower right on the image, and that ΔωAF (RG) changes in the negative direction at the edge in this direction, the upper right is orthogonal to the lower left. It can be interpreted that the resolution of G decreases in the direction and ωAF_G increases. In order to correct this, for example, by applying the 3 × 3 smoothing filter shown in FIG. 19B only to the R color component, the resolution of R is reduced in the direction from the upper right to the lower left. Smoothing that matches the resolution of G may be performed. Thus, the change in ΔωAF (RG) due to the edge direction is corrected as shown in FIG. As a result, the variation of ΔωAF (RG) due to the edge direction is eliminated. Therefore, in step S5, ΔωAF (RG) = 0.3 is used as the reference value for the edge in any direction, depending on the magnitude thereof. The focal position detection process can be performed.

  In the latter case, the reference value is 0.3 and does not become 0, but this is not a problem. The value of ΔωAF (RG) in the in-focus state depends on the focusing operation such as the optical system and contrast AF. In many cases, it is seen with all color components of R, G, and B. Since the state where the contrast is improved overall is defined as the in-focus state, if attention is paid to only two colors (here, only R and G), ΔωAF (RG) is 0 even in the in-focus state. Not exclusively. The value after correction may be any value, and it is important to correct the image so that the dependency on the ΔωAF (RG) edge direction is eliminated.

  Thereafter, the process proceeds to step S5, and the control device 104 uses the axial chromatic aberration by executing the processes in steps S4 to S14 in FIG. The focus position is detected. Thereafter, the process ends.

According to the third embodiment described above, the following effects can be obtained.
The control device 104 corrects the change in resolution between the color components caused by factors other than axial chromatic aberration, and then determines the in-focus state. As a result, the edge blur between two colors caused by axial chromatic aberration in the focused state regardless of the state of the optical system, such as the edge direction and the colors on both sides of the edge, the cross-sectional shape across the edge, the shooting distance and the zoom position included in the input image The reference value of the width difference can be made uniform, and the accuracy of in-focus position detection performed by comparing the difference value with the reference value can be improved. Also, when performing focusing operations in subject tracking or video shooting, the subject moves in the input image and the edge direction rotates, the subject moves back and forth, the shooting distance changes, Since the reference value in the in-focus state does not change even if the subject color changes due to a change in conditions, the focus adjustment can be performed by performing a focus state detection using stable axial chromatic aberration.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the above-described first embodiment, the example in which the converted edge blur width difference Δw is calculated using Expression (12) has been described. However, the converted edge blur width difference Δw may be calculated by any of the following methods (A) to (D).

(A) The control device 104 measures the blur width wR of the R component edge and the blur width wG of the G component edge directly from the normalized edge detected as shown in FIG. The converted edge blur width difference Δw may be calculated by the following equation (13). According to this method, since Δw can be calculated by simple calculation, the amount of calculation for calculating Δw can be reduced as compared with the embodiment described above.

(B) The control device 104 may measure the gradient of the edge detected from the evaluation region and normalized as shown in FIG. 4, and indirectly calculate the converted edge blur width difference Δw. Specifically, the relationship of the blur width wR of the R component edge, the gradient gR of the R component edge, and the contrast difference C on both sides of the edge is expressed by the following equation (14). The relationship represented by the following equation (15) exists between the width wG, the gradient g of the edge of the G component, and the contrast difference C on both sides of the edge.

この方法の場合も、簡易な計算でΔｗを算出できるため、上述した実施の形態よりもΔｗを算出するための計算量を少なくすることができる。 For this reason, the control apparatus 104 can calculate conversion edge blur width difference (DELTA) w by following Formula (16).

Also in this method, since Δw can be calculated by a simple calculation, the amount of calculation for calculating Δw can be reduced as compared with the above-described embodiment.

(C) The control device 104 may measure the sum of color differences in the edge region E and indirectly calculate Δw. For example, considering the normalized R component edge R (x) and G component edge G (x) shown in FIG. 4, the color difference sum ΔaR-G between the R and G2 colors of the edge region E is Using the brightness area aR of the edge of the R component and the brightness area aG of the edge of the G component shown in FIG.
ΔaR−G = aR−aG (17)

Here, in the converted edge, the luminance area aR of the R component edge represented by the area of the triangle having the points 6a, 6b, and 6c as vertices, the blur width wR of the R component edge, There is a relationship expressed by the following equation (18) between the contrast difference C, the luminance area aG of the edge of the G component represented by the area of a triangle having the points 6d, 6e, and 6f as vertices, and G The relationship expressed by the following equation (19) exists between the blur width wG of the edge of the component and the contrast difference C on both sides of the edge.

この方法の場合も、簡易な計算でΔｗを算出できるため、上述した実施の形態よりもΔｗを算出するための計算量を少なくすることができる。 Therefore, the control device 104 can calculate the converted edge blur width difference Δw by the following equation (20).

Also in this method, since Δw can be calculated by a simple calculation, the amount of calculation for calculating Δw can be reduced as compared with the above-described embodiment.

(D) The control device 104 converts the edge region E in the R component edge R (x) and the G component edge G (x) after normalization shown in FIG. Are divided into sub-edge regions EL and EH. As a result, the blur width wR of the R component edge and the blur width wG of the G component edge are divided into two by a width common to R and G, respectively. In other words, the blur width wR of the edge of the R component is divided into two parts each of wR / 2 with the intermediate value of the edge region E as the boundary, and the blur width wG of the edge of the G component is wG with the intermediate value of the edge region E as the boundary. / 2 divided into two.

また、サブエッジ領域ＥＨについてのＲ、Ｇ２色間の色差和ΔａＨＲ−Ｇは、点７ａ、点７ｅ、点７ｇを頂点とする三角形の面積と、点７ａ、点７ｅ、点７ｆ、点７ｇを頂点とする台形の面積との差分となり、次式（２２）により算出される。

Next, the control device 104 calculates the color difference sum between the two colors for each of the divided sub-edge regions EL and EH. This sum of color differences is the difference between the luminance area of the edge of the R component and the luminance area of the edge of the G component as shown in the equation (17) in (C) described above. In the case of the example shown in FIG. 7, the color difference sum ΔaLR-G between the R and G2 colors for the sub-edge region EL is the area of a triangle having points 7a, 7b and 7c as vertices, and points 7a and 7d. , The difference from the area of the triangle having the point 7c as a vertex, and is calculated by the following equation (21).

Further, the color difference sum ΔaHR-G between the R and G2 colors for the sub-edge region EH is the area of the triangle having the points 7a, 7e, and 7g as vertices, and the points 7a, 7e, 7f, and 7g as vertices. And is calculated by the following equation (22).

この方法によれば、軸上色収差がある２色で共通するサブエッジ領域ＥＬ、ＥＨを参照して色差和を算出するため、上述した（Ａ）〜（Ｃ）の方法よりも精度良く換算エッジボケ幅差Δｗを算出することができる。 The control device 104 can calculate the converted edge blur width difference Δw, that is, ΔwR−G by the following equation (23) based on the equations (21) and (22).

According to this method, since the color difference sum is calculated with reference to the sub-edge regions EL and EH that are common to two colors having axial chromatic aberration, the converted edge blur width is more accurate than the methods (A) to (C) described above. The difference Δw can be calculated.

  Here, an example in which the edge region E is divided into two sub-edge regions EL and EH with an intermediate value as the boundary has been described. However, the edge region E is further divided into sub-edge regions, and the color difference sums for the respective sub-edge regions. May be calculated. In this case, the converted edge blur width difference Δw can be indirectly calculated by taking an appropriate weighted linear sum with respect to the color difference sum calculated for each sub-edge region.

(2) In the first embodiment described above, the control device 104 describes an example in which the edge region E is separated into the sub-edge regions EL and EH using the average value Emean of the pixels belonging to the edge region E as a threshold value. did. However, the control device 104 may use a value based on a statistical distribution of pixel values in the flat region F or the edge region E as a threshold value and separate the edge region E into sub-edge regions EL and EH. At this time, the sub-edge regions EL and EH may include overlapping pixels.

(3) In the above-described first embodiment, the control device 104 has described the example in which the edge region E is separated into two sub-edge regions EL and EH. However, the control device 104 may separate the edge region E into three or more sub-edge regions E1 to En. In this case, instead of the above-described evaluation value I, a value equivalent to the evaluation value I may be calculated by coefficient weighting calculation of the color difference sum of E1 to En by a predetermined coefficient, and each sub-edge region is further hierarchically arranged. The evaluation value I may be calculated by classification. If there is an edge in the evaluation area image that cannot be approximated to the converted edge, the evaluation value I may be corrected in consideration thereof.

(4) In the first embodiment described above, if the image is upsampled before the edge region is separated into the two sub-edge regions EL and EH, the in-focus position accuracy can be further improved. it can. In particular, when the lens position is close to the in-focus state and the edge width is sharp, the resolution of the region separation is improved by upsampling, so a great improvement in accuracy is expected. At this time, upsampling at a high magnification may be performed within a range where calculation load is allowed. The pixel interpolation method at the time of image enlargement may be simple bilinear interpolation using surrounding pixels.

(5) In the second embodiment described above, a reduced Bayer array RAW image obtained by adding and reading 2 × 2 pixels has been described as an example, but the scope of application of the present invention is not limited thereto. For example, the present invention may be applied to an original Bayer array RAW image. In the Bayer array RAW image interpolation processing, there is a case where interpolation on the luminance color difference plane is performed or an edge enhancement effect is included. Although they produce excellent images in terms of image quality, they may have the effect of eliminating subtle differences in the blur width between two colors for focal position detection using axial chromatic aberration. In such cases, interpolation is performed. More accurate focus position detection can be performed by using a RAW image that is not performed. The present invention may be applied to an interpolated image. By interpolating the Bayer array RAW image and selecting only the pixels derived from the original Bayer array RAW image from the images having RGB values in all pixels, it is possible to select between the color components affected by image processing such as interpolation. It may be possible to avoid the effect of resolution changes. In any case, it is important to select, from the input image, a combination of pixels whose resolution and sample position match as much as possible for each of the two color combinations for performing focal position determination based on axial chromatic aberration.

(6) In the second and third embodiments described above, an image generated by an interpolation device that is not optimized for pixel addition readout is taken as an example, but the subject of the present invention is not limited to this. Absent. For example, all image processing such as interpolation processing adaptive to the edge direction for the purpose of improving image quality, noise removal, edge enhancement and tone conversion, and images in which the resolution between the color components has changed due to the image sensor are also included. . The conditions for changing the resolution between each color component include not only those depending on the edge direction, but also those depending on the state of the optical system such as the color on both sides of the edge, the cross-sectional shape (edge shape) across the edge, the shooting distance and zoom position, etc. It is.

(7) In the above-described third embodiment, as shown in FIG. 18, an example has been described in which the change in ΔωAF (RG) due to the edge direction of the edges in eight vertical and horizontal diagonal directions is measured. However, a more accurate correction method and coefficient may be determined by investigating changes with respect to a finer edge inclination.

(8) In the above-described third embodiment, an example in which correction is performed by smoothing in step S4 of FIG. 6 has been described, but the correction means is not limited to smoothing. For example, processing that can correct a change in resolution between color components such as edge enhancement may be used. In addition to the change to the edge direction, the same change is checked by the optical system state, such as the color on both sides of the edge, the cross-sectional shape across the edge, the shooting distance and the zoom position, and the means to correct this effect is determined in advance. May be. In step S4, the feature of the edge of the input image may be examined each time, and the correction strength may be adaptively changed from a predetermined coefficient.

(9) In the above-described first to third embodiments, the example in which the present invention is applied to the camera 100 has been described. However, the present invention can also be applied to other photographing apparatuses having an autofocus function.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control device, 105 Memory card slot, 106 Monitor

Claims (13)

Image input means for inputting a subject image formed by an optical system having axial chromatic aberration as an input image;
Edge detection means for detecting an edge for each color component from the input image input by the image input means;
Correction means for correcting a change in resolution between the color components caused by the edge direction at the time of pixel addition reading from the image sensor or image processing based on the edge of each color component detected by the edge detection means; ,
The difference between the blur widths of the edges of the two color components having the longitudinal chromatic aberration among the edges of the color components detected from the image after the resolution change between the color components is corrected by the correction unit is calculated. A calculation means;
An imaging apparatus comprising: an in-focus state determining unit that determines an in-focus state based on the difference in blur width calculated by the calculating unit. The imaging device according to claim 1,
The calculating means calculates a predetermined evaluation value based on a color difference sum of edges of two color components having the axial chromatic aberration, and calculates the difference in blur width using the calculated evaluation value. An imaging device. The imaging device according to claim 2,
The imaging device classifies edges of two color components having axial chromatic aberration into a flat portion and an edge portion, and calculates the color difference sum for the edge portion as an object. The imaging device according to claim 1,
The image-capturing apparatus, wherein the calculating unit measures a blur width of edges of two color components having the longitudinal chromatic aberration, and calculates a difference between the blur widths based on the measurement result. The imaging device according to claim 1,
The imaging unit calculates the difference between the blur widths based on a blur width and an edge gradient of two edge components having axial chromatic aberration and a contrast difference between both sides of the edge. The imaging device according to claim 1,
The imaging device calculates the difference in blur width based on a luminance area of an edge of two color components having axial chromatic aberration and a contrast difference between both sides of the edge. In the imaging device according to any one of claims 1 to 6,
Color correction means for performing correction so as to eliminate the contrast difference between both sides of the edge of the two color components having the longitudinal chromatic aberration;
The image-capturing apparatus according to claim 1, wherein the calculating unit calculates the difference in blur width using an image after correction by the correcting unit. In the imaging device according to any one of claims 1 to 7,
The image pickup apparatus, wherein the calculating unit calculates the difference in blur width after performing an upsampling process on the input image. In the imaging device according to any one of claims 1 to 8 ,
The change in resolution between the color components depends on the edge direction with respect to the pixel arrangement of the image, the color on both sides of the edge, the shape of the edge, and the state of the optical system. The imaging device according to claim 1 ,
The image pickup apparatus, wherein the correction unit performs correction by applying a smoothing filter having a coefficient set in advance to a pixel of a color component whose resolution is to be matched with that of another color component. The imaging device according to claim 1 ,
The image pickup apparatus according to claim 1, wherein the correction unit performs correction by applying an edge enhancement filter having a coefficient set in advance to a pixel of a color component whose resolution is to be matched with another color component. The imaging device according to claim 10 or 11 ,
The imaging device according to claim 1, wherein the correction unit changes the coefficient based on a feature of an edge by an input image. An image input procedure for inputting a subject image formed by an optical system having axial chromatic aberration as an input image;
An edge detection procedure for detecting an edge for each color component from the input image input in the image input procedure;
Based on the edge for each color component detected in the edge detection procedure, a correction procedure for correcting a change in resolution between the color components caused by the edge direction at the time of pixel addition reading from the image sensor or image processing ;
The difference between the blur widths of the edges of the two color components having the longitudinal chromatic aberration among the edges of the color components detected from the image after the resolution change between the color components is corrected by the correction procedure is calculated. Calculation procedure,
A focus determination program for causing a computer to execute an in-focus state determination procedure for determining an in-focus state based on the difference in blur width calculated in the calculation procedure.

Images