JP2013097154A - Distance measurement device, imaging apparatus, and distance measurement method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a measurement distance apparatus and the like which derive highly reliable distance information on the basis of images obtained from an imaging element.SOLUTION: A distance measurement apparatus includes: an imaging optical system 9; a band limiting filter 12 which makes the center of gravity for an amount of light different in three or more bands of light beams passing through a pupil area of the imaging optical system 9; a color imaging element 22; a phase difference autofocus (AF) unit 42 which calculates a distance to a subject on the basis of color images that correspond to multiple bands and are obtained from the imaging element 22; a phase difference suitability determination unit 41 which determines whether or not the color images are suitable for phase difference calculation; and an autofocus (AF) control unit 38 which causes, when three or more color images are determined by the phase difference suitability determination unit 41 as being suitable for the phase difference calculation, the phase difference AF unit 42 to calculate a distance to the subject on the basis of either a correlation value being a combination of selected two suitable color images or a correlation value being all combinations of two suitable color images.

Description

  The present invention relates to a distance measuring device, an imaging device, and a distance measuring method that can acquire distance information based on an image obtained from an image sensor.

  The distance information is used for AF processing by an automatic focus adjustment (AF) mechanism, for creating a stereoscopic image, or for image processing (for example, subject extraction processing, background extraction processing, or blur amount control). The present invention can be used to realize various functions in the imaging apparatus, such as for performing image processing.

  Various methods for obtaining such distance information have been proposed. For example, an active distance measurement method in which illumination light is irradiated and reflected light from a subject is received to perform distance measurement, or a baseline length. The focus lens is driven so as to increase the contrast of images acquired by a plurality of image pickup devices (for example, stereo cameras) arranged in accordance with the principle of triangulation, or the image acquired by the image pickup device itself. Various methods such as a contrast AF method have been proposed.

  However, the active distance measurement method requires a dedicated member for distance measurement such as a light projection device for distance measurement, and the triangular distance measurement method requires a plurality of image pickup devices. Will increase. On the other hand, since the contrast AF method uses an image acquired by the imaging apparatus itself, a dedicated distance measuring member or the like is not required. However, the contrast value peak is obtained by performing multiple imaging while changing the position of the focus lens. Therefore, it takes time to search for a peak corresponding to the in-focus position, and it is difficult to perform high-speed AF.

  Under such a background, as a technology to acquire distance information while suppressing an increase in size and cost, a light beam that passes through the pupil of the lens is divided into a plurality of light beams, and a light beam that passes through one pupil region in the lens The distance information to the subject is obtained by performing a correlation calculation between the pixel signal obtained from the pixel signal and the pixel signal obtained from the light beam that has passed through another pupil region different from the one pupil region in the lens. Techniques to do this have been proposed. As a technique for simultaneously acquiring such pupil division images, for example, there is a technique for disposing a light shielding plate (mask) on a pixel for distance detection.

  As a technique for performing pupil division without using a light shielding plate (mask), for example, Japanese Patent Laid-Open No. 2001-174696 discloses a pupil color division filter (for example, an RG transmission filter) having different spectral characteristics for each partial pupil. And a GB transmission filter) are interposed in the photographic optical system, and a subject image from the photographic optical system is received by a color image pickup device, thereby dividing the pupil by color. That is, the image signal output from the color image sensor is color-separated, and the relative shift amount between the same subjects on each color image (for example, R image and B image in the above example) is detected, thereby focusing. Two pieces of focusing information are acquired: the focusing shift direction indicating whether the position is shifted to the near distance side or the far distance side, and the focusing shift amount which is a shift amount from the in-focus position in that direction. The

  Furthermore, a technique for determining the moving direction of the lens using the chromatic aberration of the lens is disclosed. For example, in the conventional technique column of Japanese Patent Laid-Open No. 10-200904, in the case where the RGB color components are equal subjects, the high-frequency component of the R color is the highest if the lens position is closer to the in-focus position, Based on the result of comparing the high-frequency components of each color using the fact that the high-frequency component of B color is the highest if the lens position is at infinity from the focus position, the lens position is closer to the focus position A technique is described in which the lens driving direction is determined by determining whether the lens is on the infinity side or not, thereby eliminating the driving in the reverse direction and increasing the focusing speed.

JP 2001-174696 A JP-A-10-200904

  However, in the technique described in Japanese Patent Laid-Open No. 2001-174696, for example, when only the R and G components exist in the phase difference calculation area of the subject (when the B component is almost 0), the B component Since the pixel value cannot be acquired, the phase difference calculation between the R component and the B component cannot be performed.

  Also, with the technique described in Japanese Patent Laid-Open No. 10-200904, the driving direction cannot be accurately estimated in the case of a subject in which the RGB color components are not uniform. Furthermore, in an interchangeable lens imaging device, there are lenses with small chromatic aberration depending on the lens to be mounted, and in this case, a difference in high-frequency components hardly occurs, and it is difficult to obtain distance information accurately. Become.

  The present invention has been made in view of the above circumstances, and provides a distance measuring device, an imaging device, and a distance measuring method capable of acquiring more reliable distance information based on an image obtained from an imaging element. The purpose is to do.

  In order to achieve the above object, a distance measuring device according to an aspect of the present invention includes an imaging device that receives light in three or more wavelength bands, performs photoelectric conversion, and generates a color image for each wavelength band. A band limit that varies the center of gravity of the amount of light when passing through the pupil region of the imaging optical system and the imaging optical system that forms light of the wavelength band as a subject image on the imaging device. A filter, a phase difference calculation unit that calculates a distance to a subject based on a phase difference amount caused by a difference in the position of the center of gravity of the plurality of color images obtained from the image sensor, and obtained from the image sensor A phase difference suitability determining unit that determines whether or not the color image is suitable as a color image used by the phase difference computing unit for calculation, and the number of color images determined to be suitable by the phase difference suitability determining unit When there are 3 or more In the phase difference calculation unit, a correlation value of a combination of two color images selected from the three or more color images, or a correlation of all combinations of two color images among the three or more color images And a phase difference calculation control unit that calculates a distance to the subject based on the value.

  An imaging device according to another aspect of the present invention includes the distance measuring device according to the above aspect.

  In the distance measuring method according to still another aspect of the present invention, the light is divided into light of three or more wavelength bands by changing the center of gravity of the amount of light when passing through the pupil region to three or more. The light for each wavelength band is formed as a subject image, the light for each wavelength band is received and photoelectrically converted to generate a color image for each wavelength band, and the color image is used for phase difference calculation 2 is selected from the three or more color images when it is determined whether the image is suitable and the number of the color images determined to be suitable for the phase difference calculation is three or more. In this method, the distance to the subject is calculated based on the correlation value of a combination of two color images or the correlation value of all combinations of two color images among the three or more color images.

  According to the distance measuring device, the imaging device, and the distance measuring method of the present invention, it is possible to acquire more reliable distance information based on the image obtained from the imaging element.

1 is a block diagram illustrating a configuration of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a block diagram showing a configuration of a phase difference AF unit in the first embodiment. FIG. 3 is a diagram for explaining a pixel array of the image sensor in the first embodiment. The figure for demonstrating one structural example of the band-limiting filter in the said Embodiment 1. FIG. FIG. 3 is a diagram illustrating the barycentric positions of RGB colors when passing through a pupil region of a lens in the first embodiment. In the said Embodiment 1, the top view which shows the mode of a subject light beam condensing when imaging the to-be-photographed object in a focus position. FIG. 3 is a plan view showing a state of subject light flux condensing when an image of a subject closer to the in-focus position is imaged in the first embodiment. FIG. 3 is a diagram illustrating a blur shape formed by light from one point on a subject that is closer to the in-focus side than the in-focus position in the first embodiment. In the said Embodiment 1, the figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object in the short distance side from a focus position for every color component. In the said Embodiment 1, the top view which shows the mode of a subject light beam condensing when imaging the to-be-photographed object from the in-focus position. FIG. 4 is a diagram showing a blur shape formed by light from one point on a subject that is farther than the in-focus position in the first embodiment. In the said Embodiment 1, the figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object on the far side from the focus position for every color component. In the said Embodiment 1, the diagram which shows the example of the correlation value graph obtained by calculating by a SAD system, shifting the relative position of two images. FIG. 3 is a diagram showing an example of a correlation value graph obtained by performing calculation according to the ZNCC method while shifting the relative positions of two images in the first embodiment. 5 is a flowchart showing ranging processing in the imaging apparatus according to the first embodiment. FIG. 16 is a flowchart showing a first example of processing for determining compatibility of each color in the phase difference detection area in step S3 of FIG. 15 in the first embodiment. In the said Embodiment 1, the flowchart which shows the 2nd example of the process of a compatibility determination of each color of the phase difference detection area of step S3 of FIG. In the said Embodiment 1, the flowchart which shows the 3rd example of the process of the compatibility determination of each color of the phase difference detection area of step S3 of FIG. In the said Embodiment 1, the flowchart which shows the 4th example of the process of the compatibility determination of each color of the phase difference detection area of step S3 of FIG. FIG. 16 is a flowchart showing a first example of an adaptive color set selection process in step S5 of FIG. 15 in the first embodiment. FIG. FIG. 16 is a flowchart showing a second example of matching color set selection processing in step S5 of FIG. 15 in the first embodiment. FIG. 9 is a flowchart illustrating distance measurement processing in the imaging apparatus according to the second embodiment of the present invention. FIG. 4 is a block diagram showing a configuration of a phase difference AF unit in the second embodiment. 10 is a flowchart illustrating ranging processing in the imaging apparatus according to the third embodiment of the present invention. In the said Embodiment 3, the figure which shows the example of the correlation value graph of R image and B image. In the said Embodiment 3, the figure which shows the example of the correlation value graph of R image and G image. In the said Embodiment 3, the figure which shows the example of the correlation value graph of G image and B image. In the said Embodiment 3, the figure which shows the example of the correction | amendment correlation value graph of R image and G image which rescaled according to the correlation value graph of R image and B image for the horizontal axis. The figure which shows the example of the correction | amendment correlation value graph of G image and B image which rescaled according to the correlation value graph of R image and B image in the said Embodiment 3. In the third embodiment, the correlation value graph obtained by adding the correlation value graph between the R image and the B image, the correction correlation value graph between the R image and the G image, and the correction correlation value graph between the G image and the B image. The figure which shows the example of a graph.

Embodiments of the present invention will be described below with reference to the drawings.
[Embodiment 1]

  FIG. 1 to FIG. 21 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing the configuration of the imaging apparatus.

  An imaging device including the distance measuring device according to the present embodiment is configured as a digital still camera, for example. However, here, a digital still camera is taken as an example, but the imaging device may be any device as long as it has a color imaging element and has an imaging function. Still cameras, video cameras, mobile phones with cameras, personal digital assistants with cameras (PDAs with cameras), personal computers with cameras, surveillance cameras, endoscopes, digital microscopes (digital microscopes), binoculars with ranging functions, etc. .

  The imaging apparatus includes a lens unit 1 and a body unit 2 that is a main body portion to which the lens unit 1 is detachably attached via a lens mount. Here, a case where the lens unit 1 is detachable will be described as an example, but of course, it may not be detachable.

  The lens unit 1 includes an imaging optical system 9 including a lens 10 and a diaphragm 11, a band limiting filter 12, a lens control unit 14, and a lens side communication connector 15.

  The body unit 2 includes a shutter 21, an imaging element 22, an imaging circuit 23, an imaging drive unit 24, an image processing unit 25, an image memory 26, a display unit 27, an interface (IF) 28, and a system controller. 30, a sensor unit 31, an operation unit 32, a strobe control circuit 33, a strobe 34, and a body side communication connector 35. In addition, although the recording medium 29 is also described in the body unit 2 of FIG. 1, this recording medium 29 is a memory card (smart media, SD card, xD picture card, etc.) which is detachable with respect to the imaging device. Since it is configured, the configuration may not be unique to the imaging apparatus.

  First, the imaging optical system 9 in the lens unit 1 is for forming a subject image on the imaging element 22. The lens 10 of the imaging optical system 9 includes a focus lens for performing focus adjustment. Although the lens 10 is generally composed of a plurality of lenses in general, only one lens is shown in FIG. 1 for simplicity.

  The diaphragm 11 of the imaging optical system 9 is for adjusting the brightness of the subject image formed on the imaging element 22 by regulating the passage range of the subject light flux passing through the lens 10. When the aperture diameter of the aperture 11 is changed, the pupil region of the imaging optical system 9 is also changed.

  The band limiting filter 12 is disposed on the optical path of the photographing light beam that passes through the image pickup optical system 9 and reaches the image pickup device 22 (preferably at or near the position of the stop 11 of the image pickup optical system 9). The barycentric position of the amount of light when the light in the band passes through the pupil region of the imaging optical system 9 is changed to three or more. The band limiting filter 12 in this embodiment is a first band (here, the band indicates a wavelength band) in the photographic light flux that attempts to pass through a first area that is a part of the pupil area of the imaging optical system 9. The same applies hereinafter.) The first band limitation that blocks the light of the second band and the third band and blocks the light of the second band and the third band, and the second band that is another part of the pupil area of the imaging optical system 9 This is a filter that performs second band limitation that blocks light in the second band in the imaging light flux to be attempted and passes light in the first band and the third band.

  FIG. 4 is a diagram for explaining a configuration example of the band limiting filter 12. In the band limiting filter 12 of the configuration example shown in FIG. 4, the pupil region of the imaging optical system 9 is divided into a first region and a second region. In other words, the band limiting filter 12 has a left half of the G (green) component and R (red) when the image pickup device is viewed from the image pickup device 22 in a standard posture (so-called posture in which the camera is held in a normal horizontal position). RG filter 12r that passes the component and blocks the B (blue) component (that is, the B (blue) component is one of the first band and the second band), the right half is the G component and The GB filter 12b is configured to pass the B component and block the R component (that is, the R component is the other of the first band and the second band). Therefore, the band limiting filter 12 allows the G component included in the light passing through the aperture 11 of the imaging optical system 9 to pass through the entire region (that is, the G component is the third band), and the R component is opened. Only the half area of the aperture is allowed to pass, and the B component is allowed to pass only for the remaining half of the aperture.

  As described above, the band limiting filter 12 limits the pass region in the pupil region of the imaging optical system 9 by, for example, the band (color) of the light to be transmitted, so that the light of each band divided by the band limiting filter 12 can be imaged optically. The barycentric position (color pupil barycentric position) of the amount of light when passing through the pupil region of the system 9 is made different for each band (color).

  FIG. 5 is a diagram illustrating the barycentric positions of the RGB colors when passing through the pupil region of the lens 10.

  The center of gravity Pg of the light amount when G passes through the pupil region is the center of the pupil region of the lens 10 and coincides with the optical axis center of the imaging optical system 9. Further, the gravity center position Pr of the light amount when R passes through the pupil region is a position deviated from the center of the optical axis of the imaging optical system 9 and shifted to the left side in FIG. Furthermore, the barycentric position Pb of the amount of light when B passes through the pupil region is a position decentered from the center of the optical axis of the imaging optical system 9 and shifted to the right in FIG. In this positional relationship, the centroid distance between the R centroid position Pr and the B centroid position Pb is Drb, the centroid distance between the R centroid position Pr and the G centroid position Pg is Drg, and the G centroid positions Pg and B. The center-of-gravity distance from the center-of-gravity position Pb is Dgb. At this time, the center-of-gravity distance Drb is, for example, twice the center-of-gravity distance Drg and the center-of-gravity distance Dgb (however, this is the case of the band limiting filter 12 as shown in FIG. Depending on the value).

  If the RGB spectral transmission characteristics of the band limiting filter 12 are different from the RGB spectral transmission characteristics of an element filter (see FIG. 3) configured on the imaging device 22, for example, the RG filter 12r and the GB filter 12b. As described later, the accuracy of the position information acquired from the images obtained based on the difference in the spatial position decreases, or a light amount loss due to spectral characteristic mismatch occurs. Therefore, it is desirable that the spectral transmission characteristic of the band limiting filter 12 is the same as or as close as possible to the spectral transmission characteristic of the element filter of the image sensor 22.

  The lens control unit 14 controls the lens unit 1. That is, the lens control unit 14 drives the aperture 11 to change the aperture diameter and drive the lens 10 based on a command received from the system controller 30 via the lens side communication connector 15 and the body side communication connector 35. Control to focus.

  The lens-side communication connector 15 enables communication between the lens control unit 14 and the system controller 30 by connecting the lens unit 1 and the body unit 2 with a lens mount and connecting to the body-side communication connector 35. Connector.

  Next, the shutter 21 in the body unit 2 is an optical shutter for adjusting the exposure time of the image sensor 22 by regulating the passage time of the subject light beam reaching the image sensor 22 from the lens 10. Although an optical shutter is used here, an element shutter (electronic shutter) by the image sensor 22 may be used instead of or in addition to the optical shutter.

  The image sensor 22 captures a subject image related to light in three or more wavelength bands (for example, RGB, but is not limited thereto) formed by the imaging optical system 9 through the band limiting filter 12. Each is a color image pickup device that receives light and performs photoelectric conversion to generate an image for each band (hereinafter, referred to as a color image as appropriate), and is configured as, for example, a CCD or a CMOS. Here, the configuration of the color image sensor may be a single-plate image sensor provided with an on-chip element color filter, or a three-plate system using a dichroic prism that performs color separation into RGB color lights. However, it may be an image sensor of a method capable of acquiring RGB imaging information according to the position in the depth direction of the semiconductor at the same pixel position, or any imaging element that can acquire imaging information of a plurality of wavelength bands. It does n’t matter.

  For example, with reference to FIG. 3, a configuration example of a single-plate color image sensor often used in a general digital still camera will be described. FIG. 3 is a diagram for explaining the pixel arrangement of the image sensor 22. In the present embodiment, the plurality of wavelength band lights transmitted by the element color filter mounted on-chip are R, G, and B. As shown in FIG. An image sensor is configured. Therefore, when the image pickup device 22 has the configuration as shown in FIG. 3, only one color component is obtained per pixel, so that the image processing unit 25 performs a demosaicing process. A color image in which three colors of RGB are aligned per pixel is generated.

  The image pickup circuit 23 amplifies (gain adjusts) the image signal output from the image pickup device 22, or performs A / D conversion when the image pickup device 22 is an analog image pickup device and outputs an analog image signal. Thus, a digital image signal (hereinafter also referred to as “image information”) is generated (when the image pickup device 22 is a digital image pickup device, it is already digital when it is input to the image pickup circuit 23). Therefore, A / D conversion is not performed). The imaging circuit 23 outputs an image signal to the image processing unit 25 in a format corresponding to the imaging mode switched by the imaging driving unit 24.

  The imaging drive unit 24 supplies a timing signal and power to the imaging element 22 and the imaging circuit 23 based on a command from the system controller 30 to cause the imaging element to perform exposure, reading, element shuttering, and the like, and also to the imaging element 22. Control is performed to execute gain adjustment and A / D conversion by the imaging circuit 23 in synchronism with the above operations. The imaging drive unit 24 also performs control to switch the imaging mode of the imaging element 22.

  The image processing unit 25 performs various digital image processing. The image processing unit 25 performs, for example, black level correction, white balance adjustment, γ correction, defective pixel correction, demosaicing, image information color information conversion processing, image information pixel number conversion processing, and the like. Further, the image processing unit 25 includes an inter-color correction unit 36 and a color image generation unit 37.

  Since the band limiting filter 12 described above limits the pass region in the pupil region of the imaging optical system 9 according to the band (color) of the light to pass, the pass light has different pupil pass region areas depending on the band. Therefore, the brightness will be different (in the case of the configuration shown in FIG. 4, the G light passes through the entire pupil region, but the R light and B light pass through only half of the pupil region. The R image and the B image are, for example, half as bright as the G image). The inter-color correction unit 36 is for correcting such a difference in brightness between bands (between colors).

  Further, when the band limiting filter 12 as shown in FIG. 4 is used, a spatial positional shift occurs between the R image and the B image in the out-of-focus portion (FIGS. 7 to 7 described later). 12). The color image generation unit 37 corrects such a spatial positional shift and performs a colorization process for generating a planar image without a color shift.

  The image memory 26 is a memory capable of high-speed writing and reading, and is constituted by, for example, an SDRAM (Synchronous Dynamic Random Access Memory). The image memory 26 is used as a work area for image processing and is also a work area of the system controller 30. Also used as For example, the image memory 26 not only stores the final image processed by the image processing unit 25 but also appropriately stores each intermediate image in a plurality of processing steps by the image processing unit 25.

  The display unit 27 includes an LCD or the like, and an image processed for display by the image processing unit 25 (an image read from the recording medium 29 and processed for display by the image processing unit 25 is also included). Display). Specifically, the display unit 27 performs live view image display, confirmation display when recording a still image, reproduction display of a still image or a moving image read from the recording medium 29, and the like.

  The interface (IF) 28 is detachably connected to the recording medium 29, and transmits information to be recorded on the recording medium 29 and information read from the recording medium 29.

  The recording medium 29 records an image processed for recording by the image processing unit 25 and various data related to the image, and is configured as a memory card or the like as described above.

  The sensor unit 31 is, for example, a camera shake sensor configured by an acceleration sensor or the like for detecting blur of the imaging device, a temperature sensor for measuring the temperature of the imaging device 22, and a brightness for measuring the brightness around the imaging device. Includes brightness sensor, etc. The detection result by the sensor unit 31 is input to the system controller 30. Here, the detection result by the camera shake sensor is used to drive the image pickup device 22 and the lens 10 to perform camera shake correction or to perform camera shake correction by image processing. The detection result by the temperature sensor is used to control the drive clock by the imaging drive unit 24 and to estimate the amount of noise in the image obtained from the image sensor 22. Furthermore, the detection result by the brightness sensor is used, for example, to appropriately control the luminance of the display unit 27 according to the ambient brightness.

  The operation unit 32 changes a power switch for turning on / off the power of the image pickup device, a release button including a two-stage press button for inputting an image pickup operation such as a still image or a moving image, an image pickup mode, and the like. Mode buttons, cross keys used to change selection items, numerical values, and the like. A signal generated by the operation of the operation unit 32 is input to the system controller 30.

  The strobe control circuit 33 controls the light emission amount and the light emission timing of the strobe 34 based on a command from the system controller 30.

  The strobe 34 is a light source that emits illumination light to the subject under the control of the strobe control circuit 33.

  As described above, the body side communication connector 35 is connected between the lens control unit 14 and the system controller 30 when the lens unit 1 and the body unit 2 are coupled by the lens mount and connected to the lens side communication connector 15. It is a connector that enables communication.

  The system controller 30 controls the body unit 2 and also controls the lens unit 1 via the lens control unit 14, and is a control unit that integrally controls the imaging apparatus. The system controller 30 reads a basic control program of the image pickup apparatus from a non-illustrated non-volatile memory such as a flash memory, and controls the entire image pickup apparatus in accordance with an input from the operation unit 32.

  For example, the system controller 30 controls each part in the lens unit 1 via the lens control unit 14, controls and drives the shutter 21, and a camera shake (not shown) based on a detection result by the acceleration sensor of the sensor unit 31. The correction mechanism is controlled to perform camera shake correction or the like. Further, the system controller 30 sets the mode of the imaging device (settings such as a still image mode for capturing a still image and a moving image mode for capturing a moving image) in response to an input from the mode button of the operation unit 32. It is intended to do.

  Furthermore, the system controller 30 includes an AF (autofocus) control unit 38 and a distance calculation unit 39.

  The distance calculation unit 39 includes a phase difference AF unit 42 and a contrast AF unit 43.

  The phase difference AF unit 42 functions as a phase difference calculation unit, and at least of three or more band images (three or more color images) having different color pupil barycentric positions in the image obtained from the image sensor 22. Based on the two color images, the phase difference amount is calculated, and the distance to the subject is calculated. Specifically, the phase difference AF unit 42 is the amount of phase difference between a plurality of color images generated due to the difference in the barycentric position of the light amount when passing through each pupil region, that is, the R image and the B image. Calculate at least one of the phase difference amount, the phase difference amount between the R image and the G image, and the phase difference amount between the B image and the G image, and calculate the distance information to the subject based on the calculated phase difference amount. It is supposed to be.

  Here, in the example of the band limiting filter 12 shown in FIG. 4, light in the three bands of RGB is allowed to pass through with the color centroid positions different from each other. However, the band limiting filter 12 may be a filter that allows light in four or more bands to pass therethrough, and the position of the center of gravity of the color pupil of the light in the band that passes through does not have to be all different. For example, the band limiting filter 12 is a filter that transmits light in four bands R, G1, G2, and B, and the color pupil centroid positions of G1 and G2 may be the same (color pupil centroid position). It is sufficient if there are three or more different bands). For the bands where the color pupil centroid positions are the same, it is determined in advance which band color image is used for phase difference detection, or which is used for phase difference detection based on the histogram of each color image. It shall be decided whether to use. For example, when G1 of G1 and G2 is used for phase difference detection, the color pupil centroid positions of R, G1, and B are independent, and the number of independent color pupil centroid positions is 3. is there. The number of such independent color pupil centroid positions is generally referred to as “maxN” (maxN is an integer of 3 or more).

  The distance information obtained by the phase difference AF unit 42 is used for, for example, autofocus (AF), and the system controller 30 controls the lens unit 1 based on the distance information to perform phase difference AF (of course, the distance information The target of use is not limited to AF, and may be used for blur amount control, stereoscopic image generation, and the like.

  The contrast AF unit 43 performs contrast AF by driving the focus lens based on the image obtained from the image sensor 22. More specifically, the contrast AF unit 43 may be an image signal output from the image processing unit 25 (this image signal may be a G image having a high ratio including a luminance component, or color shift may be performed by the above-described colorization processing. A contrast signal (also referred to as an AF evaluation value) is generated from the corrected luminance signal image, and the focus lens in the lens 10 is controlled via the lens control unit 14. . That is, the contrast AF unit 43 extracts a high-frequency component by applying a filter, for example, a high-pass filter, to the image signal to obtain a contrast value. Then, the contrast AF unit 43 acquires the contrast value by changing the focus lens position, moves the focus lens in the direction in which the contrast value increases, and further acquires the contrast value. By repeatedly performing such processing, control is performed so that the focus lens is driven to the focus lens position (focus position) at which the maximum contrast value is obtained.

  The AF control unit 38 is a phase difference calculation control unit, and controls the distance calculation unit 39 based on the image obtained from the image sensor 22 to at least one of the phase difference AF unit 42 and the contrast AF unit 43 described above. The distance calculation using is performed.

  The AF control unit 38 includes a phase difference matching determination unit 41. The phase difference matching determination unit 41 is an image in which images of a plurality of bands having different color pupil centroid positions described above obtained from the image sensor 22 (a plurality of color images having different color pupil centroid positions) are suitable for phase difference detection. (Or whether or not it is suitable as a color image used for phase difference calculation) (or to what extent it is suitable for phase difference detection).

  Next, FIG. 2 is a block diagram showing a configuration of the phase difference AF unit 42.

  The phase difference AF unit 42 includes a color signal selection unit 51, a reference correction unit 52, and a phase difference calculation unit 53.

  The color signal selection unit 51 has a set of color images (color pupil barycentric positions different from each other) used for the phase difference calculation from among the color images determined to be suitable for the phase difference calculation by the phase difference matching determination unit 41 described above. Image of two bands) is selected.

  The reference correction unit 52 determines the distance of the center of gravity when the light of the two bands related to the two color images selected by the color signal selection unit 51 passes through the pupil region of the lens 10 (the center of gravity as shown in FIG. 5). The phase difference amount calculated by the phase difference calculation unit 53 is corrected according to the distance, or the subject distance calculated based on the phase difference amount is corrected (however, when correction is not necessary). Will of course not be corrected).

  Here, the relationship between the phase difference amount calculated by the phase difference AF unit 42 and the subject distance is stored in advance in a nonvolatile memory such as a flash memory (not shown) in the system controller 30, for example. However, if the relationship between the phase difference amount and the subject distance is stored for all combinations of two color images having different centroid positions, a large amount of storage capacity is required. Therefore, the relationship between the phase difference amount and the subject distance is stored only for a standard set of color images (for example, an R image and a B image), and in the case of a combination of two non-standard color images, The reference correction unit 52 is configured to perform correction according to the centroid distance.

  The phase difference calculation unit 53 obtains a phase difference amount between these two color images (performs phase difference detection) by performing a phase difference calculation on the two color images selected by the color signal selection unit 51. is there.

  In the example shown in FIG. 2, the color signal selection unit 51 and the reference correction unit 52 are provided in the phase difference AF unit 42, but may be provided in the AF control unit 38, the system controller 30 or the image. You may provide in the process part 25 and you may employ | adopt another structure.

  First, the spatial positional shift of each color image will be described with reference to FIGS. FIG. 6 is a plan view showing a state of subject light flux condensing when imaging a subject at the in-focus position, and FIG. 7 is a subject light flux concentrating when imaging a subject closer to the in-focus position. FIG. 8 is a plan view showing the state of light, FIG. 8 is a diagram showing the shape of a blur formed by light from one point on the subject closer to the focus position, and FIG. 9 is a closer distance than the focus position. FIG. 10 is a diagram showing, for each color component, the shape of a blur formed by light from one point on a subject on the side, and FIG. 10 is a diagram of subject light flux collection when imaging a subject farther from the in-focus position. FIG. 11 is a plan view showing a state, FIG. 11 is a diagram showing the shape of a blur formed by light from one point on the subject on the far side from the in-focus position, and FIG. 12 is on the far side from the in-focus position. It is a figure which shows the shape of the blur formed with the light from one point on a certain subject for every color component. In the description of the blur shape, the case where the aperture of the diaphragm 11 has a circular shape will be described as an example.

  When the subject OBJc is at the in-focus position, the light emitted from one point on the subject OBJc has a G component that passes through the entire band limiting filter 12 as shown in FIG. Both the R component that passes only through the filter 12r and the B component that passes through only the other half of the GB filter 12b of the band limiting filter 12 are condensed at one point on the image sensor 22, and the subject image IMGrgb as a point image is obtained. In order to form, although there is a difference in the amount of light according to the area of the pupil passing region as described above, no positional deviation occurs between colors. Therefore, when the subject OBJc at the in-focus position is imaged, a subject image IMGrgb without color blur is formed.

  On the other hand, when the subject OBJn is, for example, closer to the in-focus position, the light emitted from one point on the subject OBJn is used for the G component as shown in FIGS. A subject image IMGg that forms a circular blur is formed, a subject image IMGr that forms a semicircular blur of the left half is formed for the R component, and a subject image IMGb that forms a semicircular blur of the right half is formed for the B component. Therefore, when the subject OBJn that is closer to the in-focus position is imaged, a blurred image is formed in which the R component subject image IMGr is shifted to the left side and the B component subject image IMGb is shifted to the right side. The left and right positions of the R component and the B component are the same as the left and right positions of the R component transmission region (RG filter 12r) and the B component transmission region (GB filter 12b) in the band limiting filter 12 when viewed from the image sensor 22. . Further, as shown in FIGS. 8 and 9, the G component subject image IMGg is a blurred image extending over the R component subject image IMGr and the B component subject image IMGb. As the subject OBJn is further away from the in-focus position, the blur increases, and the separation distance between the gravity center position Cr of the R component subject image IMGr and the gravity center position Cb of the B component subject image IMGb, the R component The distance between the centroid position Cr of the subject image IMGr and the centroid position Cg of the G component image IMGg, and the distance between the centroid position Cg of the G component image IMGg and the centroid position Cb of the B component image IMGb Will grow.

  On the other hand, when the subject OBJf is on the far side of the in-focus position, for example, the light emitted from one point on the subject OBJf causes a circular blur for the G component as shown in FIGS. A subject image IMGg is formed, a subject image IMGr that forms a semicircular blur of the right half is formed for the R component, and a subject image IMGb that forms a semicircular blur of the left half is formed for the B component. Therefore, when the subject OBJf is imaged farther than the in-focus position, a blurred image is formed in which the R component subject image IMGr is shifted to the right side and the B component subject image IMGb is shifted to the left side. The left and right positions of the R component and the B component are opposite to the left and right positions of the R component transmission region (RG filter 12r) and the B component transmission region (GB filter 12b) of the band limiting filter 12 when viewed from the image sensor 22. . Similarly, the G component subject image becomes a blurred image extending over the R component subject image IMGr and the B component subject image IMGb (see FIGS. 11 and 12). As the subject OBJf is further away from the in-focus position, the blur increases, and the separation distance between the centroid position Cr of the R component subject image IMGr and the centroid position Cb of the B component subject image IMGb, the R component The distance between the centroid position Cr of the subject image IMGr and the centroid position Cg of the G component image IMGg, and the distance between the centroid position Cg of the G component image IMGg and the centroid position Cb of the B component image IMGb Will grow.

  Accordingly, the subject distance can be calculated by calculating the phase difference amount based on the correlation with respect to a combination of two color images having different centroid positions when passing through the pupil region of the lens 10.

  Note that the G component subject image IMGg becomes the same information as the blurred image when the band limiting filter 12 is not used, on either the short distance side or the long distance side. It can be used as a standard image in the colorization process performed by 37.

  Next, an example of a correlation value graph created when the phase difference calculation unit 53 performs phase difference detection will be described with reference to FIGS. 13 and 14. FIG. 13 is a diagram showing an example of a correlation value graph obtained by performing the calculation by the SAD method while shifting the relative positions of the two images. FIG. 14 is a graph by performing the calculation by the ZNCC method while shifting the relative positions of the two images. It is a diagram which shows the example of the correlation value graph obtained. Here, the horizontal axis of the graphs of FIGS. 13 and 14 represents the image shift amount (unit: pixel pitch) of the two images to be correlated, and the vertical axis represents the correlation value obtained as a result of the correlation calculation. .

  First, when two images are subjected to correlation calculation by the SAD method, a correlation value forming a V-shaped valley as shown in FIG. 13 is obtained, and an image shift amount at the bottom of the valley close to 0 becomes a calculated phase difference amount. .

  Further, when two images are subjected to correlation calculation by the ZNCC method, a correlation value forming an inverted V-shaped peak as shown in FIG. 14 is obtained, and the phase difference amount obtained by calculating the image shift amount at the peak of the peak close to 1 is obtained. Become.

  Next, FIG. 15 is a flowchart showing distance measurement processing in the imaging apparatus.

  When this process is started, a phase difference detection area that is a target range for acquiring distance information is selected (step S1). This phase difference detection area is automatically set according to the shooting mode set in the imaging apparatus or manually set by the user. Specific examples of the phase difference detection area include all pixel areas, a subject area of interest automatically set by the system controller 30 (for example, a face detected by face detection of the image processing unit 25, or the center of the screen), An area set or selected by the user.

  Subsequently, an image is acquired by the imaging element 22 (step S2), and based on the acquired image, the phase difference matching determination unit 41 performs a matching determination process for each color image in the phase difference detection area (step S3).

  Then, based on the processing result of step S3, the AF control unit 38 determines whether an image of three or more colors is suitable for phase difference detection (step S4).

  Here, when it is determined that an image of three or more colors is suitable for phase difference detection, the color signal selection unit 51 based on the control of the AF control unit 38 selects from among the three or more colors determined to be compatible. A matching color set selection process for selecting one set (two colors) used for phase difference acquisition is performed (step S5).

  Next, under the control of the AF control unit 38, the phase difference calculation unit 53 creates a correlation value graph of the selected set of color images, and acquires a phase difference based on the created correlation value graph (step S6). .

  Further, the reference correction unit 52 controls the acquired phase difference based on the control of the AF control unit 38 to determine the center-of-gravity distance when the light related to the two selected colors passes through the pupil region of the lens 10 (as shown in FIG. 5). Correction is performed according to the center of gravity distance) (step S7). That is, for example, when the band limiting filter 12 having the configuration shown in FIG. 4 is used, the centroid distance Drb is twice the centroid distance Drg and the centroid distance Dgb. Therefore, when the colors selected by the color signal selection unit 51 are R and G, or G and B, the image shift amount half that when the selected colors are R and B (FIGS. 13 and 14). Only the image shift amount shown on the horizontal axis of FIG. Therefore, for example, when a combination of R and B is adopted as a standard, when the selected color is R and G, or G and B, correction processing for doubling the image shift amount at the in-focus position is performed. Become.

  Based on the corrected phase difference, the system controller 30 determines whether or not the current driving position of the lens 10 is the in-focus position (step S8).

  Here, if it is determined that the lens is not in focus, the system controller 30 performs driving to bring the lens 10 closer to the in-focus position via the lens control unit 14 (step S9), and the process returns to step S2 and described above. Repeat the process.

  On the other hand, if it is determined in step S4 that the number of color images suitable for phase difference detection is less than three, the AF control unit 38 determines whether only two color images are suitable for phase difference detection. Is determined (step S10).

  If it is determined that only two color images are suitable for phase difference detection, the two color images determined to be compatible are selected as a set of color images in the above-described step S6. Process.

  If it is determined in step S10 that the number of color images suitable for phase difference detection is less than two colors, the AF control unit 38 controls to cause the contrast AF unit 43 to perform contrast AF and focus. (Step S11).

  In this way, when the process of step S11 ends or when it is determined in step S8 that the focus is achieved, the distance measurement process ends.

  Next, FIG. 16 is a flowchart showing a first example of the compatibility determination process for each color in the phase difference detection area in step S3 of FIG.

  When this process is started, a counter n for counting the number of colors is reset to 1 (step S21).

  Next, information on the nth color in the phase difference detection area (the first color information when this process is first performed) is read (step S22). However, the information to be read is the color information of the independent color pupil centroid position as described above, and may be used for phase difference detection (the above-described R, G1, G2, B). In the example of R, G1, B).

  Then, the maximum pixel value maxL that can be taken as the pixel value (for example, when the pixel value is expressed by 10 bits, that is, 0 to 1023, maxL = 1023) is a predetermined value k less than 1 (for example, k = 0.5). It is determined whether or not the number of pixels having a pixel value equal to or greater than the value multiplied by) is equal to or greater than a predetermined ratio (for example, 30%) of all the nth color pixels in the phase difference detection area (step S23).

  Here, the phase difference suitability determination unit 41 determines that the information of the nth color is suitable for phase difference detection when it is determined that the number of pixels having a pixel value equal to or greater than k × maxL is equal to or greater than a predetermined ratio. (Step S24) When it is determined that the ratio is less than the predetermined ratio, it is determined that the information of the nth color is incompatible with the phase difference detection (Step S25).

  When the processing of step S24 or step S25 is performed, it is determined whether or not the counter n has reached the number maxN of independent color pupil centroid positions as described above (step S26).

  If it is determined that maxN has not been reached, the counter n is incremented (step S27) and the process returns to step S22 to perform the processing described above for the next color information.

  If it is determined that the counter n has reached maxN, the process returns to the process shown in FIG. 15 from the compatibility determination process for each color in the phase difference detection area.

  FIG. 17 is a flowchart showing a second example of the compatibility determination process for each color in the phase difference detection area in step S3 of FIG.

ここに、Ｘi は位相差検出エリア内の第ｎ番目の色の画素の画素値、ＰＮは位相差検出エリア内の第ｎ番目の色の画素の全画素数である。 When this process is started, after performing the above-described processes of step S21 and step S22, the average pixel value AveL of the nth color information in the phase difference detection area is calculated as in the following equation (1): Step S31).
[Equation 1]

Here, Xi is the pixel value of the nth color pixel in the phase difference detection area, and PN is the total number of pixels of the nth color pixel in the phase difference detection area.

  Next, it is determined whether or not the calculated average pixel value AveL is within a predetermined range, that is, a predetermined lower limit value L1 or more and a predetermined upper limit value L2 or less (step S32). Here, an example of the predetermined lower limit value L1 is 0.3 × maxL, and an example of the predetermined upper limit value L2 is 0.7 × maxL (in the case of maxL = 1023, 0.3 × maxL = 307, 0.7 × maxL = 716) is, of course, not limited to these examples. This process is a process for confirming whether the nth color in the phase difference detection area is not over-exposed or under-exposed.

Here, when it is determined that the average pixel value AveL is equal to or larger than the predetermined lower limit value L1 and equal to or smaller than the predetermined upper limit value L2, the standard deviation σ of the pixel value of the nth color information is expressed by the following formula 2. (Step S33).
[Equation 2]

  Then, it is determined whether or not the calculated standard deviation σ is equal to or greater than a predetermined lower limit standard deviation σ1 (for example, σ1 = 200 when maxL = 1023) (step S34). In order to perform phase difference detection, the subject needs to have contrast, and even if the average pixel value is a moderate value, subjects with a flat contrast (for example, a blue sky with a uniform color) should perform phase difference detection. Difficult to do. Therefore, this process is a process for confirming whether the nth color information in the phase difference detection area relates to a flat subject.

  If it is determined that the standard deviation σ is greater than or equal to the predetermined lower limit standard deviation σ1, the process proceeds to step S24, and the nth color information is determined to be suitable for phase difference detection.

  On the other hand, when it is determined in step S32 that the average pixel value AveL is less than the predetermined lower limit value L1 or larger than the predetermined upper limit value L2, or in step S34, the standard deviation σ is less than the predetermined lower limit standard deviation σ1. If it is determined, the process goes to step S25 to determine that the nth color information is incompatible with the phase difference detection.

  Thereafter, the process of step S26 is performed, and the process proceeds to the process of step S27 according to the determination result of step S26, or the process returns to the process shown in FIG. 15 from the compatibility determination process of each color in the phase difference detection area.

  Next, FIG. 18 is a flowchart showing a third example of the compatibility determination process for each color in the phase difference detection area in step S3 of FIG.

  When this process is started, the processes of steps S21, S22, S31, and S32 described above are performed.

  If it is determined in step S32 that the average pixel value AveL is not less than the predetermined lower limit value L1 and not more than the predetermined upper limit value L2, the edge amount V of the nth color in the phase difference detection area is detected. (Step S41).

Here, the detection of the edge amount V can be performed using various filters, but a Sobel filter is given as an example. For example, if a 3 × 3 filter represented by the following formula 3 is used, an edge in the vertical direction can be detected.
[Equation 3]

  For example, the filter shown in Equation 3 is sequentially applied to the nth color pixel in the phase difference detection area while shifting the center pixel position one pixel at a time, and the result is added to obtain the result in the phase difference detection area. The edge amount V of the nth color is calculated.

  Next, whether or not the absolute value | V | of the detected edge amount V is equal to or greater than a predetermined lower limit edge amount | V1 | (for example, | V1 | = 200 when maxL = 1023). Determination is made (step S42).

  Here, if it is determined that the edge amount | V | is equal to or larger than the predetermined lower limit edge amount | V1 |, there is an edge suitable for phase difference detection in the phase difference detection area. Thus, the nth color information is determined to be suitable for phase difference detection.

  On the other hand, when it is determined in step S32 that the average pixel value AveL is less than the predetermined lower limit value L1 or larger than the predetermined upper limit value L2, or in step S42, the edge amount | V | is the predetermined lower limit edge amount | V1. If it is determined that it is less than |, the process goes to step S25 to determine that the nth color information is incompatible with the phase difference detection.

  Thereafter, the process of step S26 is performed, and the process proceeds to the process of step S27 according to the determination result of step S26, or the process returns to the process shown in FIG. 15 from the compatibility determination process of each color in the phase difference detection area.

  FIG. 19 is a flowchart showing a fourth example of the compatibility determination process for each color in the phase difference detection area in step S3 of FIG.

  When this process is started, the processes of steps S21, S22, S31, and S32 described above are performed.

  If it is determined in step S32 that the average pixel value AveL is not less than the predetermined lower limit value L1 and not more than the predetermined upper limit value L2, frequency analysis such as Fourier analysis is performed (step S51).

  Next, based on the result of the frequency analysis, it is determined whether or not a predetermined frequency component (for example, an appropriate high-frequency component related to the pattern or contour of the subject) is greater than or equal to a predetermined value (step S52).

  Here, if it is determined that the predetermined frequency component is greater than or equal to the predetermined value, the process goes to step S24 to determine that the nth color information is suitable for phase difference detection.

  On the other hand, when it is determined in step S32 that the average pixel value AveL is less than the predetermined lower limit value L1 or larger than the predetermined upper limit value L2, or in step S52, it is determined that the predetermined frequency component is less than the predetermined value. If YES in step S25, the flow advances to step S25 to determine that the nth color information is incompatible with phase difference detection.

  Thereafter, the process of step S26 is performed, and the process proceeds to the process of step S27 according to the determination result of step S26, or the process returns to the process shown in FIG. 15 from the compatibility determination process of each color in the phase difference detection area.

  In addition, with reference to FIG. 16 to FIG. 19, some examples of the compatibility determination process of each color in the phase difference detection area have been described. However, the present invention is not limited thereto, and other processes may be used.

  Next, FIG. 20 is a flowchart showing a first example of the process of selecting a set of compatible colors in step S5 of FIG.

  When this processing is started, the color signal selection unit 51 calculates the average pixel value of each compatible color determined to be suitable for phase difference detection in the phase difference detection area using the above-described Equation 1 (step S61). . However, when the process of step S3 is performed according to any of FIGS. 17 to 19, since the calculation result already exists, the calculation result may be used.

  Next, the color signal selection unit 51 uses the upper two color images with the average pixel value (a combination of the color image with the largest average pixel value and the second largest color image) as one set of color images for phase difference detection. Is selected (step S62), and the process returns to the process shown in FIG.

  Next, FIG. 21 is a flowchart showing a second example of the process of selecting a suitable color set in step S5 of FIG.

  When this process is started, the color signal selection unit 51 selects the two colors having the most distant center of gravity when passing through the pupil region from among the compatible colors determined to be suitable for the phase difference detection in the phase difference detection area. Are selected as a set of colors used for phase difference detection (step S71), and the process returns to the process shown in FIG.

  20 and 21 have been described with reference to some examples of the process of selecting a set of compatible colors, but the present invention is not limited to these, and other processes may be used. For example, the color signal selection unit 51 may select two colors most suitable for phase difference detection based on the determination result of the phase difference matching determination unit 41.

  Specifically, when the phase difference matching determination unit 41 performs the processing shown in FIG. 16, it is conceivable to select two colors in which pixels having pixel values equal to or larger than k × maxL exist at the highest ratio.

  When the phase difference matching determination unit 41 performs the process shown in FIG. 17, two colors having the largest standard deviation σ are selected from among colors having an average pixel value AveL that is not less than the lower limit value L1 and not more than the upper limit value L2. It is possible to choose.

  Further, when the phase difference matching determination unit 41 performs the processing shown in FIG. 18, the edge amount | V | is the largest among the colors having the average pixel value AveL that is not less than the lower limit value L1 and not more than the upper limit value L2. It is conceivable to select a color.

  In addition, when the phase difference matching determination unit 41 performs the process shown in FIG. 19, the predetermined frequency component having the largest predetermined frequency component is selected from the colors having the average pixel value AveL that is not less than the lower limit value L1 and not more than the upper limit value L2. It is conceivable to select a color.

  According to the first embodiment, the two bands most suitable for phase difference detection are selected from the three or more transmission bands configured in the band limiting filter 12, and the colors of the selected two bands are selected. Since the phase difference detection is performed using the image, it becomes possible to acquire more reliable distance information based on the image obtained from the image sensor.

  When two or more colors suitable for phase difference detection are not detected, contrast AF is performed, so that the subject distance can be measured even for a subject that is not suitable for phase difference detection.

  Whether or not the color image is suitable for phase difference detection is determined by determining whether or not pixels having a pixel value equal to or greater than the maximum pixel value maxL that can be taken as the pixel value multiplied by a predetermined value k less than 1 are all in the phase difference detection area. When the determination is made based on whether or not the predetermined number of pixels of the nth color is present, there is an advantage that the processing is simple.

  Further, when determining whether or not a color image is suitable for phase difference detection based on the average pixel value and standard deviation, it is possible to easily exclude a flat subject having a small change in pixel value from a phase difference detection target. It becomes possible.

  Furthermore, when determining whether or not a color image is suitable for phase difference detection based on the average pixel value and the edge amount, it is possible to easily exclude a subject having few edges from the phase difference detection target.

  When determining whether or not a color image is suitable for phase difference detection based on the average pixel value and frequency component, it is possible to easily exclude a subject with few high-frequency components from the phase difference detection target. Become.

  In addition, when there are three or more compatible colors, the top two colors of the average pixel value are used for phase difference detection, so that reliable phase difference detection is performed based on the two brightest color images. Can do.

In addition, when there are three or more compatible colors, the two colors with the most distant center of gravity when passing through the pupil region are used for phase difference detection, so that the most accurate phase difference detection can be performed with high reliability. It can be carried out.
[Embodiment 2]

  22 and FIG. 23 show the second embodiment of the present invention. FIG. 22 is a flowchart showing the distance measuring process in the imaging apparatus, and FIG. 23 is a block diagram showing the configuration of the phase difference AF unit 42. In the second embodiment, parts that are the same as those in the first embodiment are given the same reference numerals and description thereof is omitted, and only differences are mainly described.

  In the above-described first embodiment, when there are three or more compatible colors, two colors most suitable for phase difference detection are selected from them, a correlation value graph of the selected two colors is created, and the created correlation The phase difference is detected based on the value graph. On the other hand, the second embodiment creates a correlation value graph of all the matching color combinations, and performs more reliable phase difference detection based on the correlation value graph giving the highest correlation value. ing.

  Therefore, the phase difference AF unit 42 of the present embodiment is one in which the color signal selection unit 51 is omitted as shown in FIG.

  Next, distance measurement processing in the imaging apparatus will be described with reference to FIG.

  When this process is started, the above-described steps S1 to S4 are performed.

  If it is determined in step S4 that the image of three or more colors is suitable for phase difference detection, the AF control unit 38 determines the combination of all the color images determined to be compatible in step S3. The phase difference AF unit 42 is caused to create a correlation value graph (step S81). If the number of matching colors is n (n is an integer of 3 or more), the number of correlation value graphs created here is nC2. In the following, a case where the ZNCC method as shown in FIG. 14 is adopted, that is, a case where the correlation value is higher as the correlation value is higher will be described as an example.

  In the case of the ZNCC method, a peak, that is, a maximum correlation value exists in each of the nC2 correlation value graphs. Therefore, under the control of the AF control unit 38, the phase difference AF unit 42 selects a correlation value graph that gives the largest correlation value among the maximum values of these nC2 correlation values as a correlation value graph used for phase difference detection. (Step S82).

  Then, under the control of the AF control unit 38, the phase difference calculation unit 53 acquires the phase difference (and consequently the subject distance) based on the image shift amount that gives the maximum correlation value in the selected correlation value graph (step S83).

  On the other hand, if it is determined in step S4 that the number of color images suitable for phase difference detection is less than three, the process in step S10 described above is performed, and the AF control unit 38 performs only two color images. Determines whether or not is suitable for phase difference detection.

  If it is determined that only two color images are suitable for phase difference detection, the phase difference calculation unit 53 is controlled by the AF control unit 38 to detect the phase difference between the two color images determined to be compatible. (As a result, the subject distance) is acquired (step S84).

  If it is determined in step S10 that the number of color images suitable for phase difference detection is less than two, the AF control unit 38 determines whether only one color image is suitable for phase difference detection. Is determined (step S85).

  Here, when it is determined that only one color image is suitable for phase difference detection, the phase difference calculation unit 53 controls the correlation value graph between the compatible color and the non-conforming color under the control of the AF control unit 38. Is created (step S86). When the number of independent color pupil centroid positions as described above is maxN, the number of correlation value graphs created here is (maxN-1).

  Next, the AF control unit 38 acquires the largest correlation value from the maximum correlation values existing in each of the created (maxN−1) correlation value graphs (step S87).

  Whether or not the AF control unit 38 can detect the phase difference based on whether or not the acquired maximum correlation value is equal to or greater than a predetermined value (for example, 0.7 or 0.8 in the ZNCC method). Is determined (step S88).

  Here, if it is determined that the phase difference can be detected, the phase difference AF unit 42 uses the correlation value graph that gives the maximum correlation value under the control of the AF control unit 38 as the correlation value graph used for the phase difference detection. (Step S89), and based on the image shift amount giving the maximum correlation value in the selected correlation value graph, the phase difference calculation unit 53 obtains the phase difference (and hence the subject distance) under the control of the AF control unit 38. (Step S90).

  When the phase difference (and thus the subject distance) is acquired by any of the processes of steps S83, S84, or S90, the process of step S7 described above is performed, and the reference correction unit 52 is acquired by the control of the AF control unit 38. The phase difference thus corrected is corrected in accordance with the center-of-gravity distance when the light related to the selected two colors passes through the pupil region of the lens 10.

  Furthermore, in-focus determination is performed in step S8, and if it is determined that it is not in focus, the process proceeds to step S9.

  If it is determined in step S85 that there is no color image suitable for phase difference detection, or if it is determined in step S88 that phase difference detection is not possible, the process proceeds to contrast AF in step S11.

  In this way, the process of step S11 ends, or if it is determined in step S8 that the in-focus state is obtained, the distance measurement process ends.

According to the second embodiment, the effects similar to those of the first embodiment described above except for the effect related to the selection of the matching color set are obtained, and the correlation value graph of all the matching color combinations is created. Since the correlation value graph that gives the highest correlation value is selected and phase difference detection is performed based on the selected correlation value graph, phase difference detection is performed with high actual correlation, that is, high reliability. Can do.
[Embodiment 3]

  FIGS. 24 to 30 show the third embodiment of the present invention, FIG. 24 is a flowchart showing distance measurement processing in the imaging apparatus, and FIG. 25 is a diagram showing an example of a correlation value graph of an R image and a B image. 26 is a diagram illustrating an example of a correlation value graph between an R image and a G image, FIG. 27 is a diagram illustrating an example of a correlation value graph between a G image and a B image, and FIG. 28 is a correlation between the R image and the B image on the horizontal axis. FIG. 29 is a diagram illustrating an example of a corrected correlation value graph of an R image and a G image that has been rescaled according to a value graph, and FIG. 29 illustrates a G image and a B image that are rescaled according to the correlation value graph of an R image and a B image. FIG. 30 is a diagram illustrating an example of the corrected correlation value graph of FIG. 30. FIG. 30 illustrates a correlation value graph of the R image and the B image, a corrected correlation value graph of the R image and the G image, and a corrected correlation value graph of the G image and the B image. It is a figure which shows the example of the addition correlation value graph obtained by adding.

  In the third embodiment, parts that are the same as those in the first and second embodiments are given the same reference numerals, description thereof is omitted, and only differences are mainly described.

  In the first and second embodiments, when there are three or more compatible colors, the distance to the subject is calculated based on the correlation value of the combination of two color images selected from the three or more color images. (There is a difference between selecting two colors and then creating a correlation value graph, or creating a correlation value graph and then substantially selecting two colors.) Two colors suitable for phase difference detection are substantially selected. On the other hand, in the third embodiment, the distance to the subject is calculated based on the correlation value of the combination of all two color images of three or more color images. It is used in an integrated manner (by creating correlation value graphs for all combinations of matching colors and integrating the created correlation value graphs) to perform more reliable phase difference detection.

  Therefore, in the phase difference AF unit 42 of the present embodiment, the color signal selection unit 51 is omitted as in the case of the above-described second embodiment shown in FIG.

  Further, the reference correction unit 52 of the first and second embodiments described above corrects the phase difference amount or the subject distance according to the center-of-gravity distance, but the reference correction unit 52 of the present embodiment will be described later. Furthermore, the correlation value graph is rescaled (that is, corrected so as to have the same scale) in the horizontal direction (the direction of the image shift amount axis) according to the center-of-gravity distance.

  Next, distance measurement processing in the imaging apparatus will be described with reference to FIG.

  When this process is started, the above-described steps S1 to S4 are performed.

  If it is determined in step S4 that an image of three or more colors is suitable for phase difference detection, the process of step S81 described above is performed, and the number of compatible colors is n (n is an integer of 3 or more). Then, nC2 correlation value graphs are created. When the band limiting filter 12 as shown in FIG. 4 is used, the correlation value graph between the R image and the B image as shown in FIG. 25 and the correlation between the R image and the G image as shown in FIG. It is assumed that a value graph and a correlation value graph between the G image and the B image as shown in FIG. 27 are obtained.

  In the band limiting filter 12 as shown in FIG. 4, the centroid distance Drg and the centroid distance Dgb are half of the centroid distance Drb. Therefore, in the RG correlation value graph shown in FIG. 26 and the GB correlation value graph shown in FIG. 27, the scale in the horizontal direction (the direction of the image shift amount axis) is half that of the RB correlation value graph shown in FIG.

  Therefore, for example, based on the RB correlation value graph, the reference correction unit 52 controls the RG correlation value graph shown in FIG. 26 and the GB correlation value graph shown in FIG. Rescaling in the direction (direction of the image shift amount axis) (step S101). By this rescaling, it is assumed that, for example, an RG correction correlation value graph as shown in FIG. 28 and a GB correction correlation value graph as shown in FIG. 29 are obtained (the rescaled correlations in FIGS. 28 and 29). The value graphs RG and GB are represented as RG ′ and GB ′).

  Next, under the control of the AF control unit 38, the phase difference AF unit 42 generates an RB correlation value graph shown in FIG. 25, an RG correction correlation value graph shown in FIG. 28, and a GB correction correlation value graph shown in FIG. Addition is performed to create an addition correlation value graph as shown in FIG. 30 as an integrated correlation value graph (step S102).

  Then, based on the created addition correlation value graph, the phase difference calculation unit 53 acquires the phase difference (and hence the subject distance) under the control of the AF control unit 38 (step S103).

  On the other hand, if it is determined in step S4 that the number of color images suitable for phase difference detection is less than three, the process in step S10 described above is performed, and the AF control unit 38 performs only two color images. Determines whether or not is suitable for phase difference detection.

  Here, when it is determined that only two color images are suitable for phase difference detection, the phase difference calculation unit 53 performs the process of step S84 described above under the control of the AF control unit 38 and determines that it is suitable. The phase difference (and consequently the subject distance) between the two color images thus obtained is acquired, the process of step S7 described above is performed, and the reference correction unit 52 controls the AF control unit 38 to obtain the acquired phase difference. Correction is performed in accordance with the center-of-gravity distance when the light of the selected two colors passes through the pupil region of the lens 10.

  If it is determined in step S10 that the number of color images suitable for phase difference detection is less than two, the process in step S85 described above is performed, and the AF control unit 38 performs only one color image. Determines whether or not is suitable for phase difference detection.

  If it is determined that only one color image is suitable for phase difference detection, the process of step S86 described above is performed, and the phase difference calculation unit 53 is controlled by the AF control unit 38 to match the matching color. Create a correlation graph between each non-conforming color. When the number of independent color pupil centroid positions as described above is maxN, the number of correlation value graphs created here is (maxN-1).

  Next, the reference correction unit 52 controls the generated (maxN−1) correlation value graphs in the horizontal direction (in the horizontal direction (according to the centroid distance) in the same manner as in step S101 described above. Rescaling is performed in the direction of the image displacement amount axis (step S104).

  Further, under the control of the AF control unit 38, the phase difference AF unit 42 adds (maxN−1) correlation value graphs after rescaling in the same manner as in the process of step S102 described above, thereby obtaining an added correlation value graph. An integrated correlation value graph is created (step S105).

  Then, the AF control unit 38 acquires the maximum correlation value from the created addition correlation value graph (step S106).

  Subsequently, the acquired maximum correlation value is equal to or greater than a predetermined value (for example, 0.7 × (maxN−1) or 0.8 × (maxN−1) in the ZNCC method) as in the process of step S88 described above. Based on whether or not there is, the AF control unit 38 determines whether or not the phase difference can be detected. However, here, unlike the process of step S88 of FIG. 22, the difference is that the predetermined value to be compared with the maximum correlation value is multiplied by [(maxN-1) times] the correlation value graph.

  If it is determined that the phase difference can be detected, the phase difference calculation unit 53 acquires the phase difference based on the addition correlation value graph under the control of the AF control unit 38 (step S107).

  When the phase difference (and hence the subject distance) is acquired by any one of the processes in steps S103, S7, or S107, the focus determination is performed in step S8. If it is determined that the subject is not in focus, the process proceeds to step S9. Proceed to processing.

  If it is determined in step S85 that there is no color image suitable for phase difference detection, or if it is determined in step S88 that phase difference detection is not possible, the process proceeds to contrast AF in step S11.

  In this way, the process of step S11 ends, or if it is determined in step S8 that the in-focus state is obtained, the distance measurement process ends.

  In the above description, the phase difference detection target is the addition correlation value graph obtained by adding all the correlation value graphs, but the present invention is not limited to this. For example, an average correlation value graph obtained by averaging all correlation value graphs may be created as an integrated correlation value graph, and phase difference detection may be performed based on the average correlation value graph. Further, all the correlation value graphs are statistically processed (for example, correlation value graphs with greatly different values are excluded, or correlation value graphs with high value similarity are increased in weight, etc.) to provide one integrated correlation. A value graph may be created and phase difference detection may be performed based on the integrated correlation value graph.

  According to the third embodiment as described above, effects similar to those of the first and second embodiments described above except for the effect related to the selection of the matching color set are obtained, and a correlation value graph of all the matching color combinations is created. Then, the correlation value graph created is rescaled and then added to create an integrated correlation value graph, and phase difference detection is performed based on the created integrated correlation value graph. Therefore, highly reliable phase difference detection can be performed.

In the above description, the imaging apparatus including the distance measuring device has been mainly described. However, the distance measuring method for performing the above-described processing, the distance measuring program for performing the above-described processing, and the computer that records the distance measuring program. It may be a recording medium readable by the above.
Further, although only an example in which distance measurement is used for autofocusing is shown, a distance measurement result by this distance measurement method may be used for image processing, a stereoscopic image, and the like.

  Further, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1 ... Lens unit 2 ... Body unit 9 ... Imaging optical system 10 ... Lens (a focus lens is included)
DESCRIPTION OF SYMBOLS 11 ... Diaphragm 12 ... Band-limiting filter 12b ... GB filter 12r ... RG filter 14 ... Lens control part 15 ... Lens side communication connector 21 ... Shutter 22 ... Image sensor (color image sensor)
DESCRIPTION OF SYMBOLS 23 ... Imaging circuit 24 ... Imaging drive part 25 ... Image processing part 26 ... Image memory 27 ... Display part 28 ... Interface (IF)
29 ... Recording medium 30 ... System controller (control unit)
DESCRIPTION OF SYMBOLS 31 ... Sensor part 32 ... Operation part 33 ... Strobe control circuit 34 ... Strobe 35 ... Body side communication connector 36 ... Intercolor correction part 37 ... Color image generation part 38 ... AF control part (phase difference calculation control part)
39 ... Distance calculation unit 41 ... Phase difference suitability determination unit 42 ... Phase difference AF unit (phase difference calculation unit)
43 ... Contrast AF unit 51 ... Color signal selection unit 52 ... Reference correction unit 53 ... Phase difference calculation unit

Claims (17)

An image sensor that receives and photoelectrically converts light in three or more wavelength bands, and generates a color image for each wavelength band; and
An imaging optical system that forms light of the wavelength band as a subject image on the imaging device;
A band limiting filter that varies the center of gravity of the amount of light when passing through the pupil region of the imaging optical system to three or more,
A phase difference calculation unit that calculates a distance to a subject based on a phase difference amount caused by a difference in the center of gravity of the plurality of color images obtained from the image sensor;
A phase difference suitability determining unit that determines whether or not the color image obtained from the image sensor is suitable as a color image used by the phase difference calculating unit;
Two color images selected from the three or more color images in the phase difference calculation unit when the number of color images determined to be suitable by the phase difference matching determination unit is three or more. A phase difference calculation control unit that calculates a distance to the subject based on a correlation value of a combination of the above or a correlation value of all combinations of two color images among the three or more color images;
A distance measuring device comprising: A color signal selection unit that selects a combination of two color images from the color images determined to be suitable by the phase difference matching determination unit;
The phase difference calculation control unit causes the phase difference calculation unit to calculate a distance to the subject based on a correlation value of a combination of two color images selected by the color signal selection unit. The distance measuring apparatus according to 1.   The phase difference calculation control unit has a maximum of all combinations of two color images in the color images determined to be suitable for the phase difference calculation by the phase difference matching determination unit. 2. The distance to the subject is calculated based on a combination of two color images that respectively calculate correlation values and give a maximum correlation value among the plurality of maximum correlation values. Distance measuring device.   The phase difference calculation control unit is suitable for the phase difference calculation in the phase difference calculation unit when only one color image is determined to be suitable for the phase difference calculation by the phase difference matching determination unit. A maximum correlation value is calculated for each combination of the color image determined to be suitable and the color image not determined suitable for the phase difference calculation, and the maximum correlation among the plurality of maximum correlation values is calculated. 4. The distance measuring apparatus according to claim 3, wherein a distance to the subject is calculated based on a combination of two color images giving values.   The phase difference amount calculated by the phase difference calculation unit or the distance to the subject is the centroid between the centroid positions of the light amounts when the light related to the two color images subjected to the phase difference calculation passes through the pupil region. The distance measuring device according to any one of claims 2 to 4, further comprising a reference correction unit configured to correct according to the distance. The phase difference calculation unit creates a correlation value graph indicating a correlation value with respect to an image shift amount of a combination of two color images.
The correlation value graph created by the phase difference calculation unit is used as the centroid distance between the centroid positions of the light amounts when the light related to the two color images for which the correlation value graph is created passes through the pupil region. Accordingly, a reference correction unit that corrects in the direction of the image shift amount axis is further provided,
The phase difference calculation control unit is
Causing the phase difference calculation unit to create the correlation value graph for all combinations of two color images among all color images determined to be suitable by the phase difference matching determination unit;
Let the reference correction unit correct all the created correlation value graphs to the same scale as the axial direction of the image shift amount,
2. The phase difference calculation unit is configured to create an integrated correlation value graph by integrating all the correlation value graphs, and to calculate a distance to the subject based on the integrated correlation value graph. The described distance measuring device.   The phase difference calculation control unit determines that the phase difference calculation unit is suitable for the phase difference calculation when there is one color image determined to be suitable by the phase difference matching determination unit. The distance measuring apparatus according to claim 6, wherein the correlation value graph is created for all combinations of a color image and a color image that has not been determined to be suitable for the phase difference calculation.   When there are two color images determined to be suitable for the phase difference calculation by the phase difference suitability determination unit, the phase difference calculation control unit causes the phase difference calculation unit to store the two color images. The distance measuring apparatus according to claim 1, wherein a distance to the subject is calculated based on a combination correlation value. A contrast autofocus unit for performing contrast autofocus based on an image obtained from the image sensor;
The phase difference calculation control unit causes the contrast autofocus unit to calculate a distance to the subject when there is no color image determined to be suitable by the phase difference matching determination unit. The distance measuring device according to claim 8.   The phase difference matching determination unit determines whether or not the color image is suitable as a color image to be used for the calculation, and has a pixel value equal to or larger than a value obtained by multiplying a maximum pixel value that can be taken as a pixel value by a predetermined value less than 1. The distance measuring device according to claim 1, wherein the determination is based on whether the color image is greater than or equal to a predetermined ratio.   The phase difference conformity determining unit determines whether the color image is suitable as a color image used for the calculation, an average pixel value of the color image is within a predetermined range, and a standard deviation of the pixel values of the color image The distance measuring device according to claim 1, wherein the determination is based on whether or not is greater than or equal to a predetermined lower standard deviation.   The phase difference conformity determination unit determines whether the color image is suitable as a color image used for the calculation, an average pixel value of the color image is within a predetermined range, and an absolute value of an edge amount of the color image The distance measuring device according to claim 1, wherein the distance is determined based on whether or not is equal to or greater than a predetermined lower limit edge amount.   The phase difference conformity determination unit determines whether the color image is suitable as a color image used for the calculation, whether an average pixel value of the color image is within a predetermined range, and a predetermined frequency component of the color image is The distance measuring device according to claim 1, wherein the determination is based on whether or not the predetermined value or more exists.   The color signal selection unit selects a combination of the color image having the largest average pixel value and the second largest color image from the color images determined to be suitable by the phase difference matching determination unit. The distance measuring device according to claim 2, wherein   The color signal selection unit selects a combination of two color images having the most distant center of gravity when passing through the pupil region from among color images determined to be suitable by the phase difference matching determination unit. The distance measuring device according to claim 2, wherein the distance measuring device is selected.   An imaging apparatus comprising the distance measuring device according to any one of claims 1 to 15. Dividing the light into three or more wavelength bands by changing the center of gravity of the amount of light when passing through the pupil region to three or more,
Form the divided light for each wavelength band as a subject image,
Receiving light for each wavelength band, photoelectrically converting, generating a color image for each wavelength band,
Determining whether the color image is suitable as a color image used for phase difference calculation;
When the number of the color images determined to be suitable for the phase difference calculation is three or more, a correlation value of a combination of two color images selected from the three or more color images, or the A distance measuring method comprising: calculating a distance to a subject based on correlation values of all combinations of two color images among three or more color images.

Images