JP2014038151A - Imaging apparatus and phase difference detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus and a phase difference detection method that can adaptively change a correlation width in accordance with a focusing degree.SOLUTION: The imaging apparatus includes: an imaging element; a phase difference image generation part 20; and a phase difference detection part 82. The imaging element images a first subject image and a second subject image having parallax with respect to the first subject image. The phase difference image generation part 20 generates a first image Icorresponding to the first subject image and a second image Icorresponding to the second subject image on the basis of the picked-up image. The phase difference detection part 82 performs correction calculation while relatively shifting the first image Iand the second image I, and detects a phase difference between the first image Iand the second image I. The phase difference detection part 82 sets a correlation width being an area width for performing a product sum of the correlation calculation in accordance with a shift amount of the shift.

Description

  The present invention relates to an imaging device, a phase difference detection method, and the like.

  Conventionally, a method of detecting a phase difference between two images having parallax by correlation calculation is known. As such a method, for example, the methods described in Patent Documents 1 and 2 are known.

  In Patent Document 1, a predetermined size for performing stereo matching is set, a correlation value is obtained with the set size, and the size for performing stereo matching is reset based on the obtained correlation value, and this is repeated to obtain a final value. A method is disclosed in which correlation calculation is performed by adaptively setting a correlation width by performing correlation calculation with an automatically set size.

  In Patent Document 2, a reference parallax is set, a search range for stereo matching is specified based on an image point that gives the reference parallax, and a stereo matching is performed in the search range, thereby narrowing down a search range for performing correlation calculation. A technique is disclosed.

JP 2007-172500 A JP 2010-128622 A

  The method of detecting the phase difference by the correlation calculation as described above is used, for example, for phase difference AF (Auto-Focus) for performing focus control using the detected phase difference, or for three-dimensional measurement for obtaining the distance to the subject. ing.

  However, in a general image, a focused part and a non-focused part coexist, so that the correlation width (the width of product sum in correlation calculation) is adaptively changed according to the degree of focus. There is a problem that it is necessary. For example, if the correlation width is not adaptively changed, the calculation for obtaining the correlation peak with a predetermined correlation width is repeated while sequentially changing the correlation width, so that the amount of calculation increases. In the above-mentioned Patent Document 1, the correlation width is adaptively set. However, it is necessary to perform the correlation width setting process before the correlation calculation, and there is a problem that the calculation amount increases accordingly.

  According to some aspects of the present invention, it is possible to provide an imaging device, a phase difference detection method, and the like that can adaptively change the correlation width in accordance with the degree of focus.

  According to one aspect of the present invention, an imaging element that captures a first subject image and a second subject image having a parallax with respect to the first subject image and obtains the captured image, and the first image based on the captured image. A phase difference image generation unit that generates a first image corresponding to one subject image and a second image corresponding to the second subject image; and a correlation calculation while relatively shifting the first image and the second image A phase difference detection unit that detects a phase difference between the first image and the second image, and the phase difference detection unit calculates a correlation width that is a region width for performing a product-sum of the correlation calculation, The present invention relates to an imaging device that is set according to the shift amount of the first image and the second image in the correlation calculation.

  According to one aspect of the present invention, a first subject image and a second subject image having parallax are captured, and a first image corresponding to the first subject image and a second image corresponding to the second subject image are captured images. The correlation width is set according to the shift amount in the correlation calculation, the correlation calculation is performed with the correlation width according to the shift amount, and the phase difference between the first image and the second image is detected. Thereby, the correlation width can be adaptively changed according to the degree of focus.

  Also, in one aspect of the present invention, the phase difference detection unit obtains a correlation value by repeatedly performing the correlation calculation while sequentially increasing the shift amount and sequentially increasing the correlation width, and the correlation value is set to a maximum value. The shift amount at the time may be detected as the phase difference.

  In the aspect of the invention, the phase difference detection unit may calculate the shift amount corresponding to the first peak value of the correlation value when the correlation value is obtained while sequentially increasing the shift amount. You may detect as.

  In the aspect of the invention, when the phase difference detection unit determines that the first peak value is detected, the shift larger than the shift amount when it is determined that the first peak value is detected. The correlation calculation may be terminated without obtaining the correlation value for the quantity.

  In the aspect of the invention, the phase difference detection unit sets an angular frequency corresponding to the correlation width when the phase difference is detected, and sets a Fourier coefficient of a pixel value at the angular frequency as the first image. The shift amount when the difference between the declination angle of the Fourier coefficient for the first image and the declination angle of the Fourier coefficient for the second image is minimized is calculated for the second image. You may detect as a phase difference.

  Moreover, in one aspect of the present invention, the phase difference detection unit compares the pixel value of the captured image with the pixel value of a comparison image that is at least one of the first image and the second image, A focusing direction of the imaging optical system may be determined, and a shift direction for shifting the first image and the second image in the correlation calculation may be set based on the determined focusing direction.

  In one aspect of the present invention, a multiband estimation unit, and an optical filter that divides a pupil of an imaging optical system into a first pupil and a second pupil having a transmission wavelength band different from that of the first pupil are provided. The imaging device includes a first color filter having a first transmittance characteristic, a second color filter having a second transmittance characteristic, and a third color filter having a third transmittance characteristic, The band estimation unit sets first to fifth bands corresponding to the overlapping portion and the non-overlapping portion of the first to third transmittance characteristics, and sets the first to third color pixel values constituting the captured image. The first to fifth band component values are estimated based on the phase difference image generation unit, and the phase difference image generation unit calculates the band component values corresponding to the transmission wavelength band of the first pupil among the first to fifth bands. Acquired as a first image and transmitted through the second pupil of the first to fifth bands The component values of the bands corresponding to the bands may be acquired as the second image.

  According to another aspect of the present invention, a first subject image and a second subject image having a parallax with respect to the first subject image are captured to obtain a captured image, and the first subject image is obtained based on the captured image. A first image corresponding to one subject image and a second image corresponding to the second subject image are generated, and a correlation width, which is a region width for performing a product sum of correlation operations, is defined as the first image in the correlation operations. The present invention relates to a phase difference detection method that is set according to the shift amount of the second image and detects a phase difference between the first image and the second image by the correlation calculation.

2 is a basic configuration example of an imaging optical system. Explanatory drawing about the phase difference detection method of this embodiment. A characteristic example of the correlation width with respect to the shift amount and a characteristic example of the correlation value with respect to the shift amount FIG. 4A and FIG. 4B are explanatory diagrams showing the principle of the phase difference detection method of the present embodiment. FIG. 3 is a principle explanatory diagram of a phase difference detection method of the present embodiment. Explanatory drawing about the phase difference detection method of the sub pixel of this embodiment. Explanatory drawing about a declination difference. Explanatory drawing about the method of detecting the phase difference of a sub pixel from a declination difference. 1 is a configuration example of an imaging apparatus according to a first embodiment. The flowchart of a phase difference detection process. The flowchart of the phase difference detection process in the case of performing the phase difference detection of a sub pixel. FIG. 12A and FIG. 12B are explanatory diagrams of the focusing direction determination method in the second embodiment. The structural example of the imaging device in 3rd Embodiment. Explanatory drawing about a band division | segmentation method. FIGS. 15A to 15F are explanatory diagrams of multiband estimation processing. Explanatory drawing about the unknown number estimation process in a multiband estimation process. FIGS. 17A to 17F are explanatory diagrams of multiband estimation processing. Explanatory drawing about the unknown number estimation process in a multiband estimation process.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Outline of this Embodiment As a device for capturing a phase difference image (parallax image), for example, a stereo image capturing device using a twin-lens camera, a single-lens camera using a pupil division method, and the like are known. For example, three-dimensional image generation, three-dimensional shape measurement, high-speed phase difference AF, and the like are realized by the phase difference image captured by such an apparatus. Particularly in the case of three-dimensional shape measurement and phase difference type high-speed AF, the phase difference (deviation amount of two phase difference images) at a specific position of the phase difference image is detected, and the optical axis direction is detected using the principle of triangulation. It is common to obtain distance information. In order to detect such a phase difference, a corresponding point is found by correlation calculation from an image (pixel value pattern) in the vicinity of the position where the phase difference is to be detected, and a deviation amount (that is, a phase difference) between the corresponding points is determined. It is normal to seek.

  However, in the phase difference image obtained by the imaging device, the focused area and the blurred area coexist, and areas with various degrees of focus are mixed. For example, in the case where the phase difference is obtained in all regions of the phase difference image, such as in three-dimensional shape measurement, it is necessary to detect the phase difference in regions having various degrees of focus. Therefore, it is necessary to adaptively change the width of product sum in the correlation calculation (hereinafter referred to as “correlation width” as appropriate) in accordance with the degree of focusing. For example, in an image in a focused area, if the correlation width is large, a true phase difference cannot be obtained due to the occurrence of a plurality of correlation peaks. Therefore, it is necessary to set the correlation width small. On the other hand, in an image of a blurred area, if the correlation width is set to be small, no correlation peak occurs, and a true phase difference cannot be obtained. Therefore, it is necessary to set the correlation width to be large.

  As a correlation calculation method when an appropriate correlation width is not known in this manner, for example, while changing the correlation width, a plurality of product-sum calculations are performed by changing the shift amount of the phase difference image for each correlation width. There is a technique for obtaining a correlation peak for each correlation width and obtaining a maximum value from the plurality of correlation peaks. However, this method has a problem that the number of product-sum operations is enormous. In particular, when the correlation calculation is applied to all regions of the phase difference image, it is impractical in terms of speeding up the calculation.

  For example, in Patent Document 1 described above, although the correlation width can be set adaptively, it is necessary to perform a correlation width setting process before the correlation calculation, which increases the amount of calculation. In Patent Document 2 described above, the calculation speed is increased by narrowing the search range for performing the correlation calculation, but the correlation width cannot be set adaptively.

Therefore, in the present embodiment, as shown in FIG. 3 and the like, the shift amount δ for bringing the two phase difference images closer in the correlation calculation is sequentially increased, and the correlation width W is increased in accordance with the shift amount δ. . Then, the shift amount δ _k that maximizes the correlation value is detected as a phase difference.

When the shift amount is δ _k, the correlation width is W _k , and the shift amount δ _k corresponds to the degree of focus at the detection target position. Therefore, in the present embodiment, the correlation width W _k adaptively changes with the degree of focus, and the phase difference can be detected with an appropriate correlation width W _k . Further, in this embodiment, since the phase difference can be detected only by obtaining the correlation peak once, the number of product-sum operations is significantly reduced compared to the method of obtaining the correlation peak for each correlation width. Can do. In addition, since the correlation width W changes with the shift amount δ in the correlation calculation, it is not necessary to perform a process of determining the correlation width W before the correlation calculation, and the correlation calculation can be speeded up.

2. First embodiment 2.1. Basic Configuration of Imaging Optical System Next, the phase difference detection method in the present embodiment will be described in detail. First, the first embodiment will be described.

  FIG. 1 shows a basic configuration example of the imaging optical system in the present embodiment. As shown in FIG. 1, the imaging optical system includes an imaging lens LNS that forms an image of a subject on the sensor surface of the imaging device, and an optical filter FLT that separates a band between the first pupil and the second pupil. In the following description, an example in which the pupil is divided in the horizontal scanning direction of the imaging sensor, the first pupil is the right pupil, and the second pupil is the left pupil will be described. The direction of pupil division from the left pupil to the right pupil is also referred to as “parallax direction” as appropriate, and the pixel position along this parallax direction (that is, the horizontal scanning direction) is represented by a position x. In the present embodiment, the pupil separation direction is not limited to the horizontal scanning direction, and may be separated in any direction perpendicular to the optical axis of the imaging optical system.

The optical filter FLT has a right pupil filter FL1 having transmittance characteristics ^{f R} (first filter), and Hidarihitomi filter FL2 having different transmittance characteristics ^{f L} (second filter) and ^{f R,} a. The optical filter FLT is provided at the pupil position of the imaging optical system (for example, the diaphragm installation position), and the filters FL1 and FL2 correspond to the right pupil and the left pupil, respectively.

The imaging light transmitted through the imaging lens LNS and the optical filter FLT includes a subject image that passes through the right pupil and a subject image that passes through the left pupil. The image pickup device picks up an imaging light beam including these subject images as an image. Then, using the fact that the transmittance characteristics f ^R and f ^{L of} the left and right pupils are clearly band-separated, the pixel value I ^R (x) of the right pupil image (first image) and the left pupil image ( The pixel value I ^L (x) of the second image) is obtained. Note that an arbitrary pixel on the image sensor should be represented by a two-dimensional address, but here it will be described by a pixel value at a position in one direction on the image sensor for convenience. In addition, I ^R (x) and I ^L (x) represent pixel values at the position x, but I ^R (x) and I ^L (x) are appropriately used as codes representing the entire image.

Note that details of a method for generating I ^R (x) and I ^L (x) will be described later in the third embodiment, but the method of separating the left and right pupil images in the present embodiment is not limited to this. That is, as long as two images having a phase difference can be acquired, the correlation calculation method of this embodiment can be applied to the images.

As shown in FIG. 1, when the sensor surface of the image sensor is at the focus position FP, the right pupil image I ^R (x) and the left pupil image I ^L (x) are not shifted in the parallax direction, and the position is Match. On the other hand, when the sensor surface of the image sensor is at the defocus positions Dr and Df, the right pupil image I ^R (x) and the left pupil image I ^L (x) are shifted in the parallax direction. Specifically, at the defocus position Df, the focus is in the rear focus state behind the sensor surface of the image sensor, the right pupil image I ^R (x) is shifted to the right (+ x direction), and the left pupil image I ^L (x) shifts to the left (−x direction). On the other hand, the defocus position Dr is a front focus state where the focus is on the front side of the sensor surface of the image sensor, the right pupil image I ^R (x) is shifted to the left side, and the left pupil image I ^L (x) is on the right side. Shift.

In FIG. 1, I ^R (x) and I ^L (x) at the defocus position Dr are illustrated as examples, and the deviation amount of I ^R (x) and I ^L (x) is represented by δ. In the present embodiment, the shift amount δ is calculated as a phase difference by correlation calculation.

2.2. Phase Difference Detection Method Next, a method for detecting the phase difference between the right pupil image I ^R (x) and the left pupil image I ^L (x) will be described. In the following, I ^R (x) and I ^L (x) in the rear focus state will be described as an example, but the correlation calculation can be performed in the same manner in the front focus state.

As shown in FIG. 2, the shift amount for bringing the waveforms of the right pupil image I ^R (x) and the left pupil image I ^L (x) closer to each other is changed to δ ₀ , δ ₁ , δ ₂ ,. For the right pupil image I ^R (x) and the left pupil image I ^L (x). Here, δ _j represents the shift amount in the j-th calculation step (j is an integer of 0 or more), and δ ₀ = 0 and δ _j <δ _{(j + 1)} . In the rear focus state, the right pupil image I ^R (x) is shifted by δ _j to the left (−x direction), and the left pupil image I ^L (x) is shifted by δ _j to the right (+ x direction).

Sum of products of the shift amount [delta] _j is performed by the shift amount [delta] correlation width is set to correspond to the _j W _j, the correlation width W _j is the position xi (hereinafter to be the correlation value on the image as appropriate, (Referred to as “attention position”). Specifically, the correlation value C (δ _j ) at the position xi is obtained by the following expressions (1) and (2).

FIG. 3 shows a characteristic WD of the correlation width W _j with respect to the shift amount δ _j and a characteristic CV of the correlation value C (δ _j ) with respect to the shift amount δ _j .

As indicated by the characteristic WD, the correlation width W _j increases in conjunction with increasing the shift amount δ _j . Is basically set so that _{W j <W (j + 1} ) , of not all _{W j <W (j + 1} ) need not be relative to [delta] _j, for example near the [delta] ₀ (Fig. 3 cases In this case, W _j = W _{(j + 1)} in δ _{0 to} δ ₂ ). The relationship between the correlation width W _j and the shift amount δ _j may be set in advance, for example, from the relationship between the phase difference of the image and the degree of blur. That is, the larger the phase difference of the image, the more the image is blurred. Therefore, the correlation width W _j is increased in conjunction with the shift amount δ _j . This will be described in detail later with reference to FIG.

As shown by the characteristic CV, when the correlation value C (δ _j ) is obtained while changing the shift amount to δ ₀ , δ ₁ , δ ₂ ,..., The shift amount δ _j becomes the phase difference at the target position xi. Peak when matched. The shift amount at this time is represented by δ _k (k is a natural number), the correlation width is represented by W _k , and the peak value of the correlation value is represented by C (δ _k ) = C _max . In this embodiment, the phase difference is detected by detecting this correlation peak. Phase difference detection is 2.delta. _K. As a correlation peak detection method, for example, peak hold processing may be performed on the correlation value C (δ _j ).

  As shown in E1, if it is determined that the first correlation peak has been detected in the peak detection, the correlation calculation is terminated without obtaining the subsequent correlation values. For example, the correlation value held in the peak hold process is compared with the newly calculated correlation value, and when the determination that the newly calculated correlation value is smaller continues for a predetermined number of times, the correlation calculation is terminated. . In this way, it is possible to prevent an erroneous correlation peak from being detected as a phase difference, and the correlation calculation is terminated midway, so that the processing speed can be increased.

Next, the principle of increasing the correlation width W _j in conjunction with the shift amount δ _j will be described in principle with reference to FIGS. In the following description, the rear focus state is taken as an example.

  FIG. 4A schematically shows a phase difference image in the vicinity of the focus position FP in FIG. f (x) represents an image captured by an ideal imaging system in which the pupil is not divided and is in focus. In FIG. 4A, 1 changes at a representative frequency of the captured image. Two peaks are denoted by f (x).

An actually captured image is a convolution of the point spread function (PSF) of the imaging optical system with respect to this f (x). If the point spread functions for the right and left pupils are PSF ^R (x) and PSF ^L (x), respectively, the right pupil image is obtained as I ^R (x) = f (x) * PSF ^R (x) The pupil image is obtained by I ^L (x) = f (x) * PSF ^L (x). Here, “*” represents a convolution operation.

Since the barycentric positions of PSF ^R (x) and PSF ^L (x) are different, I ^R (x) and I ^L (x) shift to the left and right accordingly. In the rear focus state, as shown in FIG. 4A, I ^R (x) shifts to the right side and I ^L (x) shifts to the left side. In the vicinity of the focus position, the interval between the gravity center positions of PSF ^R (x) and PSF ^L (x) is narrow, and therefore the phase difference δ between I ^R (x) and I ^L (x) is small. Further, since the distribution of PSF ^R (x) and PSF ^L (x) is narrow, I ^R (x) and I ^L (x) are also distributed in a relatively narrow range of width W. Since I ^R (x) and I ^L (x) are f (x) blurred by the point spread function at the target position xi, the spread of the distribution of I ^R (x) and I ^L (x) is , Corresponding to the frequency of the image at the target position xi.

When obtaining a correlation value from I ^R (x) and I ^L (x), if the correlation width is made too wide than the width W, pixels of I ^R (x) and I ^L (x) are included in the correlation width. A correlation peak may occur when a plurality of value peaks are included and peaks that do not actually correspond overlap. On the other hand, if the correlation width is made narrower than the width W, the peak of the pixel values of I ^R (x) and I ^L (x) is not included in the correlation width, and the correlation peak cannot be detected. From the above, it can be seen that the correlation peak can be appropriately detected by setting the correlation width to the width W.

FIG. 4B schematically shows a phase difference image in the vicinity of the defocus position Df in FIG. At the defocus position, the distance between the gravity center positions of PSF ^R (x) and PSF ^L (x) is wide, and therefore the phase difference δ ′ between I ^R (x) and I ^L (x) is the phase difference δ at the focus position. Bigger than. Further, since the distribution of PSF ^R (x) and PSF ^L (x) is wide, the width W ′ in which I ^R (x) and I ^L (x) are distributed is wider than the width W at the focus position. Corresponding to this width W ′, the correlation width is set larger at the defocus position than at the focus position. It can also be seen that the width W ′ increases as the phase difference δ ′ increases, so that the correlation width should be increased as the phase difference δ ′ increases (that is, the image at the position of interest xi is blurred).

When the correlation calculation of the present embodiment is applied to the above I ^R (x) and I ^L (x), as shown in FIG. 5, the shift amount δ _j increases and I ^R (x) The interval of I ^L (x) decreases as 2 (δ _k −δ ₀ ), 2 (δ _k −δ ₁ ),..., And the correlation width increases as W ₀ , W ₁ ,. To go. When the interval between I ^R (x) and I ^L (x) is 2 (δ _k −δ _k ) = 0 and I ^R (x) and I ^L (x) match, the correlation value becomes the peak value. The correlation width W _{k at} this time is set to about the distribution width of I ^R (x) and I ^L (x) as described above, and is a correlation width that can appropriately detect the correlation peak.

Thus, by setting a correlation width _{W j} in synchronization with the shift amount [delta] ^_j, an appropriate correlation to focus degree of ^I R (x) and ^I L (x) (i.e. the phase difference 2.delta. _K) it is possible to detect a correlation peak with a width W _k.

2.3. Sub-pixel phase difference detection method In the above-described method of calculating the product sum of pixel values, it is difficult to detect the phase difference with a smaller accuracy (sub-pixel) than the pixel interval. Therefore, in this embodiment, after detecting the phase difference by the product sum, the phase difference detection of the subpixel may be further performed. Hereinafter, the phase difference detection of the sub-pixel will be described.

As illustrated in FIG. 6, it is assumed that a shift of φ remains between the right pupil image I ^R (x + δ _k ) shifted by the shift amount δ _k and the left pupil image I ^L (x−δ _k ). The shift amount δ _k is the phase difference between I ^R (x) and I ^L (x) obtained by the correlation calculation by the product sum as described with reference to FIG. The shift amount φ is any amount smaller than the pixel interval. In the present embodiment, this deviation amount φ is detected.

In the detection process of the deviation amount φ, first, coefficients A ^L , B ^L , A ^R (d), and B ^R (d) are obtained by the following equations (3) and (4). The coefficients A ^L and B ^L correspond to the Fourier coefficient of I ^L (x−δ _k ) at the angular frequency ω. The coefficients A ^R (d) and B ^R (d) correspond to the Fourier coefficient of I ^R (x + δ _k ) at the angular frequency ω, and are obtained by shifting the product sum range by d.

Here, the angular frequency ω is determined by the correlation width W _k when the phase difference 2δ _k is detected, and ω = π / W _k . cos (ωx) and sin (ωx) are functions defined in a range of a width W _k centered on xi as shown in FIG. The data of the functions cos (ωx) and sin (ωx) have values at intervals that are narrower than the interval between pixels, and the detection accuracy of φ can be improved as the interval is reduced.

Next, the declination difference Δθ is obtained by the following equation (5). As shown in FIG. 7, tan ⁻¹ {B ^L / A ^L } is a declination of coefficients A ^L and B ^L , and tan ⁻¹ {B ^R (d) / A ^R (d) is a coefficient A ^{R (d),} which is the argument of ^B R (d). The declination difference Δθ is the difference between them.
Δθ = tan ⁻¹ {B ^L / A ^L } −tan ⁻¹ {B ^R (d) / A ^R (d)}] (5)

Next, as shown in FIG. 8, by changing the shift amount d, and detects the minimum value [Delta] [theta] _min declination difference [Delta] [theta], detects the shift amount d corresponding to the minimum value [Delta] [theta] _min as the phase difference of the sub-pixels . As shown in FIG. 6, when the shift amount between I ^R (x + δ _k ) and I ^L (x−δ _k ) is φ, the detected phase difference is d = φ. The final phase difference is obtained as 2δ _k + φ from the phase difference 2δ _k obtained by the product sum of I ^R (x) and I ^L (x) and the phase difference φ of the subpixel.

According to the embodiment described above, even if a minute phase difference φ remains after the phase difference 2δ _k is obtained by the correlation calculation by the product sum, the phase difference φ can be detected with sub-pixel accuracy. Further, the angular frequency ω for obtaining the deviation difference Δθ is determined (ω = π / W _k ) according to the correlation width W _k when the phase difference 2δ _k is detected by the correlation calculation by the product sum, so that the image becomes The deviation angle difference Δθ corresponding to the possessed frequency can be obtained. That is, as described in FIG. 4A and the like, the representative frequency of the image is determined according to the phase difference 2δ _k (that is, the degree of blur), and includes, for example, about one wave of that frequency. A correlation width W _k is determined. This means that the frequency component of the angular frequency ω = π / W _k is abundant in the image. By obtaining the Fourier coefficient at the angular frequency ω, the Fourier coefficient having a large absolute value is obtained, and the accuracy is obtained. The deviation angle difference Δθ can be determined well.

In the above-described embodiment, the shift amount d when the deviation angle difference Δθ is minimized is determined. However, the present embodiment is not limited to this, and the difference [{B ^L / A ^L } − {B ^R (d ) / A ^R (d)}] (except when A ^L = 0 or A ^R (d) = 0) may be obtained.

2.4. Imaging Device FIG. 9 shows a configuration example of the imaging device in the first embodiment. The imaging apparatus includes an imaging lens LNS, an optical filter FLT, an imaging unit 10, a phase difference image generation unit 20, a focus control unit 80, and a phase difference detection unit 82. Note that the present embodiment is not limited to the configuration shown in FIG. 9, and various modifications such as omitting some of the components (for example, the focus control unit 80) or adding other components are possible. It is. For example, FIG. 9 shows a configuration for performing the phase difference AF, but for example, a configuration in which the three-dimensional information of the subject is acquired based on the detected phase difference may be used.

  The imaging unit 10 can include an imaging device and an imaging processing unit. The imaging element images a subject imaged by the imaging lens LNS and the optical filter FLT. The imaging processing unit performs imaging operation control, A / D conversion processing of analog pixel signals, and the like.

Phase difference image generator 20, based on the image captured by the imaging unit 10, generates the right pupil images I ^R and Hidarihitomi image I ^L. These two images are referred to as phase difference images. As a method for generating a phase difference image, various methods such as a known method (for example, a method described in Japanese Patent Laid-Open No. 2001-174696 and Japanese Patent Laid-Open No. 10-276964) can be employed. Or you may produce | generate a phase difference image by the multiband estimation process demonstrated in 3rd Embodiment.

The phase difference detection unit 82 detects the phase difference by applying the above-described phase difference detection method to the phase difference images I ^R and I ^L.

The focus control unit 80 performs autofocus control based on the detected phase difference information. That is, the focus control unit 80 calculates a focus control amount (a movement amount and a movement direction of the imaging lens LNS) based on the phase difference information and the information on the shift direction of the phase difference images I ^R and I ^L , and the focus control is performed. Control to move the focus lens based on the amount is performed. The shift direction of the phase difference images I ^R and I ^L may be determined exploratively, for example, in correlation calculation.

2.5. Phase Difference Detection Process FIG. 10 shows a flowchart of the phase difference detection process of this embodiment. When this process is started, a point xi (target pixel) for which a phase difference is to be acquired is set (step S1). For example, when performing phase difference AF, the focus control unit 80 sets a point xi to be focused on the captured image.

Next, the phase difference image generation unit 20 acquires phase difference images I ^R and I ^L (step S2). Next, the phase difference detection unit 82 sets the correlation width W _j = W ₀ and the shift amount δ _j = δ ₀ = 0 (step S3). Next, the phase difference detection unit 82 brings the phase difference images I ^R and I ^L close by the shift amount δ _j set in step S3 (step S4). Next, the phase difference detection unit 82 calculates the correlation value C (δ _j ) at the point xi with the correlation width W _j set in step S3 (step S5). Next, the phase difference detection unit 82 determines whether or not the correlation value C (δ _j ) is a peak value (step S6).

When it is determined that the correlation value C (δ _j ) is not a peak value, the phase difference detection unit 82 sets the correlation width W _j = W _{j + 1} and the shift amount δ _j = δ _{j + 1} (step S7), and step S4 Run again. On the other hand, when it is determined that the correlation value C (δ _j ) is the peak value, the phase difference detection unit 82 outputs the phase difference 2δ _j = 2δ _k based on the shift amount δ _j (step S8). The phase difference detection process ends. As the information of the phase difference is not limited to the phase difference 2.delta. _K itself, may output the shift amount [delta] _k.

  FIG. 11 shows a flowchart of the phase difference detection process when the sub-pixel phase difference detection process is performed. Steps S21 to S27 are the same as steps S1 to S7 in FIG.

If it is determined in step S26 that the correlation value C (δ _j ) is a peak value, the phase difference detection unit 82 determines whether or not to detect a phase difference with subpixel accuracy (step S28). For example, it may be determined based on setting information from the user, or may be determined based on the magnitude of the correlation value C (δ _j ).

When it is determined not to detect the phase difference with sub-pixel accuracy, the phase difference detection unit 82 outputs the phase difference 2δ _j = 2δ _k based on the shift amount δ _j and ends the phase difference detection process. When it is determined that the phase difference with sub-pixel accuracy is detected, the phase difference detection unit 82 obtains the deviation angle difference Δθ between the phase difference images I ^R and I ^{L at} the shift amount d (step S29). The initial value of d is, for example, d = 0. Next, the phase difference detection unit 82 determines whether or not the declination difference Δθ is a minimum value (step S30).

If the declination difference Δθ is not the minimum value, the phase difference detection unit 82 sets the shift amount d = d + α (step S31) and executes step S29 again. Here, α is a change amount (step) when the shift amount d is changed, and is set according to the accuracy of detecting the phase difference of the sub-pixels. On the other hand, when the declination difference Δθ is the minimum value, the phase difference detection unit 82 outputs the phase difference 2δ _j + d = 2δ _k + φ based on the shift amounts δ _j , d (step S32). The detection process ends.

According to the above embodiment, as described in FIG. 9, the imaging device includes the imaging element (imaging unit 10), the phase difference image generation unit 20, and the phase difference detection unit 82. The imaging device captures a first subject image and a second subject image having a parallax with respect to the first subject image to obtain a captured image. Based on the captured image, the phase difference image generation unit 20 generates a first image I ^R (x) (right pupil image) corresponding to the first subject image and a second image I ^L (x corresponding to the second subject image). ) (Left pupil image). The phase difference detection unit 82 performs correlation calculation while relatively shifting the first image I ^R (x) and the second image I ^L (x), and performs the first image I ^R (x) and the second image I ^L. The phase difference of (x) is detected. At this time, the phase difference detection unit 82 sets a correlation width W _j that is an area width for performing a product-sum of correlation calculations according to the shift amount δ _j of the shift.

  For example, in this embodiment, the imaging optical system (imaging lens LNS, optical filter FLT) includes a first subject image (a subject image that has passed through the right pupil) and a second subject image that has parallax with respect to the first subject image. (Subject image that has passed through the left pupil). Then, the image sensor picks up the first subject image and the second subject image. Here, “parallax” means that the imaging position changes (shifts) due to the relative position difference between the subject and the observation point (in this embodiment, the pupil of the optical system).

In this way, as described with reference to FIG. 4A and the like, the phase difference can be detected with an appropriate correlation width according to the degree of focus at the target position xi. That is, since the phase difference 2δ _k (δ _{k in} FIG. 3) represents the degree of focusing of the image, an appropriate correlation width corresponding to the degree of focusing is obtained by linking the shift amount δ _j and the correlation width W _j. possible to detect a phase difference 2δ _k in W _k.

In the present embodiment, as described with reference to FIG. 3, the phase difference detection unit 82 repeatedly performs the correlation calculation while sequentially increasing the shift amount δ _j and gradually increasing the correlation width W _j, thereby calculating the correlation value C (δ _j ) is obtained, and the shift amount δ _k when the correlation value C (δ _j ) reaches the maximum value C _max is detected as a phase difference.

As described with reference to FIG. 4A and the like, the larger the phase difference 2δ _k is, the more blurred the image is, and the correlation width W _k needs to be increased. In this regard, according to the present embodiment, the correlation width W _j is increased together with the shift amount δ _j , so that the phase difference 2δ _k is set with the correlation width W _k corresponding to the degree of image blur (a typical frequency component of the image). It can be detected. Further, when the target position xi is a focused part, the correlation peak can be obtained with a narrow correlation width, that is, with a small number of calculations. Thereby, the phase difference detection process can be speeded up, and the processing load can be reduced particularly when the phase difference of the entire image is calculated (for example, shape measurement).

In the present embodiment, as described with reference to FIG. 3, when the phase difference detection unit 82 obtains the correlation value C (δ _j ) while sequentially increasing the shift amount δ _j , the correlation value C (δ _j ) The shift amount δ _k corresponding to the first peak value C _max is detected as a phase difference.

As described in FIG. 3 and the like, when a plurality of peaks of the phase difference images I ^L (x) and I ^R (x) are included in the correlation width Wj, when peaks that do not actually correspond overlap each other There is a possibility of detecting a correlation peak. In this regard, according to the present embodiment, since the first correlation peak is detected as a phase difference, the correlation peak is correctly detected when the corresponding peaks of the phase difference images I ^L (x) and I ^R (x) overlap. can do.

3. Second Embodiment 3.1. Next, a second embodiment will be described. In the second embodiment, the focus direction is determined using the right pupil image I ^R (x) and the left pupil image I ^L (x), and the shift direction in the correlation calculation is determined based on the determination result of the focus direction. decide. Note that the description of the same content as that described in the first embodiment (for example, the basic configuration of the imaging optical system and the phase difference detection method) will be omitted as appropriate.

The imaging apparatus of the second embodiment has the same configuration as that of the first embodiment described with reference to FIG. When the configuration is the same as that of the first embodiment, in the second embodiment, the phase difference detection unit 82 applies a focusing direction determination method to be described later to the phase difference images I ^R and I ^L to focus. Determine the direction. Then, based on the determined in-focus direction, the shift direction of the phase difference images I ^R and I ^L in the correlation calculation is determined.

3.2. In-focus direction determination method Next, the in-focus direction determination method of the present embodiment will be described in detail. As shown in FIGS. 12A and 12B, the DC components (with width W) of the pixel values of the captured image I (x), the right pupil image I ^R (x), and the left pupil image I ^L (x) The reference value is adjusted by setting the average value) to zero level, and the comparison pixel values I (x) ′, I ^R (x) ′, and I ^L (x) ′ are obtained. FIG. 12A shows a comparison pixel value in the rear focus state, and FIG. 12B shows a comparison pixel value in the front focus state. These comparison pixel values are obtained by the following equation (6). Hereinafter, the “comparison pixel value” is also simply referred to as “pixel value” as appropriate.

Here, the width W is a predetermined average calculation section obtained in advance by experiments or the like, for example, and is a width centered on the position x. In FIGS. 12A and 12B, the width W is illustrated by taking x = xi as an example. Further, the relationship of the following expression (7) is established among I (x) ′, I ^R (x) ′, and I ^L (x) ′.
I ′ (x) = [I ^L ′ (x) + I ^R ′ (x)] / 2 (7)

Next, it is determined whether a pixel whose focus direction is to be determined (hereinafter referred to as a “target pixel”) belongs to the increase interval Ra or the decrease interval Fa of the pixel value I (x) ′. Specifically, if the position of the target pixel is x = xi, the intersection of I (x) ′, I ^R (x) ′, and I ^L (x) ′ (the pixel values for comparison match (including substantially matching)) Position x ₀ , x ₁ of the intersection closest to xi. Position x ₀ is the intersection of the left side of the pixel of interest, the position x ₁ is the right side of the intersection point than the pixel of interest. When the following expression (8) is satisfied, it is determined that the target pixel belongs to the increasing section Ra. When the following expression (9) is satisfied, it is determined that the target pixel belongs to the increasing section Ra. For example, in FIG. 12A, it is determined that the position xi of the target pixel belongs to the decrease section Fa.
[I ′ (x ₀ ) −I ′ (x ₁ )] <0 (8)
[I ′ (x ₀ ) −I ′ (x ₁ )]> 0 (9)

Note that the pixel of interest increases when [I ^L ′ (x ₀ ) −I ^L ′ (x ₁ )] <0 or [I ^R ′ (x ₀ ) −I ^R ′ (x ₁ )] <0 is satisfied. You may determine with belonging to area Ra. Further, when [I ^L ′ (x ₀ ) −I ^L ′ (x ₁ )]> 0 or [I ^R ′ (x ₀ ) −I ^R ′ (x ₁ )]> 0 is satisfied, the target pixel decreases. You may determine with belonging to the area Fa.

Next, based on the vertical relationship between I ′ (x) and I ^L ′ (x), it is determined whether the pixel of interest is in the front focus state or the rear focus state. Specifically, a difference value E ^L between I ′ (x) and I ^L ′ (x) in a section to which the target pixel belongs is obtained by the following equation (10), and the sum is calculated based on the sign of the difference value E ^L. Determine the direction of focus. As shown in FIGS. 12 (A) and 12 (B), since the vertical relationship between I ′ (x) and I ^L ′ (x) is different in the increasing interval Ra and the decreasing interval Fa, the vertical relationship and the focus are in focus. Correspondence with direction is different. That is, in the increasing section Ra, the focusing direction is determined by the following expression (11), and in the decreasing section Fa, the focusing direction is determined by the following expression (12).

Needless to say, the in-focus direction may be determined based on the vertical relationship between I ′ (x) and I ^R ′ (x).

If the determined in-focus direction is the rear focus state, it is determined that I ^R (x) is shifted to the right (+ x direction) and I ^L (x) is shifted to the left (−x direction). Is done. In this case, correlation calculation is performed by shifting I ^R (x) to the left and I ^L (x) to the right. On the other hand, when the determined in-focus direction is the front focus state, I ^R (x) is shifted to the left (−x direction), and I ^L (x) is shifted to the right (+ x direction). It is judged. In this case, correlation calculation is performed by shifting I ^R (x) to the right and I ^L (x) to the left.

According to the above embodiment, the phase difference detection unit 82 compares the pixel value of the captured image I (x) with at least one of the first image I ^R (x) and the second image I ^L (x). The focusing direction of the imaging optical system is determined by comparing the pixel value of the image (for example, I ^L (x)). The phase difference detection unit 82 sets a shift direction for shifting the second image I ^L (x) with respect to the first image I ^R (x) in the correlation calculation based on the determined in-focus direction.

In this way, it is possible to determine whether the focus state is the front focus state or the rear focus state, and the first image I ^R (x) and the second image I ^L (x) corresponding to the determination result. You can know the direction of displacement. Thus, since it can be understood in which direction the peak of the correlation value can be detected by moving the first image I ^R (x) and the second image I ^L (x), the correlation calculation can be performed efficiently.

In the present embodiment, as described with reference to FIG. 12A and the like, the phase difference detection unit 82 determines the direction in which the first image I ^R (x) and the second image I ^L (x) are shifted due to the parallax as the parallax direction. In the case of (+ x direction), a region (Ra or Fa) in which the pixel value of the captured image I (x) increases or decreases in the parallax direction is specified. As described in the above formulas (10) to (12), the phase difference detection unit 82 determines the magnitude of the pixel value of the captured image I (x) and the pixel value of the comparison image I ^L (x) in the specified region. Based on the relationship, the focus state is in a state where the focus is closer to the subject as viewed from the imaging optical system, or the focus is in a position farther than the subject as viewed from the imaging optical system. It is determined whether or not it is in the previous focus state.

In this way, the in-focus direction of the imaging optical system can be determined by comparing the pixel value of the captured image I (x) with the pixel value of the comparative image I ^L (x). Further, as described with reference to FIG. 12A and the like, in the region Ra where the pixel value of the captured image I (x) increases and the region Fa where the pixel value decreases, the pixel value of the captured image I (x) and the comparison image I ^L ( The magnitude relationship of x) is different. In this regard, according to the present embodiment, it is possible to specify whether the position where the in-focus direction is to be determined belongs to the increasing area Ra or the decreasing area Fa, and the in-focus direction is determined based on the magnitude relationship in the specified area. Can be judged.

4). Third Embodiment 4.1. Next, a third embodiment will be described. In the third embodiment estimates the image of five bands from RGB image, among the images of the five bands, from an image of the band corresponding to the right pupil generates a right pupil images I ^R, the bands corresponding to the left pupil generating a Hidarihitomi image I ^L from the image. Note that the description of the same contents as those described in the first embodiment and the second embodiment (for example, the basic configuration of the imaging optical system and the phase difference detection method) will be omitted as appropriate.

  FIG. 13 shows a configuration example of an imaging apparatus in the third embodiment. This imaging device includes an imaging lens LNS, an optical filter FLT, an imaging unit 10, a phase difference image generation unit 20, an output unit 25, a display image generation unit 30, a monitor display unit 40, a spectral characteristic storage unit 50, and a multiband estimation unit. 70, a focus control unit 80, a phase difference detection unit 82, a data compression unit 90, and a data recording unit 100. Note that the present embodiment is not limited to the configuration in FIG. 13, and various components such as omitting some of the components (for example, the output unit 25 and the data compression unit 90) and adding other components. Variations are possible.

  The imaging unit 10 can include an imaging device and an imaging processing unit. The imaging element is, for example, an imaging element of RGB three primary colors, and images a subject imaged by the imaging lens LNS and the optical filter FLT. The imaging processing unit performs control of imaging operations, A / D conversion processing of analog pixel signals, demosaicing processing for Bayer images, and the like.

  The spectral characteristic storage unit 50 stores spectral characteristic data of a color filter included in the image sensor, and outputs the data to the multiband estimation unit 70. Here, strictly speaking, the RGB components to be imaged are determined not only by the color filter but also by the spectral sensitivity characteristics of the imaging element and the spectral characteristics of the imaging lens LNS. That is, the spectral characteristic storage unit 50 stores spectral characteristic data including the spectral characteristics of the image sensor and the imaging lens LNS.

The multiband estimation unit 70 performs multiband estimation processing based on the captured image and spectral characteristic data. Specifically, as shown in FIG. 13, the wavelength range of the color filter of the image sensor is divided into five bands {r ^L , r ^R , g, b ^L , b ^R }. The right pupil of the optical filter FLT transmits the band {r ^R , g, b ^R }, and the left pupil transmits the band {r ^L , g, b ^L }. The multiband estimation unit 70 estimates pixel values of five bands {r ^L , r ^R , g, b ^L , b ^R } from the RGB pixel values of the captured image I (x).

The phase difference image generation unit 20 determines the right pupil image I ^R = {r ^R , g, b ^R } and the left pupil image I from the pixel values of the five bands {r ^L , r ^R , g, b ^L , b ^R }. ^L = {r ^L , g, b ^L } is generated. These two images are referred to as phase difference images.

The phase difference detection unit 82 applies the above-described phase difference detection method to the same color components {r ^R , r ^L } of the phase difference images I ^R and I ^L (that is, R pixel values of I ^R and I ^L ). The phase difference is detected. Note that {b ^R , b ^L } (that is, B pixel values of I ^R and I ^L ) may be used as the same color components of the phase difference images I ^R and I ^L. The phase difference detection unit 82 may perform the in-focus direction determination described in the second embodiment, and may determine the shift direction of the phase difference images I ^R and I ^L in the correlation calculation. Note that the present embodiment is not limited to this, and the direction in which the phase difference images I ^R and I ^L are shifted in the correlation calculation may be determined exploratoryly.

  In this way, by performing the correlation calculation in conjunction with the shift amount δk and the correlation width Wk, the calculation amount can be reduced and the calculation efficiency can be improved. Further, when determining the in-focus direction, it is not necessary to determine the shift direction in a search, so that the calculation efficiency can be further improved.

The focus control unit 80 performs autofocus control based on the detected phase difference information. Specifically, the focus control unit 80 includes a focus control amount calculation unit 84 and a focus lens drive control unit 86. Focus control amount calculating section 84 calculates the defocus amount of the imaging lens LNS on the basis of the information of the phase difference, the focus control based on the defocus amount and the phase difference image I ^R, the deviation direction of the information I ^L The amount (the moving amount and moving direction of the imaging lens LNS) is calculated. The focus lens drive control unit 86 performs control to move the focus lens based on the moving amount and moving direction of the imaging lens LNS.

The display image generation unit 30 generates an RGB image for display from the pixel values of the five bands {r ^L , r ^R , g, b ^L , b ^R }. For example, a pixel value of {r ^R , g, b ^R } is an RGB image. The monitor display unit 40 displays the image generated by the display image generation unit 30.

The output unit 25 performs a process of outputting the phase difference images I ^R and I ^L generated by the phase difference image generation unit 20 and the information of the phase difference detected by the phase difference detection unit 82 and the information on the in-focus direction. As output processing, for example, saving to an external storage device, display on an external monitor, output to an external information processing device, and the like are assumed. In the external information processing apparatus, for example, three-dimensional information of a subject is obtained based on phase difference information and focus direction information. The output unit 25 may obtain the three-dimensional information and display it on the monitor display unit 40.

  The data compression unit 90 performs processing for compressing the RGB image from the imaging unit 10. The data recording unit 100 records compressed RGB image data and color filter spectral characteristic data. These recorded data can be used for multiband estimation processing and phase difference detection processing in post processing after photographing. This post-processing may be performed by an information processing device configured separately from the imaging device.

4.2. Multiband estimation process 4.2.1. Next, a multiband estimation process for obtaining a phase difference image from a captured image will be described in detail. In the following description, an RGB Bayer array image sensor will be described as an example. However, the present embodiment is not limited to this, and it is sufficient that the image sensor has an overlapping portion in the transmittance characteristics of the color filter.

FIG. 14 is an explanatory diagram for band division. As shown in FIG. 14, the five-band components b ^R , b ^L , g, r ^R , and r ^L are components that are determined according to the spectral characteristics of the imaging system. FIG. 14 shows the transmittance characteristics F _R , F _G , and F _B of the color filter of the imaging sensor as the spectral characteristics of the imaging system. Strictly speaking, the spectral characteristics of the imaging system excludes, for example, the color filter. It also includes the spectral characteristics of the image sensor, the spectral characteristics of the optical system, and the like. In the following, for the sake of simplicity, it is assumed that spectral characteristics of the image sensor and the like are included in the transmittance characteristics F _R , F _G and F _B of the color filter shown in FIG.

As shown in FIG. 14, the band component corresponding to the overlapping portion of the transmittance characteristic F _B of the blue filter and the transmittance characteristic F _G of the green filter is b ^L , and the non-existence of the transmittance characteristic F _B of the blue filter component of the band corresponding to the overlapping portion is b ^R. Further, components of the band corresponding to the overlap between the transmittance characteristic F _G of the transmittance characteristic F _R and the green filters of the red filter is r ^R, corresponding to the non-overlapping portion of the transmission characteristic F _R of the red filter component of the band is ^{r L.} Further, components of the band corresponding to non-overlapping portions of the transmission characteristic F _G of the green filter is g. Here, the non-overlapping portion is a portion that does not overlap with the transmittance characteristics of other color filters.

5-band of BD1~BD5, the transmittance characteristic F _R, F _G, may be determined depending on the shape and degree of overlap F _B, need not be the band itself bandwidth and the overlapping portion of the transmission characteristic . For example, the band of the overlapping portion of the transmittance characteristics F _G and F _B is approximately 450 nm to 550 nm, but the band BD2 only needs to correspond to the overlapping portion, and does not need to be 450 nm to 550 nm.

In the third embodiment, the band {b ^R , g, r ^R } is assigned to the right pupil filter FL1 in FIG. 1 as the transmission wavelength region f ^R , and the band {b is set to the transmission wavelength region f ^L to the left pupil filter FL2. Assign b ^L , g, r ^L }. As shown in FIG. 14, the wavelength division light {b ^R , g, r ^R } that has passed through the right pupil and the wavelength division light {b ^L , g, r ^L } that has passed through the left pupil have distinct wavelength bands. It is separated. On the other hand, the spectral characteristics {F _R , F _G , F _B } of the color filter of the image sensor are characteristics in which the wavelength bands of adjacent spectral characteristics overlap. Considering this overlapping state, the red, green, and blue pixel values R, G, and B in each pixel after the demosaicing process can be modeled as in the following equation (13).
R = g _R + r ^R _R + r ^L _R ,
_{^{G = b L G + g G}} + r R G,
B = b ^R _B + b ^L _B + g _B (13)

As shown in FIG. 15C, the component {b ^R _B , b ^L _B , g _B } corresponds to the wavelength-divided light {b ^R , b ^L , g} that has passed through the blue filter having the spectral characteristic F _B. . As shown in FIG. 15D, the component {b ^L _G , g _G , r ^R _G } corresponds to the wavelength-divided light {b ^L , g, r ^R } that has passed through the green filter having the spectral characteristic F _G. . Further, as shown in FIG. 17 (D), the component _{^{{g R, r R R,}} r L R} is the spectral characteristics _F wavelength division light passed through the red filter _{^{R {g, r R, r}} L} in Correspond. The superscript suffix for each component represents whether the right pupil “R” or the left pupil “L” has passed, and the subscript suffix represents the red filter “R”, the green filter “G”, and the blue filter It indicates which of “B” passed.

4.2.2. {B ^R , b ^L , (g + r ^R )} Estimation Processing Next, using FIG. 15A to FIG. 16, the component values {b ^R _B , b ^L _B , A process for estimating g _B }, {b ^L _G , g _G , r ^R _G }, and {g _R , r ^R _R , r ^L _R } will be described.

First, using the above equation (13), the wavelength band {b ^L , g} overlapping with the pixel value {B, G} is removed based on the difference between the pixel values {B, G}, and the component b ^R and A process of deriving a relational expression of the components {b ^R , b ^L , (g + r ^R )} by obtaining the relationship of the component [g + r ^R ] is performed.

It should be noted here that, as shown in FIGS. 15A to 15D, the component b ^L _B of the pixel value _B and the component b of the pixel value G correspond to the wavelength band b ^L. it is a ^L _G, component ^b _L ^B, the ^b _{L G,} the spectral characteristics _F B, is that the relative gain of _{F G} is multiplied. Therefore, the components b ^L _B and b ^L _G are different values by the relative gain, and it is necessary to correct the components b ^L _B and b ^L _{G to} be equal.

As shown in FIGS. 15C and 15D, the pixel value G is a reference (for example, “1”), the component ratio of (b ^L _B + g _B ) is k _B1, and the component ratio of b ^L _G If k _B2 , the following equation (14) is established. Here, k _B1 / k _B2 is, for example, the gain ratio of the spectral characteristics F _B and F _G in the band b ^{L.
_{b L B + g B = (}} k B1 / k B2) × b L G (14)

Considering the gain of the spectral characteristics F _B in the bands b ^L and g, the component g _B is considered to be sufficiently smaller than the component b ^L _B. Therefore, in order to make the components b ^L _B and b ^L _G equal, the component ( b ^L _B + g _B ) and the component b ^L _G should be substantially equal. If the value obtained by correcting the component (b ^L _B + g _B ) is (b ^L _B '+ g _B '), the correction shown in the following equation (15) may be performed using the above equation (14).
^{_{b L B '+ g B'}} ≒ b L G = (k B2 / k B1) × (b L B + g B) (15)

Since the component (b ^L _B + g _B ) is included in the pixel value B, in order to correct the component (b ^L _B + g _B ), the pixel value B is eventually corrected. When this corrected B is B ′, the relationship of the following equation (16) is obtained.
B ′ = (k _B2 / k _B1 ) B (16)

From the above equation (16), B 'components ^{_{{b R B', b L}} B ', g B'} is the following equation (17).
^{_{b R B '= (k B2}} / k B1) × b R B,
b ^L _B '+ g _B ' ≈b ^L _G (17)

From the above equations (13) and (17), when the pixel value B ′ and the pixel value G are expressed using the components {b ^R _B ′, b ^L _G , g _G , r ^R _G }, the following equation (18) It becomes like this.
_{^{B '= b R B' +}} (b L B '+ g B') = b R B '+ b L G,
_{^{G = b L G + (g}} G + r R G) (18)

Next, as shown in the following equation (19), the difference between the corrected pixel value B ′ and the pixel value G is taken to remove the overlapping component b ^L. Further, the following equation (20) is established from the above equation (18).
_{^{B'-G = [b R B}} '+ b L G] - [b L G + g G + r R G]
= B ^R _B '-(g _G + r ^R _G ) (19)
_{^{b L G = B'-b R}} B '(20)

When b ^R _B ′ is an unknown (dominant variable), the relational expression of {b ^R _B ′, b ^L _G , (g _G + r ^R _G )} is expressed by the following equation (21) from the above equations (19) and (20). It is required as follows.
b ^R _B '= unknown number (dominant variable)
_{^{b L G = B'-b R}} B '
_{^{g G + r R G = b}} R B '- (B'-G) (21)

Since {B ′, G} is a detected known value, if the unknown b ^R _B ′ is determined based on the above equation (21), {b ^R _B ′, b ^L _G , (g _G + r ^R _G ) } Are all determined. That is, it is possible to identify _{^{{b R B ', b L}} G, (g G + r R G)} likelihood pattern.

FIG. 16 is a diagram showing this relationship in principle. As shown in FIG. 16, as the unknown b ^R _B ′, a value that minimizes the error between {b ^R _B ′, b ^L _G , (g _G + r ^R _G )} and {B ′ / 2, G / 2}. Ask for. That is, by _obtaining b ^R _B ′ when the error evaluation value E _BG shown in the following equation (22) is minimized and substituting the obtained b ^R _B ′ into the above equation (21), {b ^R _{B B} ^', _b L _^G, determines the value of _{^{(g G + r R G)}} }.
^{e B = (B '/ 2} -b R B') 2 + (B '/ 2-b L G) 2,
_{^{e G = (G / 2-}} b L G) 2 + (G / 2- (g G + r R G)) 2,
E _BG = e ^B + e ^G (22)

As described above, the component {b ^R _B ′, b ^L _G , (g _G + r ^R _G )} can be estimated from the 2-band pixel values {B ′, G} of each pixel.

In the above ^{_{{b R B ', b L}} G, (g G + r R G)} and {B' has been sought ^{/ 2, G / 2} b} R B when the error is minimized for ' in this embodiment, ^{b _{R B} also seeking ^{_{', b L G, (g}} G + r R G)} and _{{α B B', α Gb} G} b R B when the error of the minimum ' Good. Here, α _B and α _Gb are values satisfying the following expression (23). α _B is for calculating an average value of {b ^R _B ′, b ^L _G } with respect to _B ′, and α _Gb is {b ^L _G , (g _G + r ^R _G )} with respect to _G It is for calculating the average value of. These are from the color filter characteristics of the imaging device shown in FIG. ^{_{13 {b R B ', b}} L G} and ^{_{{b L G, (g G}} + r R G)} it is determined in consideration of the ratio of components Good.
0 <α _B ≦ 1, 0 <α _Gb ≦ 1 (23)

4.2.3. Processing for Estimating {(b ^L + g), r ^R , r ^L } Next, processing for estimating the component {(b ^L + g), r ^R , r ^L } from the pixel values {G, R} will be described.

Using the above equation (13), the overlapping wavelength band {g, r ^R } with the pixel value {G, R} is removed based on the difference between the pixel values {G, R}, and the component [b ^L + g ] And the component r ^L to obtain a relational expression of the component {(b ^L + g), r ^R , r ^L }.

As shown in FIGS. 17A to 17D, the component r ^R _G of the pixel value _G and the component r ^R _R of the pixel value R correspond to the wavelength band r ^R , but the component r ^R _G , R ^R _R are multiplied by the relative gains of the spectral characteristics F _G , F _R. Therefore, the components r ^R _G and r ^R _R have different values corresponding to the relative gain, and it is necessary to correct the components r ^R _G and r ^R _{R to} be equal.

As shown in FIGS. 17C and 17D, the pixel value G is a reference (eg, “1”), the component ratio of (g _R + r ^R _R ) is k _R1, and the component ratio of r ^R _G If k _R2 , the following equation (24) is established. k _R1 / k _R2 is, for example, the gain ratio of the spectral characteristics F _G and F _R in the band r ^{R.
_{g R + r R R = (}} k R2 / k R1) × r R G (24)

Band g, considering the gain spectral characteristic _{F R} in ^{r R,} since the components _{g R} considered sufficiently smaller than the component ^r _{R R,} in order to equalize component ^r _R ^G, the ^r _{R R,} the components ( g _R + r ^R _R ) and the component r ^R _G may be substantially equal. If the value obtained by correcting the component (g _R + r ^R _R ) is (g _R '+ r ^R _R '), the correction shown in the following equation (25) may be performed using the above equation (24).
^{_{g R '+ r R R'}} ≒ r R G = (k R2 / k R1) × (g R + r R R) (25)

Since the component (g _R + r ^R _R ) is included in the pixel value R, in order to correct the component (g _R + r ^R _R ), the pixel value R is eventually corrected. When this corrected R is R ′, the relationship of the following equation (26) is obtained.
R ′ = (k _R2 / k _R1 ) R (26)

From the above equation (26), the component {g _R ′, r ^R _R ′, r ^L _R ′} of _R ′ becomes the following equation (27).
g _R '+ r ^R _R ' ≈r ^R _G ,
^{_{r L R '= (k R2}} / k R1) × r L R (27)

From the above formulas (13) and (27), the pixel value G and the pixel value R ′ are expressed using the components {b ^L _G , g _G , r ^R _G , r ^L _R ′}. It becomes like this.
_{^{G = b L G + (g}} G + r R G),
R ′ = (g _R ′ + r ^R _R ′) + r ^L _R ′ = r ^R _G + r ^L _R ′ (28)

Next, as shown in the following equation (29), by taking the difference between the pixel value G and the corrected pixel value R ′, the overlapping component r ^R is removed. Further, the following expression (30) is established from the above expression (28).
_{^{G-R '= [b L}} G + g G + r R G)] - [r R G + r L R']
_{^{= (B L G + g G}} ) -r L R '(29)
_{^{r R G = R'-r L}} R '(30)

When r ^L _R ′ is an unknown (dominant variable), the relational expression of {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} is expressed by the following expression (31) from the above expressions (29) and (30). It is required as follows.
r ^L _R '= unknown number (dominant variable),
_{^{r R G = R'-r L}} R ',
b ^L _G + g _G = r ^L _R '+ (G−R ′) (31)

Since {G, R ′} is a detected known value, if the unknown number r ^L _R ′ is determined based on the above equation (31), {r ^L _R ′, r ^R _G , (b ^L _G + g _G ) } Are all determined. That is, it is possible to specify a likelihood pattern of {r ^L _R ′, r ^R _G , (b ^L _G + g _G )}.

FIG. 18 is a diagram showing this relationship in principle. As shown in FIG. 18, a value that minimizes an error between {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} and {G / 2, R ′ / 2} as the unknown r ^L _R ′. As an unknown r ^L _R ′. That is, by _obtaining r ^L _R ′ when the error evaluation value E _GR shown in the following equation (32) is minimized, and substituting the obtained r ^L _R ′ into the above equation (31), {r ^L _{R ^', r R} ^G, determines the value of _{(b L G + g G)} }.
_{^{e G = (G / 2- (}} b L G + g G)) 2 + (G / 2-r R G) 2,
_{^{e G = (R '/ 2}} -r R G) 2 + (R' / 2- (r L R ')) 2,
E _GR = e ^G + e ^R (32)

As described above, the component {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} can be estimated from the 2-band pixel value {G, R ′} of each pixel.

In the above {r L R ', r R G, (b L G + g G)} and {G / 2, R' is sought in the case where an error of / 2} is minimized r L R ', In the present embodiment, even if r L R ′ when the error between {r L R ′, r R G , (b L G + g G )} and {α Gr G, α R R ′} is minimized is obtained. Good. Here, α R and α Gr are values satisfying the following expression (33). alpha R is '{r L R for', r R G} R is for calculating an average value of, alpha Gr is for G {r R G, (b L G + g G)} It is for calculating the average value of. These can be determined by considering the component ratio of {r L R ′, r R G } and {r R G , (b L G + g G )} from the color filter characteristics of the image sensor as shown in FIG. Good.
0 <α R ≦ 1, 0 <α Gr ≦ 1 (33)

4.2.4. Component value calculation processing, right pupil image and left pupil image acquisition processing Next, the values {b ^R _B ′, b ^L _G , (g _G + r ^R _G )}, {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} is used to determine the value of the component {b ^R _B , b ^L _B } constituting the pixel value B and the component {b ^L _G , g _G , r ^R _G} values and component _^{r R R constituting the pixel values ^R, calculates the value of ^r _{L R}.}

b ^R _B and r ^L _R are obtained from the above equations (17) and (27) as in the following equation (34).
_{^{b R B = (k B1 /}} k B2) × b R B ',
_{^{r L R = (k R1 /}} k R2) × r L R '(34)

b ^L _B and r ^R _R are obtained as in the following equation (35) from g _B << b ^R _B , g _R << r ^L _R and the above equation (13).
b ^L _B = B− (b ^R _B + g _B ) ≈B−b ^R _B ,
_{^{r R R = R- (r L}} R + g R) ≒ R-r L R (35)

b ^L _G and r ^R _G are obtained from the above equation (13) as in the following equation (36).
_{^{b L G = G- (g G}} + r R G),
_{^{r R G = G- (g G}} + b L G) (36)

g _G is obtained from the above equations (13) and (36) as in the following equation (37).
^{_{g G = G- (b L G}} + r R G) (37)

R, G, B component of the right pupil images I ^R and Hidarihitomi image I ^L separates the following equation (38) from the component obtained above.
I ^R = (r ^R _R , r ^R _G , b ^R _B ), I ^L = (r ^L _R , b ^L _G , b ^L _B ) (38)

According to the above embodiment, the optical filter FLT divides the pupil of the imaging optical system into the first pupil (for example, the right pupil) and the second pupil (the left pupil) having a transmission wavelength band different from that of the first pupil. To do. As described in FIG. 14, the imaging device includes a first color (e.g., blue) filter having a first transmission characteristic F _B, and the second color (green) filter having a second transmission characteristic F _G, the third color having a third transmission rate characteristic F _R and a (red) filter. The multiband estimation unit 70 sets the first to fifth bands BD1 to BD5 corresponding to the overlapping portion and the non-overlapping portion of the first to third transmittance characteristics {F _B , F _G , F _R }, and the captured image The first to fifth band component values {b ^R , b ^L , g, r ^R , r ^L } are estimated based on the pixel values {R, G, B} of the first to third colors that constitute. The phase difference image generation unit 20 uses the component values of the band corresponding to the transmission wavelength band of the first pupil among the first to fifth bands BD1 to BD5 as the first image (right pupil image) I ^R = (r ^R _R , r ^R _G , b ^R _B ), and the component value of the band corresponding to the transmission wavelength band of the second pupil among the first to fifth bands is expressed as the second image (left pupil image) I ^L = (r ^L _R , B ^L _G , b ^L _B ).

  In this way, it is possible to estimate the 5-band component values from the image composed of the pixel values of the first to third colors, and to separate the component values into the first image and the second image. Then, the phase difference can be detected by performing a correlation operation on the first image and the second image. In addition, since a plurality of colors can be transmitted through each of the first pupil and the second pupil, it is possible to suppress color misregistration in the defocused image region and improve phase difference detection accuracy in a subject with a biased color.

In this embodiment also, as described in FIG. 14, the multi-band estimator 70 includes a first band BD1 corresponding to the non-overlapping portions of the first transmission characteristic F _B, the first transmittance characteristic F _B No. a second band BD2 which corresponds to the overlapping portion of the two transmittance characteristics F _G, the third band BD3 which corresponds to the non-overlapping portion of the second transmission characteristic F _G, the second transmittance characteristic F _G and third transmission setting a fourth band BD4 corresponding to the overlapping portion of the rate characteristic F _R, and the fifth band BD5 corresponding to the non-overlapping portion of the third transmission characteristic F _R.

  Here, the overlapping portion of the transmittance characteristics is a region where the transmittance characteristics adjacent to each other on the wavelength axis overlap when the transmittance characteristics are expressed with respect to the wavelength axis as shown in FIG. . The overlapping portion is represented by a region where the transmittance characteristics overlap or a band where the bandwidths of the transmittance characteristics overlap. Further, the non-overlapping portion of the transmittance characteristic is a portion that does not overlap with other transmittance characteristics. That is, the portion obtained by removing the overlapping portion from the transmittance characteristic. The band corresponding to the overlapping portion or the non-overlapping portion is not limited to the band itself of the overlapping portion or the non-overlapping portion, and may be any band set corresponding to the overlapping portion or the non-overlapping portion. For example, the first to fifth bands may be set by dividing a band at a wavelength at which a predetermined transmittance and transmittance characteristics intersect.

By doing so, the first to fifth band component values {b ^R , b ^L , g, r ^R , r ^L } are obtained from the pixel values {R, G, B} of the first to third colors of the captured image. Can be estimated. That is, as described in the above equation (13), the pixel values (for example, B and G) having adjacent transmittance characteristics include the component value (b ^L ) of the overlapping portion. By deleting the component value (b ^L ) of this overlapping portion by the difference (B′−G) of the pixel value as in the above equation (19), the relational expression of the component values as in the above equation (21) is obtained. The component value can be estimated based on the relational expression.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. In addition, the configuration and operation of the imaging optical system and the imaging apparatus, the phase difference detection method, the focusing direction determination method, the focus control method, the multiband estimation method, and the like are not limited to those described in this embodiment, and various Can be implemented.

10 imaging unit, 20 phase difference image generating unit, 25 output unit,
30 display image generation unit, 40 monitor display unit, 50 spectral characteristic storage unit,
70 multiband estimation unit, 80 focus control unit, 82 phase difference detection unit,
84 focus control amount calculation unit, 86 focus lens drive control unit,
90 data compression unit, 100 data recording unit,
A ^L , B ^L , A ^R (d), B ^R (d) coefficient,
BD1 to BD5 1st to 5th bands, C (δ _j ) correlation value,
C _{max maximum} correlation value, Df, Dr defocus position,
_{F B,} F G, F _R transmittance characteristics, FL1 right pupil filter,
FL2 Left pupil filter, FLT optical filter, FP focus position,
Fa decreasing section, G pixel value, I (x) captured image,
I ^L (x) Left pupil image, I ^R (x) Right pupil image, LNS imaging lens,
Ra increasing interval, W _j correlation width,
b ^R , b ^L , g, r ^R , r ^L first to fifth band component values,
d shift amount, f ^L , f ^R transmittance characteristics, x position, Δθ declination difference,
Δθ _min Minimum deviation angle difference, δ _j shift amount

Claims (8)

An imaging device that captures a first subject image and a second subject image having parallax with respect to the first subject image, and obtains a captured image;
A phase difference image generation unit that generates a first image corresponding to the first subject image and a second image corresponding to the second subject image based on the captured image;
A phase difference detection unit that performs a correlation operation while relatively shifting the first image and the second image, and detects a phase difference between the first image and the second image;
With
The phase difference detector is
An imaging apparatus, wherein a correlation width, which is a region width for performing a product sum of the correlation calculation, is set according to a shift amount of the shift. In claim 1,
The phase difference detector is
The correlation calculation is repeated by sequentially increasing the shift amount and gradually increasing the correlation width to obtain a correlation value, and detecting the shift amount when the correlation value becomes the maximum value as the phase difference. An imaging device that is characterized. In claim 2,
The phase difference detector is
An imaging apparatus, wherein when the correlation value is obtained while sequentially increasing the shift amount, the shift amount corresponding to the first peak value of the correlation value is detected as the phase difference. In claim 3,
The phase difference detector is
When it is determined that the first peak value has been detected, the correlation calculation is terminated without obtaining the correlation value for the shift amount that is larger than the shift amount when it is determined that the first peak value has been detected. An imaging apparatus characterized by that. In any of claims 2 to 4,
The phase difference detector is
An angular frequency corresponding to the correlation width when the phase difference is detected is set, a Fourier coefficient of a pixel value at the angular frequency is calculated for the first image and the second image, and the first image is calculated for the first image. An imaging apparatus, wherein the shift amount when a difference between a deviation angle of a Fourier coefficient and a deviation angle of the Fourier coefficient of the second image is minimized is detected as a phase difference of subpixels. In any one of Claims 1 thru | or 5,
The phase difference detector is
By comparing the pixel value of the captured image with the pixel value of a comparison image that is at least one of the first image and the second image, the focusing direction of the imaging optical system is determined,
An imaging apparatus, wherein a shift direction for shifting the first image and the second image in the correlation calculation is set based on the determined in-focus direction. In any one of Claims 1 thru | or 6.
A multiband estimator;
An optical filter for dividing the pupil of the imaging optical system into a first pupil and a second pupil having a transmission wavelength band different from that of the first pupil;
With
The image sensor is
A first color filter having a first transmittance characteristic, a second color filter having a second transmittance characteristic, and a third color filter having a third transmittance characteristic,
The multiband estimator is
First to fifth bands corresponding to the overlapping portion and the non-overlapping portion of the first to third transmittance characteristics are set, and the first to third colors are configured based on the first to third color pixel values constituting the captured image. ~ Estimate the component value of the fifth band,
The phase difference image generation unit
The component value of the band corresponding to the transmission wavelength band of the first pupil among the first to fifth bands is acquired as the first image, and the transmission wavelength band of the second pupil among the first to fifth bands. A component value of a band corresponding to is acquired as the second image. Capturing a first subject image and a second subject image having parallax with respect to the first subject image to obtain a captured image;
Generating a first image corresponding to the first subject image and a second image corresponding to the second subject image based on the captured image;
A correlation width that is a region width for performing a product-sum of correlation calculations is set according to a shift amount of the first image and the second image in the correlation calculation,
A phase difference detection method, wherein a phase difference between the first image and the second image is detected by the correlation calculation.

Images