JP2007333720A - Correlation calculation method, correlation calculation device, focus detecting device, and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect the correlation of a pair of signal data rows. <P>SOLUTION: In this correlation calculation method, the degree of correlation between the first signal data row and the second signal data row is calculated. The method performs the first information generating step of generating first calculation data by multiplying first data in the first signal data row by the data near the second data corresponding to the first data in the second signal data row, a second information generating step of generating second calculation data by multiplying second data in the second signal data row by the data near first data in the first signal data row, and the degree-of-correlation detection step of calculating the degree of correlation between the first calculation data and the second calculation data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a correlation calculation method and a correlation calculation device for calculating the correlation of a plurality of signal data strings, and a focus detection device and an imaging device using them.

  There is known an apparatus that forms a light image of a target object on a pair of photoelectric conversion element arrays, and calculates a correlation between a pair of electric signal data strings output from the photoelectric conversion element arrays. For example, a pair of light beams that have passed through different areas of the exit pupil plane of the photographing optical system are received by an image sensor, a pair of light images formed on the image sensor are converted into a pair of electrical signal data strings, and A focus detection device that detects the focus adjustment state of a photographing optical system by calculating the correlation of a signal data string is known (see, for example, Patent Document 1).

Prior art documents related to the invention of this application include the following.
JP 04-338905 A

  However, in the above-described conventional correlation calculation method, a pair of signal data sequences A1, A2,..., AN and B1, B2,..., BN (N is the number of data) are compared while changing the shift amount k. Since the total sum Σ | An−Bn + k | of the absolute values of the differences between the two signal data sequences is simply obtained and used as the correlation amount C (k) between the two signal data sequences, for example, one of the pair of light fluxes has a photographing optical system. When the “vignetting” occurs due to the relative distortion of the pair of signal data strings output from the image sensor, the correlation between the two signal data strings cannot be accurately detected.

(1) In the correlation calculation for calculating the degree of correlation between the first signal data string and the second signal data string, the present invention provides the first data in the first signal data string and the first data in the second signal data string. The first calculation data is generated by multiplying the data in the vicinity of the second data corresponding to the data, and the second data in the second signal data string and the data in the vicinity of the first data in the first signal data string Are multiplied to generate second calculation data, and the degree of correlation between the first calculation data and the second calculation data is calculated.
(2) Further, in the correlation calculation for calculating the degree of correlation between the first signal data string and the second signal data string, the present invention first calculates the first data in the first signal data string and data in the vicinity thereof. 1 operation is performed to calculate operation data, and 2nd operation is performed on the second data corresponding to the first data in the second signal data string and data in the vicinity thereof to calculate operation data, and the operation data To calculate the first calculation data. Next, the first calculation is performed on the second data in the second signal data string and the data in the vicinity thereof to calculate the calculation data, and the second data is calculated in the first data in the first signal data string and the data in the vicinity thereof. An operation is performed to calculate operation data, and the operation data is multiplied to calculate second operation data. Then, the degree of correlation between the first calculation data and the second calculation data is calculated.

  According to the present invention, even if, for example, vignetting due to the photographing optical system occurs in one of the pair of focus detection light beams and relative distortion occurs in the pair of signal data strings output from the image sensor, both It is possible to accurately detect the correlation of the signal data sequence.

<< First Embodiment of the Invention >>
An embodiment in which the present invention is applied to a digital still camera as an imaging apparatus will be described. FIG. 1 is a diagram illustrating a configuration of a digital still camera according to an embodiment. A digital still camera 201 according to an embodiment includes an interchangeable lens 202 and a camera body 203, and the interchangeable lens 202 is attached to a mount portion 204 of the camera body 203.

  The interchangeable lens 202 includes lenses 205 to 207, a diaphragm 208, a lens drive control device 209, and the like. The lens 206 is for zooming, and the lens 207 is for focusing. The lens drive control device 209 includes a CPU and its peripheral components, and controls the driving of the focusing lens 207 and the aperture 208, detects the positions of the zooming lens 206, the focusing lens 207 and the aperture 208, and communicates with the control device of the camera body 203. Transmit lens information and receive camera information.

  On the other hand, the camera body 203 includes an image sensor 211, a camera drive control device 212, a memory card 213, an LCD driver 214, an LCD 215, an eyepiece 216, and the like. The imaging element 211 is disposed on the planned image plane (planned focal plane) of the interchangeable lens 202, captures the subject image formed by the interchangeable lens 202, and outputs an image signal. An imaging pixel (hereinafter simply referred to as an imaging pixel) is two-dimensionally arranged in the imaging element 211, and a focus detection pixel (hereinafter, referred to as a focus detection position) is replaced with an imaging pixel in a portion corresponding to the focus detection position. A column (simply called focus detection pixels) is incorporated.

  The camera drive control device 212 includes peripheral components such as a CPU and a memory, and controls the drive of the image sensor 211, processing of the captured image, focus detection and focus adjustment of the interchangeable lens 202, control of the aperture 208, display control of the LCD 215, lens drive Communication with the control device 209, sequence control of the entire camera, and the like are performed. The camera drive control device 212 communicates with the lens drive control device 209 via an electrical contact 217 provided on the mount unit 204.

  The memory card 213 is an image storage that stores captured images. The LCD 215 is used as a display of a liquid crystal viewfinder (EVF: electronic viewfinder), and a photographer can visually recognize a captured image displayed on the LCD 215 via an eyepiece lens 216.

  The subject image that has passed through the interchangeable lens 202 and formed on the image sensor 211 is photoelectrically converted by the image sensor 211, and the image output is sent to the camera drive controller 212. The camera drive control device 212 calculates the defocus amount at the focus detection position based on the output of the focus detection pixel, and sends this defocus amount to the lens drive control device 209. Further, the camera drive control device 212 sends an image signal generated based on the output of the imaging pixel to the LCD driver 214, displays it on the LCD 215, and stores it in the memory card 213.

  The lens drive control device 209 detects the positions of the zooming lens 206, the focusing lens 207, and the diaphragm 208 and calculates lens information based on the detected positions, or a lens corresponding to the detected position from a lookup table prepared in advance. Information is selected and sent to the camera drive controller 212. Further, the lens drive control device 209 calculates the lens drive amount based on the defocus amount received from the camera drive control device 212, and drives and controls the focusing lens 207 based on the lens drive amount.

  FIG. 2 shows a focus detection area on the imaging screen G set on the planned imaging plane of the interchangeable lens 202. The focus detection areas G1 to G5 are set on the imaging screen G, and the focus detection pixels of the imaging element 211 are linearly arranged in the longitudinal direction of the focus detection areas G1 to G5 on the imaging screen G. That is, the focus detection pixel array on the image sensor 211 samples the images of the focus detection areas G1 to G5 in the subject image formed on the shooting screen G. The photographer manually selects an arbitrary focus detection area from the focus detection areas G1 to G5 according to the shooting composition.

  FIG. 3 is a front view showing a detailed configuration of the image sensor 211. FIG. 3 is a partially enlarged view in which the periphery of one focus detection region on the image sensor 211 is enlarged. The image pickup device 211 includes an image pickup pixel 310 and a focus detection pixel 311 for focus detection.

  As shown in FIG. 4, the imaging pixel 310 includes the microlens 10, the photoelectric conversion unit 11, and a color filter (not shown). As shown in FIG. 5, the focus detection pixel 311 includes a microlens 10 and a pair of photoelectric conversion units 12 and 13. The photoelectric conversion unit 11 of the imaging pixel 310 is designed in such a shape that the microlens 10 receives all the light beams that pass through the exit pupil (for example, F1.0) of a bright interchangeable lens. On the other hand, the pair of photoelectric conversion units 12 and 13 of the focus detection pixel 311 are designed so as to receive all light beams passing through a specific exit pupil (for example, F2.8) of the interchangeable lens by the microlens 10.

  The imaging pixels 310 arranged two-dimensionally are provided with red (R), green (G), or blue (B) color filters, and each color filter has the spectral sensitivity characteristics shown in FIG. Yes. The imaging pixels 310 having RGB color filters are arranged in a Bayer array as shown in FIG.

  On the other hand, the focus detection pixel 311 is not provided with a color filter in order to increase the amount of light, and the spectral sensitivity characteristics thereof are the spectral sensitivity of a photodiode that performs photoelectric conversion and the spectral sensitivity of an infrared cut filter (not shown). The spectral sensitivity characteristics shown in FIG. The spectral sensitivity of the focus detection pixel 311 has a spectral sensitivity characteristic that is obtained by adding the spectral sensitivities of the green pixel G, the red pixel R, and the blue pixel B in the imaging pixel 310 shown in FIG. The light wavelength region of the sensitivity of the green pixel G, the red pixel R, and the blue pixel B is included.

  The focus detection pixels 311 are arranged densely in a straight line without a gap in the rows or columns where the B filters and G filters of the imaging pixels 310 in the focus detection regions G1 to G5 shown in FIG. When the focus detection pixel 311 is arranged in a row or a column where the B filter and the G filter of the imaging pixel 310 should be arranged, an error occurs when the pixel signal at the position of the focus detection pixel 311 is calculated by pixel interpolation. But it can be inconspicuous to human eyes. This is because the human eye is more sensitive to red than blue and because the density of green pixels is higher than that of blue and red pixels, the contribution to image degradation for a single pixel defect in the green pixel is small. .

  Note that, in the above-described imaging element in which the imaging pixels of the complementary color filter are developed two-dimensionally, the focus detection pixels 311 are arranged at pixel positions where cyan and magenta including a blue component whose output error is relatively inconspicuous should be arranged.

  FIG. 8 is a cross-sectional view of the imaging pixel 310. In the imaging pixel 310, the microlens 10 is disposed in front of the photoelectric conversion unit 11 for imaging, and the photoelectric conversion unit 11 is projected forward by the microlens 10. The photoelectric conversion unit 11 is formed on the semiconductor circuit substrate 29, and a color filter (not shown) is disposed between the microlens 10 and the photoelectric conversion unit 11.

  FIG. 9 is a cross-sectional view of the focus detection pixel 311. In the focus detection pixel 311, the microlens 10 is disposed in front of the focus detection photoelectric conversion units 12 and 13, and the photoelectric conversion units 12 and 13 are projected forward by the microlens 10. The photoelectric conversion units 12 and 13 are formed on the semiconductor circuit substrate 29.

  Next, a focus detection method based on the pupil division method will be described with reference to FIG. In FIG. 10, the microlens 50 of the focus detection pixel 311 disposed on the optical axis 91 of the interchangeable lens 202, the pair of photoelectric conversion units 52 and 53 disposed behind the microlens 50, and the interchangeable lens 202. The microlens 60 of the focus detection pixel 311 disposed outside the optical axis 91 and the pair of photoelectric conversion units 62 and 63 disposed behind the microlens 60 will be described as an example. The exit pupil 90 of the interchangeable lens 202 is set at a position of a distance d4 in front of the microlenses 50 and 60 arranged on the planned image formation surface of the interchangeable lens 202. Here, the distance d4 is a value determined according to the curvature and refractive index of the microlenses 50 and 60, the distance between the microlenses 50 and 60 and the photoelectric conversion units 52, 53, 62, and 63, and the like. In the specification, this is called a distance measuring pupil distance.

  The microlenses 50 and 60 are arranged on the planned imaging plane of the interchangeable lens 202, and the shape of the pair of photoelectric conversion units 52 and 53 is separated from the microlens 50 by the projection distance d4 by the microlens 50 on the optical axis 91. The projected image is projected onto the exit pupil 90, and the projection shape forms distance measuring pupils 92 and 93. On the other hand, the shape of the pair of photoelectric conversion units 62 and 63 is projected on the exit pupil 90 separated by the projection distance d4 by the microlens 60 outside the optical axis 91, and the projection shape forms the distance measuring pupils 92 and 93. That is, the projection direction of each pixel is determined so that the projection shapes (ranging pupils 92 and 93) of the photoelectric conversion units of the focus detection pixels coincide on the exit pupil 90 at the projection distance d4.

  The photoelectric conversion unit 52 outputs a signal corresponding to the intensity of the image formed on the microlens 50 by the focus detection light beam 72 passing through the distance measuring pupil 92 and traveling toward the microlens 50. The photoelectric conversion unit 53 outputs a signal corresponding to the intensity of the image formed on the microlens 50 by the focus detection light beam 73 passing through the distance measuring pupil 93 and traveling toward the microlens 50. The photoelectric conversion unit 62 outputs a signal corresponding to the intensity of the image formed on the microlens 60 by the focus detection light beam 82 passing through the distance measuring pupil 92 and traveling toward the microlens 60. The photoelectric conversion unit 63 outputs a signal corresponding to the intensity of the image formed on the microlens 60 by the focus detection light beam 83 passing through the distance measuring pupil 93 and traveling toward the microlens 60. Note that the arrangement direction of the focus detection pixels 311 is made to coincide with the division direction of the pair of pupil distances.

  A large number of such focus detection pixels are arranged in a straight line, and the output of the pair of photoelectric conversion units of each pixel is collected into an output group corresponding to the distance measurement pupil 92 and the distance measurement pupil 93, thereby forming a pair of distance measurement pupils 92. And 93, information relating to the intensity distribution of a pair of images formed on the focus detection pixel column by the focus detection light fluxes passing through each of them can be obtained. Furthermore, by applying an image shift detection calculation process (correlation process, phase difference detection process) to be described later to this information, the image shift amount of a pair of images can be detected by a so-called pupil division method. Then, by multiplying this image shift amount by a predetermined conversion coefficient, the current imaging plane (imaging plane at the focus detection position corresponding to the position of the microlens array on the planned imaging plane) with respect to the planned imaging plane is determined. Deviation (defocus amount) can be calculated.

  In FIG. 10, the first focus detection pixel (the microlens 60 and the pair of photoelectric conversion units 62) adjacent to the first focus detection pixel (the microlens 50 and the pair of photoelectric conversion units 52 and 53) on the optical axis 91. 63) is also schematically illustrated, but in other focus detection pixels as well, similarly, a pair of photoelectric conversion units receive light beams coming from the pair of distance measuring pupils to the respective microlenses.

  FIG. 11 is a front view showing a projection relationship on the exit pupil plane. The circumscribed circle of the distance measuring pupils 92 and 93 obtained by projecting the pair of photoelectric conversion units 12 and 13 from the focus detection pixel 311 onto the exit pupil plane 90 by the microlens 10 is a predetermined aperture F value (when viewed from the imaging plane). Hereinafter, it is referred to as a distance measuring pupil F value, which is F2.8). When the photoelectric conversion unit 11 of the imaging pixel 310 is projected onto the exit pupil plane 90 by the microlens 10, the region 94 is formed, and the region 94 is a wide region including the distance measuring pupils 92 and 93.

  In FIG. 11, the positional relationship between the center of the region 95 corresponding to the aperture opening of the interchangeable lens 202 indicated by the broken line and the center of the circumscribed circle of the distance measuring pupils 92 and 93 is the position and focal point of the exit pupil unique to the interchangeable lens 202. The detection pixel changes according to the position on the screen (distance from the optical axis) and does not necessarily match. When the size of the exit pupil of the interchangeable lens 202 is smaller than the size of the circumscribed circle of the distance measuring pupils 92 and 93 and the centers do not coincide with each other, the light beams passing through the pair of distance measuring pupils 92 and 93 are unbalanced. “Vignetting” occurs, and the amount of light of the pair of images formed by these light beams does not match, resulting in distortion.

  FIG. 12 shows the intensity distribution (light quantity) of the image signal at one focus detection position on the vertical axis and the position deviation within the focus detection position on the horizontal axis. The position deviation within the focus detection position corresponds to the positions of a plurality of focus detection pixels belonging to one focus detection position on the image sensor 211 shown in FIG. 3, for example. In the pair of image signals 400 and 401 when no vignetting occurs in the focus detection light beam, the same image signal function is simply shifted horizontally as shown in FIG. When vignetting occurs in the focus detection light beam, the amount of the focus detection light beam passing through the distance measuring pupil changes depending on the focus detection position and the position deviation within the focus detection position, and the pair of image signals 402 and 403 are shown in FIG. Thus, the same signal is not relatively shifted.

《Imaging operation》
FIG. 13 is a flowchart illustrating an imaging operation of the digital still camera (imaging device) 210 according to the embodiment. The camera drive control device 212 repeatedly executes this imaging operation when the camera is turned on in step 100. In step 110, the data of the imaging pixel 310 is read out from the imaging device 211 and displayed on the electronic viewfinder LCD 215. When the data of the imaging pixel 310 is thinned and read out, the display quality can be improved by performing the thinning out reading with a setting that does not include the focus detection pixel 311 as much as possible. Conversely, by performing thinning readout so as to include the focus detection pixel 311 and displaying it on the electronic viewfinder LCD 215 without correcting the focus detection pixel output, the focus detection position can be displayed recognizable to the user. .

  In step 120, data is read from the focus detection pixel array. Since an arbitrary area is selected in advance from the focus detection areas G1 to G5 shown in FIG. 2 by an area selection operation member (not shown), data is acquired from the focus detection pixel row corresponding to the selected focus detection area. Is read. In subsequent step 130, based on a pair of image data corresponding to the focus detection pixel row, an image shift detection calculation process, that is, a correlation calculation process, which will be described later, is performed to calculate the image shift amount, and further calculate the defocus amount. In step 140, it is checked whether or not the focus is close, that is, whether or not the calculated absolute value of the defocus amount is within the focus determination reference value.

  If it is determined that the lens is not in focus, the process proceeds to step 150, the calculated defocus amount is transmitted to the lens drive control device 209, the focusing lens 207 of the interchangeable lens 202 is driven to the focus position, and the process returns to step 110. Repeat the above operation. Even when focus detection is impossible, the process branches to step 150, a scan drive command is transmitted to the lens drive control device 209, and the focusing lens 207 of the interchangeable lens 202 is driven to scan between the infinite position and the closest position. Returning to 110, the above operation is repeated.

  On the other hand, if it is determined that it is close to the in-focus state, the process proceeds to step 160 to determine whether or not a shutter release has been performed by operating a release button (not shown). If not, the process returns to step 110 to perform the above operation. repeat. When the shutter release is performed, the process proceeds to step 170, an aperture adjustment command is transmitted to the lens drive control device 209, and the control F value determined by the exposure calculation of the aperture 208 of the interchangeable lens 202 by the camera drive control device 212, or the user Set the manually set F value.

  When the aperture control is completed, the image sensor 211 is caused to perform an imaging operation, and image data is read from the image pickup pixel 310 and all the focus detection pixels 311 of the image pickup element 211. In step 180, pixel data at each pixel position in the focus detection pixel row is interpolated based on the data of the focus detection pixel 311 and the data of the surrounding imaging pixels 310. In step 190, image data composed of the data of the imaging pixel 310 and the interpolated focus detection pixel position data is stored in the memory card 213, and the process returns to step 110 to repeat the above operation.

<Focus detection operation>
FIG. 14 is a flowchart showing details of the focus detection operation in step 130 in the imaging operation shown in FIG. In step 300, focus detection calculation processing, that is, correlation calculation processing is started. In step 310, a pair of signal data sequences (α1 to αM, β1 to βM: M is the number of data) are output as follows. High frequency cut filter processing as shown in equation (1) is performed to generate first signal data strings A1 to AN and second signal data strings B1 to BN (N is the number of data), and adversely affect correlation processing from the signal data strings. Removes noise components and high frequency components.
An = αn + 2 × αn + 1 + αn + 2,
Bn = βn + 2 × βn + 1 + βn + 2 (1)
In the formula (1), n = 1 to N. Note that the high-frequency cut filter processing in step 310 can be omitted if the calculation time is to be shortened or if it is already defocused and it is known that there are few high-frequency components.

  In step 320, in order to calculate the correlation amount between the first signal data string A1 to AN and the second signal data string B1 to BN at the shift amount k (integer), first, the first signal data string A1 to AN is calculated. The second signal data strings B1 to BN are relatively shifted by a predetermined amount k.

  In the subsequent step 330, data in an arbitrary range in the first signal data string A1 to AN, that is, the data An of interest and its neighboring data An-r to An + s (r and s are arbitrary integers; A first calculation (detailed later) is performed on this range (referred to as “small range”) to obtain calculation data, and the first signal data in the second signal data strings B1 to BN shifted by the shift amount k. A second calculation (detailed later) is performed on the data Bn + k corresponding to the attention data An in the column and the minute range data Bn-r to Bn + s in the vicinity thereof to obtain calculation data. Then, the first calculation data is calculated by multiplying the calculation data.

  In step 340, the first calculation is performed on the minute range data Bn-r to Bn + s in the second signal data strings B1 to BN to obtain calculation data, and the first signal data strings A1 to AN are obtained. The second calculation is performed on the minute range data An-r to An + s of the data to obtain calculation data. Then, the second calculation data is calculated by multiplying the calculation data.

  As will be described in detail later, the first calculation data and the second calculation data of the calculation results in steps 330 and 340 cancel out the distortion of the first signal data string and the second signal data string in any “small range”. Data.

  In steps 330 and 340 described above, an example is shown in which the first calculation and the second calculation are performed on the “small range” of the first signal data string and the second signal data string to obtain calculation data. Data in the vicinity of the data Bn + k corresponding to the position of the data of interest An in any position of the 1 signal data string A1 to AN and the position of the data of interest An in the second signal data string B1 to BN shifted by the shift amount k; For example, the first operation data is generated by multiplying Bn + 1 + k or Bn + 2 + k, and the data Bn + k of the second signal data strings B1 to BN and the first signal data strings A1 to AN. The second calculation data may be generated by multiplying the data in the vicinity of the attention data An, for example, An + 1 or An + 2.

  In step 350, the absolute value of the difference between the first calculation data and the second calculation data is calculated. This calculation result is an amount indicating the degree of correlation (similarity) between the first signal data string and the second signal data string in the “small range”. In the following step 360, the processing of steps 330 to 350 is repeated while shifting the “small range”, and the calculation result of the amount indicating the degree of correlation in step 350 is cumulatively added, whereby the correlation amount C (k) at the predetermined shift amount k. Is calculated.

In step 370, the process of steps 320 to 360 is repeated while changing the shift amount k within a predetermined range, and the correlation amount C (K) data relating to the shift amount k is calculated. Here, assuming that the first calculation data at the center position n of the shift amount k and the “small range” is En, k and the second calculation data is Fn, k, the finally obtained correlation amount C (K) is It is expressed as the following formula (2).
C (k) = Σ | En, k−Fn, k | (2)
In equation (2), the Σ operation is accumulated for n, and the range taken by n is limited to the range in which data of En, k and Fn, k exist according to the shift amount k. The shift amount k is an integer, and is a relative shift amount in units of the detection pitch of a pair of data.

In step 380, as shown in FIG. 15 (a), for example, as shown in FIG. Correlation degree is high). A shift amount x that gives a minimum value C (x) with respect to a continuous correlation amount is obtained using a three-point interpolation method according to the following equations (3) to (6).
x = kj + D / SLOP (3),
C (x) = C (kj) − | D | (4),
D = {C (kj-1) -C (kj + 1)} / 2 (5),
SLOP = MAX {C (kj + 1) -C (kj), C (kj-1) -C (kj)} (6)

In step 390, the defocus amount DEF of the subject image plane with respect to the planned image formation plane can be obtained by the following equation (7) based on the shift amount x obtained by the equation (3).
DEF = KX · PY · x (7)
In equation (7), PY is a detection pitch, and KX is a conversion coefficient determined by the size of the opening angle of the center of gravity of the pair of distance measuring pupils. Whether or not the calculated defocus amount DEF is reliable is determined as follows. As shown in FIG. 15B, when the correlation between the pair of data is low, the value of the interpolated minimum amount C (x) of the correlation amount becomes large. Therefore, when C (x) is equal to or greater than a predetermined value, it is determined that the reliability is low. Alternatively, in order to normalize C (x) with the contrast of data, if the value obtained by dividing C (x) by SLOP that is proportional to the contrast is equal to or greater than a predetermined value, it is determined that the reliability is low. Alternatively, when SLOP that is a value proportional to the contrast is equal to or less than a predetermined value, it is assumed that the subject has low contrast and the reliability of the calculated defocus amount DEF is low.

  As shown in FIG. 15C, when the correlation between the pair of data is low and there is no drop in the correlation amount C (k) between the shift ranges kmin to kmax, the minimum value C (x) is obtained. In such a case, it is determined that the focus cannot be detected. When focus detection is possible, the defocus amount is calculated by multiplying the calculated image shift amount by a predetermined conversion coefficient. In step 400, the focus detection calculation process (correlation calculation process) is terminated and the process returns.

《Correlation operation》
Next, an example of the correlation calculation process shown in FIG. 14 will be described. FIG. 16 is a diagram for explaining the concept of correlation calculation according to an embodiment. The interchangeable lens 202 is divided into first signal data strings A1 to AN indicated by solid lines and second signal data strings B1 to BN indicated by broken lines. A case where distortion caused by vignetting or the like occurs and a gain difference occurs between the waveforms of both signal data strings is shown. Even when there is such a gain difference, the “small range” characteristics in each signal data string, that is, the ratios An / An + 1 and Bn + k / Bn + 1 + k of adjacent data are large. Not affected. Since these “minor range” characteristics An / An + 1 and Bn + k / Bn + 1 + k include a denominator part, these “minor range” characteristics include common terms (An + 1 × By multiplying by (Bn + 1 + k), it is possible to suppress the influence of divergence when the denominator becomes 0 and the influence of noise when the value of the denominator is small.

  For example, the ratio An / An + 1, Bn + k / Bn + 1 + k of adjacent data sizes is multiplied by a common term (An + 1 × Bn + 1 + k) in the denominator portion, and An × Bn + 1 + k, Bn + k × An + 1 are obtained. By taking the correlation between the characteristics of the “small range” between the two signal data strings calculated in this way, in this case the absolute value of the difference, the degree of correlation (similarity) between the two signal data strings is increased. It becomes possible to detect with accuracy. Furthermore, by accumulating the degree of correlation while shifting the position of the “small range” within a predetermined range, it is possible to eliminate the accidentally generated correlation and perform highly accurate correlation detection.

  Arbitrary data An in the first signal data sequence A1 to AN and data in the vicinity of the data Bn + k corresponding to the data An in the second signal data sequence B1 to BN shifted by the shift amount k, for example, one adjacent Bn + 1 + k or two adjacent data Bn + 2 + k is multiplied to generate the first operation data (An × Bn + 1 + k) or (An × Bn + 2 + k), and the second The data Bn + k of the signal data sequence B1 to BN and the data in the vicinity of the data An of interest of the first signal data sequence A1 to AN, for example, the next one An + 1 or the next two An + 2 To generate second operation data (Bn + k × An + 1) or (Bn + k × An + 2). Then, the correlation between the first signal data string and the second signal data string is obtained by the above equation (2) based on the first calculation data and the second calculation data. This correlation calculation example will be described later as Processing Example 1 and Processing Example 2.

As another example of the correlation calculation described above, for an arbitrary range of data in the first signal data string A1 to AN, that is, the data An of interest and the minute range data An-r to An + s in the vicinity thereof. The calculation data is obtained by performing the first calculation, and the data Bn + k corresponding to the attention data An in the first signal data string in the second signal data strings B1 to BN shifted by the shift amount k and the vicinity thereof A second calculation is performed on the minute range data Bn-r to Bn + s to obtain calculation data. Then, the first calculation data is calculated by multiplying the calculation data.
Further, the first calculation is performed on the minute range data Bn-r to Bn + s in the second signal data strings B1 to BN to obtain calculation data, and the first signal data strings A1 to AN The second calculation is performed on the minute range data An-r to An + s to obtain calculation data. Then, the second calculation data is calculated by multiplying the calculation data.
Based on the first calculation data and the second calculation data, the correlation between the first signal data string and the second signal data string is obtained by the above equation (2). This correlation calculation example will be described later as Processing Example 3 to Processing Example 8.

<< Correlation calculation processing example 1 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (An × Bn + 1 + k) − (Bn + k × An + 1) | (8)
In equation (8), the Σ operation is accumulated for n, and the range taken by n is limited to a range in which data of An, An + 1, Bn + k, and Bn + 1 + k exist according to the shift amount k. . FIG. 17 shows a data flow of correlation calculation processing example 1. In this processing example 1, the target data An in the first signal data string A is shifted by one from the data Bn + k corresponding to the target data An in the second signal data string B shifted by the shift amount k. The data Bn + 1 + k is multiplied to obtain the first calculation data (An × Bn + k), and one piece is obtained from the data Bn + k of the second signal data string B and the attention data An of the first signal data string A. The second calculation data (Bn + k × An + 1) is obtained by multiplying the shifted data An + 1, and the first signal data string A is obtained by the sum of absolute values of the difference between the first calculation data and the second calculation data. A correlation amount C (k) of the second signal data string B is calculated.

  Here, a case where the correlation calculation processing example 1 is performed on a pair of image signal data strings A and B as shown in FIG. 18 will be considered. A relative distortion occurs in a pair of image signal data strings (A1, A2,..., AN), (B1, B2,..., BN) output from the image sensor 211, and as shown in FIG. It is assumed that there is a difference between the gain and inclination of the waveform (1 + sinω) of both image signal data strings. FIG. 19 shows the result of executing the correlation calculation processing example 1 on these two signal data strings A and B. As is clear from the figure, it can be seen that, even though the two signal data strings A and B are distorted, the correlation calculation processing example 1 can obtain a high correlation (a depressed position) periodically.

  According to this correlation calculation processing example 1, the above-described conventional correlation calculation method, that is, simply obtaining the sum Σ | An−Bn + k | of the absolute values of the differences between the two signal data strings A and B is used to Compared with the correlation amount C (k), the correlation amount between the two signal data strings A and B can be accurately obtained even when relative distortion occurs in the two signal data strings A and B. it can. Further, in this correlation calculation processing example 1, since the correlation calculation formula (8) is not in the fractional format, there is no divergence of calculation due to the denominator being zero, and noise in the case where the denominator is close to zero. There is no effect.

<< Correlation calculation processing example 2 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (An × Bn + 2 + k) − (Bn + k × An + 2) | (9)
In equation (9), the Σ operation is accumulated for n, and the range taken by n is limited to a range in which data of An, An + 2, Bn + k, and Bn + 2 + k exist according to the shift amount k. . FIG. 20 shows a data flow of the correlation calculation processing example 2. In this correlation calculation processing example 2, the calculation target data interval is further expanded as compared with the correlation calculation processing 1 shown in the equation (8).

  FIG. 21 shows the result of executing the correlation calculation process 2 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is apparent from the figure, it can be seen that, even though the two signal data strings A and B are distorted, the correlation calculation processing example 2 can provide a high correlation (a depressed position) periodically. According to the correlation calculation process 2, in addition to the above-described effects of the correlation calculation process 1, the data Bn + 2 + k and An + 2 at positions shifted by two from the attention data An and Bn + k are set as calculation targets. Since the target data interval is made wider, the influence of high frequency components and noise included in the signal data string can be alleviated, and noise resistance performance is improved.

<< Correlation calculation processing example 3 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (An−An + 1) × (Bn + k + Bn + 1 + k) − (Bn + k−Bn + 1 + k) × (An + An + 1) |
... (10)
In equation (10), the Σ operation is accumulated for n, and the range taken by n is limited to a range in which data of An, An + 1, Bn + k, and Bn + 1 + k exist according to the shift amount k. . FIG. 22 shows a data flow of the correlation calculation processing example 3.

  As is apparent from the equation (10), this correlation calculation processing example 3 is the difference between adjacent data with respect to the data of interest An in the first signal data string A and the minute range data An to An + 1 in the vicinity thereof, that is, A first derivative (first calculation) is performed to obtain calculation data (An−An + 1), and the attention data An of the first signal data string in the second signal data string B shifted by the shift amount k is added to the attention data An. Calculation data (Bn + k + Bn + 1 + k) is obtained by adding or averaging (second calculation) to the corresponding data Bn + k and the minute range data Bn + k to Bn + 1 + k in the vicinity thereof. . Then, the calculation data is multiplied to calculate the first calculation data (An−An + 1) × (Bn + k + Bn + 1 + k).

  Further, the difference data of the adjacent data, that is, the primary differentiation (first calculation) is performed on the minute range data Bn + k to Bn + 1 + k in the second signal data string B to obtain calculation data (Bn + k−). Bn + 1 + k) is obtained, and addition data, that is, averaging (second operation) is performed on the minute range data An to An + 1 in the first signal data string A to obtain operation data (An + An + 1). Get. Then, the second operation data (Bn + k−Bn + 1 + k) × (An + An + 1) is calculated by multiplying the operation data. Then, the correlation between the first signal data string and the second signal data string is obtained by the above equation (2) based on the first calculation data and the second calculation data.

  FIG. 23 shows a result of executing the correlation calculation process 3 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is clear from the figure, it can be seen that, although the two signal data strings A and B are distorted, the correlation calculation processing example 3 can obtain a high correlation (a depressed position) periodically. According to this correlation calculation process 3, in addition to the above-described effects of the correlation calculation process 1, the ratio between the difference (first derivative) of adjacent data and the average value of adjacent data is used as the characteristic value of the signal data string. The influence of high frequency components and noise included in the signal data string can be reduced, and the noise resistance performance is improved. Further, the DC components included in the signal data sequences A and B can be removed, and even when the two signal data sequences A and B include a DC offset, highly accurate correlation detection can be performed.

<< Correlation calculation process 4 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (An−An + 2) × (Bn + k + Bn + 2 + k) − (Bn + k−Bn + 2 + k) × (An + An + 2) |
(11)
In equation (11), the Σ operation is accumulated for n, and the range taken by n is limited to a range in which data of An, An + 2, Bn + k, and Bn + 2 + k exist according to the shift amount k. . FIG. 24 shows a data flow of the correlation calculation process 4. The correlation calculation process 4 is an operation in which the data interval to be calculated is wider than the correlation calculation process 3 shown in the equation (10).

  FIG. 25 shows the result of executing the correlation calculation process 4 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is clear from the figure, it can be seen that, although the two signal data strings A and B are distorted, the correlation calculation processing example 4 can obtain a high correlation (a depressed position) periodically. According to this correlation calculation process 4, in addition to the above-described effects of the correlation calculation process 1, the calculation target data interval is made wider, so that the influence of high frequency components and noise included in the signal data string can be reduced. This improves noise resistance.

<< Correlation calculation process 5 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (2 × An−An−1−An + 1) × (Bn−1 + k + Bn + k + Bn + 1 + k) − (2 × Bn + k−Bn−1 + k−Bn + 1 + k) × (An−1 + An + An + 1) | (12)
In the equation (12), the Σ operation is accumulated for n, and the range taken by n is An-1, An, An + 1, Bn-1 + k, Bn + k, Bn + 1 + k according to the shift amount k. It is limited to the range where the data of exists. FIG. 26 shows a data flow of the correlation calculation process 5. This correlation calculation process 5 employs a ratio between the difference between adjacent data (secondary differential; corresponding to the first calculation) and the average value of the adjacent data (corresponding to the second calculation) as the characteristic value of the signal data string. It is a correlation calculation formula in the case.

  FIG. 27 shows the result of executing the correlation calculation processing 5 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is clear from the figure, it can be seen that, even though the two signal data strings A and B are distorted, the correlation calculation processing example 5 can obtain a high correlation (a depressed position) periodically. According to the correlation calculation process 5, in addition to the above-described effects of the correlation calculation process 1, the influence of high frequency components and noise included in the signal data sequences A and B can be reduced, so that the noise resistance performance is improved. Furthermore, the DC component and the primary gradient component included in the signal data sequences A and B can be removed, and even when the two signal data sequences include a difference in DC offset or primary gradient component, high accuracy is achieved. Correlation detection is possible.

<< Correlation calculation process 6 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (2 × An−An−2−An + 2) × (Bn−2 + k + Bn + k + Bn + 2 + k) − (2 × Bn + k−Bn−2 + k−Bn + 2 + k) × (An-2 + An + An + 2) | (13)
In equation (13), the Σ operation is accumulated for n, and the range taken by n is An-2, An, An + 2, Bn-2 + k, Bn + k, Bn + 2 + k according to the shift amount k. It is limited to the range where the data of exists. FIG. 28 shows a data flow of the correlation calculation processing 6. The correlation calculation process 6 is an operation in which the calculation target data interval is wider than the correlation calculation process 5 shown in the equation (12).

  FIG. 29 shows the result of executing the correlation calculation process 6 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is apparent from the figure, it can be seen that, even though the two signal data strings A and B are distorted, the correlation calculation processing example 6 can obtain a high correlation (a depressed position) periodically. According to this correlation calculation processing example 6, in addition to the above-described effects of the correlation calculation processing example 5, the data interval to be calculated is made wider, so that the influence of high frequency components and noise included in the signal data string is reduced. Noise resistance performance can be improved.

<< Correlation calculation process 7 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (An ² × Bn−1 + k × Bn + 1 + k) − (Bn + k ² × An−1 × An + 1) | (14)
In the equation (14), the Σ operation is accumulated for n, and the range taken by n is An-1, An, An + 1, Bn-1 + k, Bn + k, Bn + 1 + k according to the shift amount k. It is limited to the range where the data of exists. FIG. 30 shows a data flow of the correlation calculation processing example 7.

  FIG. 31 shows the result of executing correlation calculation processing example 7 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is clear from the figure, it can be seen that, even though the two signal data strings A and B are distorted, the correlation calculation processing example 7 provides a high correlation (a depressed position) periodically. According to this correlation calculation processing example 7, since the correlation calculation formula (14) is not in a fractional format, there is no divergence of calculation due to the denominator being zero, and noise in the case where the denominator is close to zero. There is no effect.

<< Correlation calculation processing example 8 >>
The correlation amount C (k) is obtained by the following equation.
C (k) = Σ | (An ² × Bn−2 + k × Bn + 2 + k) − (Bn + k ² × An−2 × An + 2) | (15)
The Σ operation is accumulated for n, and the range that n takes is a range in which data of An−2, An, An + 2, Bn−2 + k, Bn + k, and Bn + 2 + k exist according to the shift amount k. It is limited to. In this correlation calculation processing example 8, the calculation target data interval is wider than the correlation calculation processing 7 shown in equation (14). FIG. 32 shows a data flow of the correlation calculation processing example 8.

  FIG. 33 shows a result of executing the correlation calculation processing example 8 on the signal data strings A and B shown in FIG. 18 in which relative distortion has occurred in the two signal data strings A and B. As is apparent from the figure, it can be seen that, although the two signal data strings A and B are distorted, the correlation calculation processing example 8 can provide a high correlation (a depressed position) periodically. According to the correlation calculation processing example 8, in addition to the above-described effects of the correlation calculation processing example 7, the data interval to be calculated is made wider, so that the influence of high frequency components and noise included in the signal data string is reduced. Noise resistance performance can be improved.

<< Modification of First Embodiment >>
3 shows an example in which the focus detection pixels 311 are arranged without a gap, but FIG. 34 shows an image pickup element in which the focus detection pixels 311 are arranged in a line at every other pixel position of the blue pixels of the imaging pixel 310. An example of 211A is shown. Although the density of the focus detection pixels 311 is reduced due to the fact that the focus detection accuracy is slightly reduced by increasing the arrangement interval of the focus detection pixels 311, the image quality obtained by interpolating the image signal at the position of the focus detection pixels 311 is improved. improves.

  In the imaging device 211 illustrated in FIG. 3, an example is illustrated in which a pair of photoelectric conversion units 12 and 13 are provided for each focus detection pixel 311 as illustrated in FIG. 5. FIG. 35 shows focus detection pixels 313 and 314 each having one photoelectric conversion unit in one pixel. The focus detection pixel 313 includes the microlens 10 and the photoelectric conversion unit 16 as illustrated in FIG. Further, the focus detection pixel 314 includes the microlens 10 and the photoelectric conversion unit 17 as illustrated in FIG. The photoelectric conversion units 16 and 17 are projected onto the exit pupil of the interchangeable lens by the microlens 10 to form distance measuring pupils 92 and 93 shown in FIG. Therefore, a pair of image outputs used for focus detection can be obtained by the focus detection pixels 313 and 314.

  FIG. 36 shows an example of an image sensor 211B in which the focus detection pixels 313 and 314 shown in FIGS. 35A and 35B are alternately arranged in a line. A pair of adjacent focus detection pixels 313 and focus detection pixels 314 correspond to the focus detection pixels 311 of the image sensor 211 shown in FIG. 3, and a pair of image outputs used for focus detection can be obtained.

  In the image pickup device 211 shown in FIG. 3, as shown in FIG. 37A, an example in which the image pickup pixels 310 provided with R, G, and B color filters are arranged in a Bayer array is shown. Is not limited to the embodiment described above. For example, as shown in FIG. 37B, G (green), yellow (Ye), magenta (Mg), and cyan (Cy) may be arranged in a complementary color arrangement. When this complementary color filter is used, the focus detection pixel 311 is arranged at a pixel position where cyan and magenta including a blue component whose output error is relatively inconspicuous should be arranged.

  In the image sensor 211 shown in FIG. 3, an example in which the focus detection pixel 311 is not provided with a color filter is shown. However, even if one color filter, for example, a green filter, is provided among the color filters of the same color as the image pickup pixel 310, The invention can be applied.

  In the imaging operation illustrated in FIG. 13, the example in which the image data in which the image signal at the position of the focus detection pixel 311 is corrected by interpolation processing is stored in the memory card 213 is shown. However, the corrected image data is stored in the electronic viewfinder 215 or the body. You may make it display on the back monitor screen not shown provided in the back.

  Note that the imaging elements 211, 211A, and 211B described above can be formed as a CCD image sensor or a CMOS image sensor.

  In the embodiment described above, focus detection by the pupil division method using microlenses has been described as an example. However, the correlation calculation method of the present invention is not limited to the focus detection of the method described above, and re-imaging pupil division is performed. The present invention can be applied to the focus detection of the method, and the effects as described above can be obtained.

  With reference to FIG. 38, focus detection of the re-imaging pupil division method to which the correlation calculation method according to the embodiment of the present invention is applied will be described. In the figure, 191 is an optical axis of the interchangeable lens, 110 and 120 are condenser lenses, 111 and 121 are aperture masks, 112, 113, 122 and 123 are aperture openings, 114, 115, 124 and 125 are re-imaging lenses, 116 126 are image sensors (CCD) for focus detection. Reference numerals 132, 133, 142, and 143 denote focus detection light beams. Reference numeral 190 denotes an exit pupil set at a distance d5 in front of the planned imaging plane of the interchangeable lens. The distance d5 is a distance determined according to the focal lengths of the condenser lenses 110 and 120 and the distances between the condenser lenses 110 and 120 and the aperture openings 112, 113, 122, and 123, and is referred to as a distance measuring pupil distance. Reference numeral 192 denotes a region of the apertures 112 and 122 projected by the condenser lenses 110 and 120 (ranging pupil), and reference numeral 193 denotes a region of the apertures 113 and 123 projected by the condenser lenses 110 and 120 (ranging pupil).

  The condenser lens 110, aperture mask 111, aperture openings 112 and 113, re-imaging lenses 114 and 115, and image sensor 116 are re-imaging pupil division type phase difference detection focus detection units that perform focus detection at one position. Configure. FIG. 38 schematically illustrates a focus detection unit on the optical axis 191 and a focus detection unit outside the optical axis. By combining a plurality of focus detection units, it is possible to realize a focus detection dedicated sensor that performs focus detection by re-imaging pupil division type phase difference detection at the five focus detection positions G1 to G5 shown in FIG. .

  The focus detection unit including the condenser lens 110 is disposed in the vicinity of the condenser lens 110 disposed in the vicinity of the planned imaging plane of the interchangeable lens, the image sensor 116 disposed behind the condenser lens 110, and disposed between the condenser lens 110 and the image sensor 116. A pair of re-imaging lenses 114 and 115 for re-imaging the primary image formed in the vicinity of the imaging plane on the image sensor 116, and a pair disposed in the vicinity (front side in the figure) of the pair of re-imaging lenses. The aperture mask 111 having the aperture apertures 112 and 113. The image sensor 116 is a line sensor in which a plurality of photoelectric conversion units are densely arranged along a straight line, and the arrangement direction of the photoelectric conversion units matches the dividing direction of the pair of distance measuring pupils (= the direction in which the aperture openings are arranged). Let

  Information corresponding to the intensity distribution of the pair of images re-imaged on the image sensor 116 is output from the image sensor 116, and image shift detection calculation processing (correlation processing and phase difference detection processing) to be described later is performed on this information. By applying this, the image shift amount of the pair of images is detected by a so-called pupil division type phase difference detection method (re-imaging method). By multiplying the image shift amount by a predetermined conversion coefficient, the deviation (defocus amount) of the current image plane with respect to the planned image plane is calculated.

  The image sensor 116 is projected on the planned imaging plane by the re-imaging lenses 114 and 115, and the detection accuracy of the defocus amount (image deviation amount) is determined by the detection pitch of the image deviation amount (in the case of the re-imaging method). This is determined by the arrangement pitch of the photoelectric conversion units projected on the planned imaging plane.

  The condenser lens 110 projects the aperture openings 112 and 113 of the aperture mask 111 as areas 192 and 193 on the exit pupil 190. These areas 192 and 193 are called distance measuring pupils. That is, a pair of images re-imaged on the image sensor 116 is formed by a light beam passing through the pair of distance measuring pupils 192 and 193 on the exit pupil 190. The light beams 132 and 133 that pass through the pair of distance measuring pupils 192 and 193 on the exit pupil 190 are referred to as focus detection light beams.

  The invention of the present application is not limited to focus detection using a method in which the light beam passing through the photographing optical system is divided into pupils, but can also be applied to distance measurement by the external light triangulation method, and has the effects described above. Can do. With reference to FIG. 39, distance measurement of an external light triangulation system to which the correlation calculation method according to the embodiment of the present invention is applied will be described. A unit composed of the lens 320 and the image sensor 326 disposed on the imaging surface thereof, and a unit composed of the lens 330 and the image sensor 336 disposed on the imaging surface thereof are disposed with a baseline length therebetween. The pair of units constitutes the distance measuring device 347. An image of the distance measuring object 350 is formed on the image sensors 326 and 336 by the lenses 320 and 330.

  The positional relationship between the images formed on the image sensors 326 and 336 changes according to the distance from the distance measuring device 347 to the distance measuring object 350. Therefore, the relative positional relationship between the two images is detected by performing image shift detection applying the present invention to the signal data of the image sensors 326 and 336, and the distance to the distance measuring object 350 is determined based on this positional relationship. Can be measured. In the external light triangulation method, dirt and raindrops adhere to the lenses 320 and 330, which causes a level difference or distortion in the pair of signals. By applying the correlation calculation method of the present invention, Even if these problems occur, the correlation between the pair of image signal signal data strings output from the image sensors 326 and 336 can be accurately detected.

  In the above description, the correlation calculation uses the sum of the absolute values of the differences between the two data, but other correlation calculation methods may be used. For example, the correlation amount may be calculated by the sum of multiplications of two data, and the relative shift amount between the two signal data strings may be detected from the shift amount that gives the peak value of the correlation amount. Alternatively, the correlation amount may be calculated from the sum of the MAX values of the two data, and the relative shift amount between the two signal data strings may be detected from the shift amount that gives the bottom value of the correlation amount. Furthermore, the correlation amount may be calculated by the sum of the MIN values of the two data, and the relative shift amount between the two signal data strings may be detected from the shift amount that gives the peak value of the correlation amount.

  The imaging device according to the embodiment of the present invention is not limited to a digital still camera or a film still camera including an interchangeable lens and a camera body, but is also applied to a lens-integrated digital still camera, a film still camera, or a video camera. can do. The present invention can also be applied to a small camera module, a surveillance camera, or the like built in a mobile phone. Furthermore, the present invention can be applied to a focus detection device other than a camera, a distance measuring device, or a stereo distance measuring device.

  The present embodiment can also be applied to an apparatus that detects a movement of a subject image or a camera shake by detecting a correlation between image sensor signals at different times. Further, it can be applied to pattern matching between an image signal of an image sensor and a specific image signal. Further, the present invention is not limited to detecting the correlation of image signal data, but can be applied to any of the correlations of data related to sound and other generally detecting the correlation of two signals, and the effects described above can be achieved. .

  As described above, in one embodiment, for example, vignetting due to the photographing optical system occurs in one of a pair of focus detection light beams, and a pair of signal data strings output from the image sensor has a relative distortion. Even if it occurs, the correlation between both signal data strings can be detected accurately.

<< Second Embodiment of the Invention >>
According to the focus detection calculation method (correlation calculation method) of the first embodiment described above, there is a gain difference between a pair of calculation target signal sequences, that is, the first signal data sequence and the second signal data sequence. However, it is possible to accurately detect the correlation between both signal data strings. However, when there is a gain difference and an offset between the first signal data string and the second signal data string, the accuracy of the correlation calculation result decreases. Therefore, in the second embodiment, a focus detection calculation method (correlation calculation method) when there is a gain difference and an offset between the first signal data string and the second signal data string will be described. In the second embodiment, only portions different from the above-described first embodiment and modifications thereof will be described.

<Focus detection operation>
FIG. 40 is a flowchart showing details of the focus detection operation of the second embodiment. Note that steps that perform the same processing as the processing shown in FIG. 14 are denoted by the same step numbers, and differences will be mainly described. In step 300, focus detection calculation processing, that is, correlation calculation processing is started. In step 310, a pair of signal data sequences (α1 to αM, β1 to βM: M is the number of data) are output as follows. High-frequency cut filter processing as shown in equation (16) is performed to generate first signal data sequences a1 to aN and second signal data sequences b1 to bN (N is the number of data), and adversely affect correlation processing from the signal data sequences. Remove noise components and high frequency components.
an = αn + 2 × αn + 1 + αn + 2,
bn = βn + 2 × βn + 1 + βn + 2 (16)
In the formula (16), n = 1 to N. Note that the high-frequency cut filter processing in step 310 can be omitted if the calculation time is to be shortened or if it is already defocused and it is known that there are few high-frequency components.

In step 315, the average values ax and bx of the respective data sequences are subtracted from the first signal data sequence a1 to aN and the second signal data sequence b1 to bN to obtain the first 'signal data sequence A1 to AN and the second' signal data. Generate columns B1-BN. For convenience, the first signal sequence data and the second signal sequence data of the first embodiment and the first signal sequence data and the second signal sequence data of the second embodiment are represented by the same symbols A1 to AN and The first signal string data A1 to AN and the second signal string data B1 to BN of the second embodiment are calculated using the following equations (17) and (18). It is the signal string data after having been processed.
ax = (Σan) / N,
bx = (Σbn) / N (17)
An = an-ax,
Bn = bn−bx (18)
In the formulas (17) and (18), n = 1 to N.
As will be described later, even if an offset is generated between the first signal data string and the second signal data string by reducing the average value, the first signal data string and the second signal data string are subjected to correlation calculation processing described later. Can be detected with high accuracy.

  In step 320, in order to calculate the correlation amount between the first 'signal data sequence A1 to AN and the second' signal data sequence B1 to BN at the shift amount k (integer), first, the first 'signal data sequence A1 to AN. The second 'signal data strings B1 to BN are shifted relative to each other by a predetermined amount k.

  In the next step 330, data in an arbitrary range in the first 'signal data string A1 to AN, that is, the data An of interest and its neighboring data An-r to An + s (r and s are arbitrary integers; The first calculation (detailed later) is performed on this range (referred to as a “small range”) to obtain calculation data, and the first among the second ′ signal data strings B1 to BN shifted by the shift amount k. 'A second calculation (detailed later) is performed on the data Bn + k corresponding to the attention data An in the signal data string and the minute range data Bn-r to Bn + s in the vicinity thereof to obtain calculation data. Then, the first calculation data is calculated by multiplying the calculation data.

  In step 340, the first calculation is performed on the minute range data Bn-r to Bn + s in the second 'signal data strings B1 to BN to obtain calculation data, and the first' signal data string A1. The second calculation is performed on the minute range data An-r to An + s in .about.AN to obtain calculation data. Then, the second calculation data is calculated by multiplying the calculation data.

  Although details will be described later, the first calculation data and the second calculation data of the calculation results in Steps 330 and 340 are both distorted in the first 'signal data string and the second' signal data string in an arbitrary "small range". Is offset data.

  Further, in steps 330 and 340 described above, an example is shown in which calculation data is obtained by performing the first calculation and the second calculation on the “small range” of the first ′ signal data string and the second ′ signal data string. The data Bn + k corresponding to the position of the target data An in any position of the first 'signal data string A1 to AN and the position of the target data An of the second' signal data string B1 to BN shifted by the shift amount k. The first operation data is generated by multiplying neighboring data, for example, Bn + 1 + k or Bn + 2 + k, and the data Bn + k of the second 'signal data sequence B1 to BN and the first' The second calculation data may be generated by multiplying the data in the vicinity of the target data An of the signal data strings A1 to AN, for example, An + 1 or An + 2.

  In step 350, the absolute value of the difference between the first calculation data and the second calculation data is calculated. The calculation result is an amount indicating the degree of correlation (similarity) between the first 'signal data string and the second' signal data string in the "small range". In the following step 360, the processing of steps 330 to 350 is repeated while shifting the “small range”, and the calculation result of the amount indicating the degree of correlation in step 350 is cumulatively added, whereby the correlation amount C (k) at the predetermined shift amount k. Is calculated.

  Hereinafter, the processing in steps 360 to 400 is the same as the processing shown in FIG.

《Correlation operation》
Next, an example of the correlation calculation process shown in FIG. 40 will be described. FIG. 41 is a diagram for explaining the concept of correlation calculation according to the second embodiment. The interchangeable lens includes first signal data strings a1 to aN indicated by solid lines and second signal data strings b1 to bN indicated by broken lines. A case in which distortion due to vignetting 202 or the like occurs and a gain difference and an offset occur in the waveforms of both signal data strings is shown.

  As for the offset, the first signal data sequence A1 to AN and the second signal signal sequence B1 to BN are generated by subtracting the respective average values from the first signal data sequence a1 to aN and the second signal data sequence b1 to bN. To cancel. Further, the gain difference remaining in the first 'signal data sequence A1 to AN and the second' signal data sequence B1 to BN is canceled as follows. “Small range” characteristics in each signal data string, that is, the ratios An / An + 1 and Bn + k / Bn + 1 + k of adjacent data are not greatly affected even when there is a gain difference. . Since these “minor range” characteristics An / An + 1 and Bn + k / Bn + 1 + k include a denominator part, these “minor range” characteristics include common terms (An + 1 × By multiplying by (Bn + 1 + k), it is possible to suppress the influence of divergence when the denominator becomes 0 and the influence of noise when the value of the denominator is small.

  For example, the ratio An / An + 1, Bn + k / Bn + 1 + k of adjacent data sizes is multiplied by a common term (An + 1 × Bn + 1 + k) in the denominator portion, and An × Bn + 1 + k, Bn + k × An + 1 are obtained. By taking the correlation between the “small range” characteristics between the two signal data strings calculated in this way, in this case the absolute value of the difference, the degree of correlation (similarity) between the two signal data strings is increased. It becomes possible to detect with accuracy. Furthermore, by accumulating the degree of correlation while shifting the position of the “small range” within a predetermined range, it is possible to eliminate the accidentally generated correlation and perform highly accurate correlation detection.

  Arbitrary data An in the first 'signal data sequence A1 to AN and data in the vicinity of data Bn + k corresponding to the data An in the second' signal data sequence B1 to BN shifted by the shift amount k, for example, one adjacent Bn + 1 + k or two adjacent data Bn + 2 + k is multiplied to generate the first operation data (An × Bn + 1 + k) or (An × Bn + 2 + k), The data Bn + k of the second 'signal data sequence B1 to BN and the data in the vicinity of the data An of interest of the first' signal data sequence A1 to AN, for example, the next one An + 1 or the next two The second operation data (Bn + k × An + 1) or (Bn + k × An + 2) is generated by multiplying by An + 2. Based on the first calculation data and the second calculation data, the correlation between the third signal data string and the fourth signal data string is obtained by the above equation (2). This correlation calculation example is the processing example 1 and the processing example 2 described above, and a description thereof will be omitted.

As another example of the correlation calculation, the data of an arbitrary range in the first 'signal data sequence A1 to AN, that is, the data An of interest and the minute range data An-r to An + s in the vicinity thereof are The calculation data is obtained by performing one calculation, and the data Bn + k corresponding to the attention data An in the first signal data string B2 to BN shifted by the shift amount k and the vicinity thereof A second calculation is performed on the minute range data Bn-r to Bn + s to obtain calculation data. Then, the first calculation data is calculated by multiplying the calculation data.
Further, the first calculation is performed on the minute range data Bn−r to Bn + s in the second ′ signal data strings B1 to BN to obtain calculation data, and the first ′ signal data strings A1 to AN are obtained. The second calculation is performed on the minute range data An-r to An + s in FIG. Then, the second calculation data is calculated by multiplying the calculation data.
Based on the first calculation data and the second calculation data, the correlation between the first signal data string and the second signal data string is obtained by the above equation (2). This correlation calculation example is the processing example 3 to the processing example 8 described above, and a description thereof is omitted.
The correlation calculation result of the first 'signal data sequence An and the second' signal data sequence Bn shows an error due to the gain difference and offset regarding the first signal data sequence an and the second signal data sequence bn having a gain difference and offset. The removed highly accurate correlation calculation result is obtained.

  Here, consider a case where the correlation calculation processing example 1 is performed on a pair of image signal data strings a and b as shown in FIG. As a pair of image signal data strings (a1, a2,..., AN) and (b1, b2,..., BN) output from the image sensor 211, two image functions 2 having a gain difference of 2 and an offset of 2 are used. * Set (2 + sinω) +2, (2 + sinω). For such a pair of image signal data sequences a and b, first, signal data sequences A and B are generated by subtracting the respective average values and canceling the offset. Next, FIG. 43A shows the result of executing the correlation calculation processing example 9 on these two signal data strings A and B. As is clear from the figure, the correlation calculation processing example 1 enables high-precision correlation (clear drop on the graph) even though the original two signal data sequences a and b have a gain difference and an offset. It can be seen that

  FIG. 43B shows a result of executing the correlation calculation processing example 9 as it is without reducing the average value for the pair of image signal data sequences a and b. As is apparent from the figure, since the original two signal data strings a and b have a gain difference and an offset, the degree of the drop on the graph is lower than that when the average value shown in FIG. It can be seen that the correlation calculation accuracy decreases accordingly. FIG. 43 (c) shows a result when the conventional correlation calculation (C (k) = Σ | an−bn + k |) is applied to the pair of image signal data strings a and b. As is apparent from the figure, when the two signal data strings a and b have a gain difference and an offset, the drop in the correlation graph does not occur at the position of the correct deviation amount, and the correlation detection becomes impossible. I understand that.

In the correlation calculation processing according to the second embodiment, first, after obtaining and subtracting the respective average values for the original pair of data strings, the correlation calculation processing examples 1 to 8 are applied to the pair of data strings. As described above, the average value may be subtracted during the correlation calculation. For example, the correlation calculation processing example 1 can be changed as follows.
C (k) = Σ | (an−ax) × (bn + 1 + k−bx) − (bn + k−bx) × (An + 1−ax) | (19)
In the equation (19), when the Σ operation range is narrower than the original data string range, the Σ operation range is used instead of the average values ax and bx of the entire original data string. The average values ax ′ and bx ′ of the data string may be used. In this way, an accurate correlation calculation result can be calculated even when the gain difference varies depending on the position in the data string.

The figure which shows the structure of the digital still camera of one embodiment The figure which shows the focus detection area on the imaging screen which is set to the planned image formation surface of the interchangeable lens Front view showing detailed configuration of image sensor Configuration diagram of imaging pixels Configuration diagram of focus detection pixels The figure which shows the spectral sensitivity characteristic of each color filter of an image pick-up pixel The figure which shows the spectral sensitivity characteristic of a focus detection pixel Cross section of imaging pixel Cross section of focus detection pixel The figure explaining the focus detection method by a pupil division system Front view showing projection relationship on exit pupil plane A graph showing the intensity distribution (light quantity) of an image signal at one focus detection position on the vertical axis and the position deviation within the focus detection position on the horizontal axis. 1 is a flowchart illustrating an imaging operation of a digital still camera (imaging device) according to an embodiment. Flow chart showing details of focus detection operation Diagram explaining correlation calculation The figure for demonstrating the thought of the correlation calculation of one Embodiment The figure which shows the data flow of the correlation calculation process example 1 The figure which shows the case where relative distortion generate | occur | produces in the two signal data strings A and B The figure which shows the result of having performed correlation calculation processing example 1 The figure which shows the data flow of the correlation calculation processing example 2 The figure which shows the result of having performed correlation calculation processing 2 The figure which shows the data flow of correlation calculation processing example 3 The figure which shows the result of having performed correlation calculation processing 3 The figure which shows the data flow of correlation calculation processing 4 The figure which shows the result of having performed correlation calculation processing 4 The figure which shows the data flow of the correlation calculation process 5 The figure which shows the result of having performed correlation calculation processing 5 The figure which shows the data flow of the correlation calculation process 6 The figure which shows the result of having performed correlation calculation processing 6 The figure which shows the data flow of the correlation calculation processing example 7 The figure which shows the result of having performed correlation calculation processing example 7 The figure which shows the data flow of the correlation calculation process example 8. The figure which shows the result of having performed correlation calculation processing example 8 The figure which shows the example of the image pick-up element which arranged the focus detection pixel at the position of the blue pixel of the image pick-up pixel every other pixel in a line. The figure which shows the focus detection pixel provided with one photoelectric conversion part in one pixel FIG. 35 is a diagram illustrating an example of an image sensor in which the focus detection pixels illustrated in FIGS. 35A and 35B are alternately arranged in a line. It is a figure which shows a Bayer arrangement | sequence and a complementary color arrangement | sequence. The figure explaining the focus detection of the re-imaging pupil division system to which the correlation calculation method by embodiment of this invention is applied The figure explaining the distance measurement of the external light triangulation system to which the correlation calculation method by embodiment of this invention is applied The flowchart which shows the detail of the focus detection operation | movement of 2nd Embodiment. The figure for demonstrating the thought of the correlation calculation of 2nd Embodiment The figure for demonstrating the example of the correlation calculation process of 2nd Embodiment The figure for demonstrating the correlation calculation processing result of 2nd Embodiment

Explanation of symbols

202 Interchangeable lens 209 Lens drive control device 211 Image sensor 212 Camera drive control device

Claims (21)

A correlation calculation method for calculating a degree of correlation between a first signal data string and a second signal data string,
First information for generating first calculation data by multiplying the first data in the first signal data string by data in the vicinity of the second data corresponding to the first data in the second signal data string Generation process,
A second information generation process for generating second operation data by multiplying the second data in the second signal data sequence by data in the vicinity of the first data in the first signal data sequence;
A correlation calculation method, comprising: executing a correlation detection process for calculating a correlation between the first calculation data and the second calculation data. A correlation calculation method for calculating a degree of correlation between a first signal data string and a second signal data string,
A first calculation is performed on the first data in the first signal data string and data in the vicinity thereof to calculate calculation data, and second data corresponding to the first data in the second signal data string and its A first information generation process for performing a second calculation on neighboring data to calculate calculation data, multiplying the calculation data, and calculating the first calculation data;
The first calculation is performed on the second data in the second signal data string and the data in the vicinity thereof to calculate the calculation data, and the first data in the first signal data string and the data in the vicinity thereof are calculated. A second information generation process for calculating the operation data by performing the second operation, and multiplying the operation data to calculate the second operation data;
A correlation calculation method, comprising: executing a correlation detection process for calculating a correlation between the first calculation data and the second calculation data. The correlation calculation method according to claim 2,
The correlation calculation method, wherein the first calculation is addition / subtraction with respect to the first data and its neighboring data, or addition / subtraction with respect to the second data and its neighboring data. The correlation calculation method according to claim 2,
The correlation calculation method, wherein the first calculation is an operation of adding the first data and its neighboring data, or an operation of adding the second data and its neighboring data. The correlation calculation method according to claim 2,
The correlation calculation method, wherein the first calculation is an average calculation of the first data and data in the vicinity thereof, or an average calculation of the second data and data in the vicinity thereof. In the correlation calculation method according to any one of claims 2 to 5,
The correlation calculation method characterized in that the second calculation is a differential calculation on the first data and its neighboring data, or a differential calculation on the second data and its neighboring data. In the correlation calculation method according to any one of claims 1 to 6,
The correlation calculation method, wherein the correlation degree detection processing calculates an absolute value of a difference between the first calculation data and the second calculation data. The correlation calculation method according to claim 7,
The first information generation process and the second information generation process are executed while changing the first data in the first signal data sequence, and the first calculation data and the second calculation are performed in the correlation degree detection process. A correlation calculation method comprising calculating a sum of absolute values of differences from data. The correlation calculation method according to claim 8,
The first information generation process and the second information generation process are executed by relatively shifting the first signal data sequence and the second signal data sequence, and the correlation degree detection process performs the first information generation process and the second information generation process for each shift amount. A correlation calculation method characterized by calculating a degree of correlation between one calculation data and the second calculation data. The correlation calculation method according to claim 9,
In the correlation degree detection process, the first signal data string and the second signal are based on a shift amount between the first signal data string and the second signal data string when the maximum degree of correlation is obtained. A correlation calculation method characterized by detecting a deviation amount from a data string. In the correlation calculation method according to any one of claims 1 to 10,
The correlation calculation method, wherein the first signal data string and the second signal data string are signal data strings corresponding to an output signal of an image sensor. The correlation calculation method according to any one of claims 1 to 11,
A correlation calculation method characterized by performing a process of removing high-frequency components of the first signal data sequence and the second signal data sequence before executing the first information generation process and the second information generation process. . In the correlation calculation method according to any one of claims 1 to 12,
A process of subtracting an average value of the first data from the first data in the first signal data string and subtracting an average value of the second data from the second data in the second signal data string; One information generation process and the second information generation process generate the first calculation data and the second calculation data by using the processed first signal data string and the second signal data string. Correlation calculation method. A correlation calculation device for calculating a degree of correlation between a first signal data string and a second signal data string,
First information for generating first calculation data by multiplying the first data in the first signal data string by data in the vicinity of the second data corresponding to the first data in the second signal data string Generating means;
Second information generating means for generating second operation data by multiplying the second data in the second signal data string by data in the vicinity of the first data in the first signal data string;
A correlation calculation device comprising: a correlation level detection means for calculating a correlation level between the first calculation data and the second calculation data. A correlation calculation device for calculating a degree of correlation between a first signal data string and a second signal data string,
A first calculation is performed on the first data in the first signal data string and data in the vicinity thereof to calculate calculation data, and second data corresponding to the first data in the second signal data string and its First information generating means for performing a second operation on neighboring data to calculate operation data, multiplying the operation data, and calculating the first operation data;
The first calculation is performed on the second data in the second signal data string and the data in the vicinity thereof to calculate the calculation data, and the first data in the first signal data string and the data in the vicinity thereof are calculated. Second information generating means for calculating the operation data by performing the second operation and multiplying the operation data to calculate the second operation data;
A correlation calculation device comprising: a correlation level detection means for calculating a correlation level between the first calculation data and the second calculation data. The correlation calculation device according to claim 14 or 15,
Subtracting the average value of the first data from the first data in the first signal data string and subtracting the average value of the second data from the second data in the second signal data string, One information generation means and the second information generation means generate the first calculation data and the second calculation data by using the processed first signal data string and second signal data string. Correlation calculation device. Photoelectric conversion means for receiving a light beam that has passed through a pair of pupil regions of the photographing optical system via a focus detection optical system and outputting a pair of subject image signals;
The image shift amount of the pair of subject image signals is detected using the correlation calculation method according to any one of claims 1 to 13 with respect to the pair of subject image signals output from the photoelectric conversion means. An image shift detection means,
A focus detection apparatus that detects a focus adjustment state of the photographing optical system based on the image shift amount. The focus detection apparatus according to claim 17, wherein
The focus detection optical system, wherein the focus detection optical system is a microlens installed for each pixel of the photoelectric conversion means. The focus detection apparatus according to claim 17, wherein
The focus detection optical system, wherein the focus detection optical system is a re-imaging optical system that re-images a subject image formed on a predetermined focal plane of the photographing optical system on the photoelectric conversion means. A pair of photoelectric conversion means for receiving a subject image via a pair of optical systems;
An image shift amount of the pair of subject image signals is detected using the correlation calculation method according to any one of claims 1 to 13 with respect to the pair of subject image signals output from the pair of photoelectric conversion means. Image deviation detecting means for
A distance measuring apparatus that measures a distance to the subject based on the image shift amount.   An imaging device comprising the focus detection device or the distance measuring device according to any one of claims 17 to 20.

Images