JP6173549B2 - Image data processing device, distance calculation device, imaging device, and image data processing method - Google Patents

Description

The present invention relates to an image data processing device, a distance calculation device, an imaging device, and an image data processing method.

  In digital still cameras and digital video cameras, pixels having a ranging function (hereinafter also referred to as “ranging pixels”) are arranged on some or all of the pixels of the image sensor, and the distance to the subject is determined by a phase difference method. Patent Document 1 proposes a solid-state imaging device that detects the above. The ranging pixel includes a plurality of photoelectric conversion units, and is configured such that light beams that have passed through different regions on the pupil of the photographing lens are guided to different photoelectric conversion units.

  Optical images generated by light beams that have passed through different pupil regions (hereinafter also referred to as “A image” and “B image”, respectively) by electrical signals generated by the photoelectric conversion unit included in each ranging pixel. Collectively, image data based on “AB image” (hereinafter, an image based on the A image is referred to as an A image and an image based on the B image is referred to as a B image) is acquired. An image shift amount (also referred to as parallax), which is a relative positional shift amount between the A image and the B image, is calculated.

  For the calculation of the image shift amount, a region-based corresponding point search technique called template matching is often used. In template matching, an A image or a B image is set as a reference image, and the other image different from the reference image is set as a reference image. A reference area (also referred to as a reference window) centered on the target point is set on the reference image, and a reference area (also referred to as a reference window) is set for a reference point corresponding to the target point on the reference image. To do. While sequentially moving the reference point, a reference point having the highest correlation between the image in the reference area and the image in the reference area is searched, and the relative positional deviation between the target point and the reference point is used to determine the image deviation amount. Perform the calculation. In general, when the size of the search area is reduced, an image shift amount calculation error caused by local calculation occurs, and therefore a relatively large area size (for example, 9 pixels × 9 pixels) is used.

  By converting this image shift amount into a defocus amount through a conversion coefficient, the distance to the subject can be calculated. According to this, unlike the conventional contrast method, it is not necessary to move the lens to measure the distance, so that high-speed and high-precision distance measurement is possible.

  Here, the distance accuracy at the time of measurement is improved by accurately obtaining the image shift amount. A factor that causes an error in the amount of image shift is a loss of light amount balance. The loss of the light amount balance is a phenomenon in which the light amount ratio between the A image and the B image changes according to the image height due to the vignetting of the light beam by the lens frame of the photographing lens, the angle characteristic of the distance measuring pixel sensitivity, or the like.

  In Patent Document 2, a uniform correction is applied after assembling the photographic lens and the image sensor to calculate a correction coefficient that makes the light quantity ratio between the A image and the B image constant, and this correction coefficient is used when calculating the image misalignment amount. A method for correcting image data has been proposed.

  Further, Patent Document 3 proposes a method for correcting the collapse of the light amount balance by performing a cross-correlation calculation in consideration of the light amount ratio between the A image and the B image.

Japanese Patent No. 4027113 Japanese Patent Laid-Open No. 2-181108 JP 2009-258231 A

  In the focus detection method disclosed in Patent Document 2, the correction coefficient is measured after the imaging optical system and the image sensor are assembled, so that the light quantity ratio between the A image and the B image is corrected without being affected by manufacturing errors. Thus, the calculation error of the image shift amount can be reduced.

  However, recent digital cameras are equipped with an optical camera shake correction function, and there is a possibility that the optical axis of the photographing lens differs between when the correction coefficient is calculated and when actual photographing is performed. That is, when the correction coefficient acquired in advance is used, it is not always possible to satisfactorily correct the collapse of the light amount balance between the A image and the B image, and the calculation accuracy of the image shift amount may be reduced. In addition, calculating the correction coefficient after assembling the photographic lens and the image sensor increases the adjustment process, which may increase the manufacturing cost.

  In the focus detection method disclosed in Patent Document 3, when calculating the correlation between the image in the search area and the image in the reference area, the light quantity balance is lost by considering the light quantity ratio between the A image and the B image. Is corrected and the calculation error of the image shift amount is reduced to improve the distance measurement accuracy.

  In template matching, there is an overlapping area in the range of the set reference area between adjacent points of interest. In the focus detection method disclosed in Patent Document 3, since the collapse of the light amount balance between the A image and the B image is corrected every time the degree of correlation is calculated, the light amount ratio is corrected a plurality of times in the overlapping region. As a result, the amount of calculation increases compared to the conventional correlation degree calculation. The increase in the amount of calculation may increase the processing time and increase the cost due to the large scale of the arithmetic circuit. In addition, since the balance of light quantity is corrected considering only the A and B images included in the reference area, there is a risk that the light quantity balance cannot be accurately corrected due to the influence of noise such as light shot noise. There is.

The present invention is, with a small amount of calculation, an object of Rukoto to correct complement the collapse of the light intensity balance of the A and B images.

The image data processing device according to the present invention is different from the first image data based on the first light flux passing through the first pupil region of the imaging optical system and the first pupil region of the imaging optical system. And second image data based on the second light flux passing through the second pupil region decentered in the first direction, and the first data along the first direction . An image data ratio calculating means for calculating an image data ratio that is a ratio of at least a part of the image data and the second image data, and a correction for calculating a correction function in the first direction based on the image data ratio A function calculating means; and a correcting means for correcting at least one of the first image data and the second image data based on the correction function .

The image data processing method according to the present invention is different from the first image data based on the first light flux passing through the first pupil region of the imaging optical system and the first pupil region of the imaging optical system. And an image data acquisition step for acquiring second image data based on the second light flux passing through the second pupil region decentered in the first direction, and the first along the first direction . An image data ratio calculating step for calculating an image data ratio that is a ratio of at least a part of the image data and the second image data, and a correction for calculating a correction function in the first direction based on the image data ratio A function calculation step, and a correction step of correcting at least one of the first image data and the second image data based on the correction function .

According to the present invention, in no small amount of calculation, it is possible to calculate the image shift amount with high accuracy, high precision becomes possible distance calculation.

1 is a configuration diagram of a digital camera according to a first embodiment. FIG. 5 is a diagram for explaining a light beam received by the photoelectric conversion unit according to the first embodiment. Flow chart of distance calculation procedure in embodiment 1 FIG. 6 is a diagram for explaining a method of calculating an image shift amount in the first embodiment. FIG. 5 is a diagram for explaining a conversion coefficient calculation method according to the first embodiment. The figure explaining the light-receiving amount of a photoelectric conversion part Flowchart of light amount balance correction processing in the first embodiment The figure explaining the light quantity balance correction process in Embodiment 1. Flowchart of light amount balance correction processing according to a modification of the first embodiment Flowchart of light amount balance correction processing according to a modification of the first embodiment Flowchart of light amount balance correction processing according to a modification of the first embodiment Configuration diagram of an image sensor according to a modification of the first embodiment Configuration diagram of a digital camera according to Embodiment 2 Flowchart of light amount balance correction processing in Embodiment 2 Flow chart showing an example of operation of a digital camera

  In the following description, a digital camera will be used as an example of an imaging apparatus including the distance calculation device of the present invention, but the application of the present invention is not limited to this. For example, the distance calculation apparatus according to the present invention can be applied to a digital distance measuring instrument.

  In the description with reference to the figures, even if the figure numbers are different, in principle, the same reference numerals are given to the parts indicating the same parts, and overlapping explanations are avoided as much as possible.

[First embodiment]
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.

<Configuration of digital camera>
FIG. 1A is a diagram illustrating a configuration of a digital camera 100 according to the present embodiment. The digital camera 100 includes an imaging optical system 120, an image sensor 101, a distance calculation unit 102, an image generation unit (not shown), and a lens drive control unit (not shown) arranged inside a camera housing 130. The The distance calculation device 110 includes an imaging optical system 120, an image sensor 101, and a distance calculation unit 102. The distance calculation unit 102 can be configured using a logic circuit. As another form of the distance calculation unit 102, a central processing unit (CPU) and a memory for storing an arithmetic processing program may be used.

The imaging optical system 120 is a photographing lens of the digital camera 100, and has a function of forming an image of a subject on the image sensor 101 that is an imaging surface. The imaging optical system 120 includes a plurality of lens groups (not shown), a diaphragm (not shown), and an optical camera shake correction mechanism (not shown), and has an exit pupil 103 at a position away from the image sensor 101 by a predetermined distance. The optical camera shake correction mechanism is composed of a correction lens having a gyro mechanism, and corrects the shake of the optical axis by operating the correction lens in a direction to cancel the camera shake. Note that reference numeral 140 in FIG. 1A denotes the optical axis of the imaging optical system 120. In the present embodiment, the z axis is parallel to the optical axis 140. Further, the x axis and the y axis are perpendicular to each other and perpendicular to the optical axis 140.

  Here, an operation example of the digital camera 100 will be described. However, the following is only an example and does not limit the present invention. FIG. 15 is a diagram for explaining the operation flow after the main power of the digital camera 100 is turned on and a shutter button (not shown) is half-pressed. First, in step 1501, information (focal length, aperture value, etc.) of the imaging optical system 120 is read and stored in a memory unit (not shown). Next, the processing of steps 1502, 1503 and 1504 is performed to adjust the focus. That is, in step 1502, based on the image data output from the image sensor 101, a defocus amount is calculated using a subject distance calculation procedure described later with reference to FIG. In step 1503, based on the calculated defocus amount, it is determined whether or not the imaging optical system 120 is in focus. If not in focus, in step 1504, the lens drive control unit drives the imaging optical system 120 to the in-focus position based on the defocus amount, and then returns to step 1502. If it is determined in step 1503 that the subject is in focus, it is determined in step 1505 whether or not the shutter has been released (so-called full press) by operating a shutter button (not shown). If it is determined that the release has not been performed, the process returns to step 1502 to repeat the above-described processing. If it is determined in step 1505 that the shutter has been released, the image data is read from the image sensor 101 and stored in a memory unit (not shown). An ornamental image can be generated by developing the image data stored in the memory unit. Also, a subject distance image (subject distance distribution) corresponding to the ornamental image can be generated by applying the subject distance calculation procedure described later with reference to FIG. 3 to the image data stored in the memory unit.

<Configuration of image sensor>
The image sensor 101 is composed of a CMOS (complementary metal oxide semiconductor) or a CCD (charge coupled device). The subject image formed on the image sensor 101 via the imaging optical system 120 is photoelectrically converted by the image sensor 101 to generate image data based on the subject image. Hereinafter, the image sensor 101 of the present embodiment will be described in more detail with reference to FIG.

  FIG. 1B is an xy sectional view of the image sensor 101. The image sensor 101 is configured by arranging a plurality of 2 × 2 pixel groups 150. The pixel group 150 is configured by arranging green pixels 150G1 and 150G2 in a diagonal direction, and red pixels 150R and blue pixels 150B in the other two pixels.

<Description of the principle of distance measurement>
Each pixel constituting the pixel group 150 of the present embodiment includes two photoelectric conversion units (161: first photoelectric conversion) having a cross-sectional shape symmetrical with respect to the xy section in the light receiving layer (203 in FIG. 2) in the pixel. Part 162; a second photoelectric conversion part) are juxtaposed. A light beam received by the first photoelectric conversion unit and the second photoelectric conversion unit in the image sensor 101 will be described with reference to FIG.

  FIG. 2 is a schematic diagram showing only the exit pupil 103 of the imaging optical system 120 and the green pixel 150G1 as a representative example of the pixels arranged in the image sensor 101. A pixel 150G1 illustrated in FIG. 2 includes a color filter 201, a microlens 202, and a light receiving layer 203. The light receiving layer 203 includes a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162. The microlens 202 is disposed so that the exit pupil 103 and the light receiving layer 203 are in a conjugate relationship. As a result, as shown in FIG. 2, the light beam 210 that has passed through the first pupil region 261 in the exit pupil 103 is incident on the first photoelectric conversion unit 161, and the light beam 220 that has passed through the second pupil region 262 is The light enters the second photoelectric conversion unit 162.

The plurality of first photoelectric conversion units 161 provided in each pixel generates first image data by photoelectrically converting the received light beam. Similarly, the plurality of second photoelectric conversion units 162 provided in each pixel photoelectrically convert the received light beam to generate second image data. An intensity distribution of an image (A image) formed on the image sensor 101 by a light beam that has mainly passed through the first pupil region can be obtained from the first image data, and the second pupil region can be obtained from the second image data. The intensity distribution of an image (B image) formed on the image sensor 101 by the light beam that has mainly passed through can be obtained. Therefore, the relative positional shift amount between the first image data and the second image data is the image shift amount between the A image and the B image. The distance to the subject can be calculated by calculating the image shift amount by a method described later and converting the image shift amount into a defocus amount via a conversion coefficient.

<Explanation of distance calculation procedure>
Hereinafter, the distance calculation procedure performed by the distance calculation unit 102 of the present embodiment will be described in detail with reference to FIG.

  In step S1, the image sensor 101 acquires the first image data and the second image data, and transmits them to the distance calculation unit 102 (image acquisition process).

  In step S2, a light amount balance correction process for correcting a break in the light amount balance is performed based on the first image data and the second image data. The light quantity balance correction process in step S2 will be described later with reference to FIG.

  In step S3, an image shift amount is calculated based on the first image data and the second image data. A method for calculating the image shift amount will be described with reference to FIG. FIG. 4 is a diagram for explaining a method of calculating the image shift amount, and shows first image data 401, second image data 402, and a photographic subject 400. An attention point 410 is set for the first image data, and a reference area 420 is set around the attention point 410. On the other hand, for the second image data, a reference point 411 is set at a position corresponding to the target point 410, and a reference area 421 is set around the reference point 411. While sequentially moving the reference point 411 within a predetermined image shift amount search range, the correlation value between the first image data in the reference area 420 and the second image data in the reference area 421 is calculated, and the highest correlation is obtained. The reference point 411 is set as the corresponding point of the attention point 410. The image shift amount search range may be determined from the maximum distance and the minimum distance for which distance calculation is desired. For example, the maximum distance is set to infinity, the minimum distance is set to the minimum shooting distance of the imaging optical system 120, and the range of the maximum image shift amount and the minimum image shift amount obtained from the maximum distance and the minimum distance is searched for the image shift amount. A range may be used. A relative position shift amount between the point of interest 410 and the corresponding point is an image shift amount. By searching for the corresponding point while sequentially moving the attention point 410, the image shift amount at each data position (each pixel position) in the first image data can be calculated. As a method for calculating the correlation value, a known method can be used. For example, a method called SSD that uses the sum of squares of the difference between each pixel data in the reference area and each pixel data in the reference area as an evaluation value is used. Can do.

なお、数式１以外の方法を用いて像ズレ量を像側デフォーカス量に変換してもよい。例えば、数式１にて基線長ｗが像ズレ量ｒに比べて十分大きいとの仮定に基づき、数式２によりゲイン値Ｇａｉｎを算出し、数式３に基づき像ズレ量を像側デフォーカス量に変換してもよい。

数式３を用いることで像ズレ量から像側デフォーカス量への変換を容易に行うことができ、被写体距離の算出に係る演算量を削減することができる。また、像ズレ量から像側デフォーカス量への変換に、変換用のルックアップテーブルを用いてもよい。この場合にも、被写体距離の算出に係る演算量を削減することができる。 In step S4, the image shift amount is converted into the image-side defocus amount via the conversion coefficient (defocus amount calculation process). A method for calculating the conversion coefficient will be described with reference to FIG. FIG. 5A shows the incident angle dependence of the light receiving sensitivity in the pixel. The horizontal axis indicates the incident angle of light incident on the pixel (the angle formed by the light beam projected on the xz plane and the z axis), and the vertical axis indicates the light receiving sensitivity. A solid line 501 indicates the light reception sensitivity of the first photoelectric conversion unit 161, and a broken line 502 indicates the light reception sensitivity of the second photoelectric conversion unit 162. FIG. 5B shows this light receiving sensitivity projected onto the exit pupil 103 and expressed as a light receiving sensitivity distribution on the exit pupil 103. The darker the color, the higher the light receiving sensitivity. FIG. 5B shows a barycentric position 511 of the light receiving sensitivity distribution of the first photoelectric conversion unit and a barycentric position 512 of the light receiving sensitivity distribution of the second photoelectric converting unit. A distance 513 between the centroid position 511 and the centroid position 512 is called a base line length, and is used as a conversion coefficient for converting an image shift amount into an image-side defocus amount. When the image shift amount is r, the base line length is w, and the pupil distance from the image sensor 101 to the exit pupil 103 is L, the image-side defocus amount ΔL is given by Equation 1.
Can be used to convert the image shift amount into the image-side defocus amount.

Note that the image shift amount may be converted into the image-side defocus amount using a method other than Equation 1. For example, based on the assumption that the baseline length w is sufficiently larger than the image shift amount r in Formula 1, the gain value Gain is calculated using Formula 2, and the image shift amount is converted into the image-side defocus amount based on Formula 3. May be.

By using Expression 3, conversion from the image shift amount to the image-side defocus amount can be easily performed, and the calculation amount related to the calculation of the subject distance can be reduced. Further, a conversion lookup table may be used for the conversion from the image shift amount to the image-side defocus amount. Also in this case, the amount of calculation related to the calculation of the subject distance can be reduced.

  In FIG. 2, the first pupil region 261 is a region where x is negative, and the second pupil region 262 is a region where x is positive and completely separated. However, since the light reaching the light receiving layer 203 actually has a certain spread due to the light diffraction phenomenon, the first pupil region 261 and the second pupil as shown in FIG. 5B show the light receiving sensitivity distribution. The region 262 will have overlapping regions. In the present embodiment, the first pupil region 261 and the second pupil region 262 will be described in a clearly separated manner for convenience.

  In step S5, the image-side defocus amount calculated in step S4 is converted into a subject distance based on the imaging relationship of the imaging optical system (subject distance calculation process). The conversion to the subject distance may be performed by other methods. For example, the image-side defocus amount is converted into the object-side defocus amount, and the object-side defocus amount calculated based on the focal length of the imaging optical system 120 and the object-side defocus amount The distance may be calculated. The defocus amount on the object side can be calculated using the defocus amount on the image side and the vertical magnification of the imaging optical system 120.

  In the distance calculation procedure of this embodiment, after the image shift amount is converted into the image-side defocus amount in step S4, the image-side defocus amount is converted into the subject distance in step S5. However, as described above, the image-side defocus amount and the object-side defocus amount or the image-side defocus amount and the subject distance can be converted by using the imaging relationship of the imaging optical system 120. Therefore, the image shift amount may be directly converted into the object-side defocus amount or the subject distance without using step S4 and / or step S5 of the present embodiment. In any case, the defocus amount (image side and / or object side) and subject distance can be accurately calculated by accurately calculating the image shift amount.

In this embodiment, the image-side defocus amount is converted into the subject distance in step S5. However, step S5 is not necessarily performed, and the distance calculation procedure may be terminated in step S4. . The amount of blurring of the subject in the ornamental image depends on the defocus amount on the image side, and a subject with a larger defocus amount on the image side captures a more blurred image. In the refocus processing for adjusting the focus position by image processing for such an image, it is not necessary to perform conversion to the subject distance, and it is sufficient if there is a defocus amount on the image side.

<Factors causing the loss of light balance>
Next, factors that cause a loss of light amount balance will be described.

  The amount of light 1 received by the first photoelectric conversion unit 161 and the amount of light 2 received by the second photoelectric conversion unit 162 vary depending on the image height due to vignetting of the imaging optical system 120, and In general, the amount of received light decreases as the area around the image sensor 101 decreases. FIG. 6A is a diagram for explaining vignetting of the light beam received by the first photoelectric conversion unit 161 and the light beam received by the second photoelectric conversion unit 162. The first photoelectric conversion unit 161 receives the light beam that has passed through the first pupil region 261 on the exit pupil 103 of the imaging optical system 120, and the second photoelectric conversion unit 162 outputs the emission of the imaging optical system 120. The light beam that has passed through the second pupil region 262 on the pupil 103 is received.

  Vignetting occurs symmetrically with respect to the optical axis 140 with respect to the light beam 621 and the light beam 622 received by the first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 located at the center image height 601. Therefore, the ratio between the light quantity 1 and the light quantity 2 is 1 (the balance of the light quantity is not lost).

  In contrast, the light beam 631 and the light beam 632 received by the first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 located at the peripheral image height 602 are asymmetric with respect to the optical axis 140 by the effective aperture 610. Vignetting occurs. Therefore, the ratio between the light quantity 1 and the light quantity 2 does not become 1 (the light quantity balance is lost).

  In addition, the loss of light amount balance also occurs when the first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 have sensitivity characteristics with high incident angle dependency as shown in FIG. FIG. 6B shows the light receiving sensitivity incident angle characteristics of the pixels as in FIG. 5A, and is a diagram for explaining the angular range of the light beam received by the central image height 601 and the peripheral image height 602. is there.

  At the central image height 601, the principal ray of the light beam composed of the light beam 621 and the light beam 622 is parallel to the optical axis, and thus the first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 are in the angular range 603. Receives light flux. In the angle range 603, the integrated value of the solid line 501 (the integrated value in the angle range 603) indicating the light receiving sensitivity of the first photoelectric converter 161 and the integrated value of the broken line 502 indicating the light receiving sensitivity of the second photoelectric converter 162. The ratio of (integral value within the angle range 603) is 1. That is, the light quantity balance does not collapse.

  On the other hand, at the peripheral image height 602, the principal ray of the light beam composed of the light beam 631 and the light beam 632 is not parallel to the optical axis, and the first photoelectric conversion unit 161 and the second photoelectric conversion unit 162. Receives the luminous flux in the angular range 604. In the angle range 604, the integrated value of the solid line 501 (the integrated value in the angle range 604) indicating the light receiving sensitivity of the first photoelectric conversion unit 161 and the integrated value of the broken line 502 indicating the light receiving sensitivity of the second photoelectric conversion unit 162. The ratio of (integral value within the angle range 604) is not 1. That is, the light quantity balance is lost.

  In the digital camera 100 of the present embodiment, the light amount balance in which the ratio of the light amount 1 and the light amount 2 does not become 1 at the peripheral image height due to the influence of the vignetting of the imaging optical system 120 and the incident angle dependency of the photoelectric conversion unit. Collapse occurs.

In the image shift amount calculation processing in step S3, the correlation (that is, the degree of similarity) between the standard area 420 and the reference area 421 is evaluated. Therefore, when the light amount balance is lost, an incorrect reference point 411 is used. It is determined that the correlation is high, and an error occurs in the calculated image shift amount.

<Detailed description of light quantity balance correction processing>
Hereinafter, a process of correcting the collapse of the light quantity balance between the first image data and the second image data (light quantity balance correction process) will be described in detail with reference to FIGS. FIG. 7 is a flowchart of the light quantity balance correction process in the present embodiment, and FIG. 8 is a diagram for explaining the light quantity balance correction process in the present embodiment.

  In step S2-1, the first partial image data is extracted from the first image data, and the second partial image data is extracted from the second image data. As shown in FIG. 8A, the first image data included in the first data range 821 is extracted from the first image data 801 as first partial image data 811. Similarly, the second image data included in the second data range 822 is extracted from the second image data 802 and used as second partial image data 812. The second data range is a position corresponding to the first data range in the first image data (the position of the first data range in the first image data and the second data range in the second image data). The position of the data range is set to the relatively same position). FIG. 8B shows the first image data when the light quantity balance is not broken as first reference image data (solid line 803), and the second image data when the light quantity balance is not broken. This is shown as second reference image data (broken line 804). Note that the horizontal axis indicates the image height (pixel position) in the x direction, and only a partial range 830 in FIG. In FIG. 8B, since there is no collapse of the light quantity balance, the first reference image data and the second reference image data are image data whose positions are relatively shifted in the x direction. On the other hand, FIG. 8C shows the first partial image data and the second partial image data included in the range 830 in FIG. From the comparison between FIG. 8B and FIG. 8C, it can be seen that the data value of the first partial image data is lowered due to the loss of the light amount balance. If the image shift amount calculation process in step S3 is performed without correcting the light intensity balance, it is determined that the correlation is high at the wrong reference point, and an error occurs in the calculated image shift amount.

  In the present embodiment, the first data range 821 is limited to a partial range of the first image data 801. However, the first data range 821 may be set to the entire area of the first image data 801. Absent. Desirably, by setting the first data range 821 to the entire area of the first image data 801, the number of data used for calculation of the correction function in step S2-3 to be described later can be increased, and the approximation error is reduced. be able to.

なお、ｘｐは部分画像データ中のｘ方向と対応する方向の座標値を示し、ｙｐは部分画像データ中のｙ方向と対応する方向の座標値を示している。 In step S2-2, a process of calculating an image data ratio (image data ratio calculation process), which is a ratio between the first partial image data and the second partial image data, is performed. When the first partial image data is Pimg1 (xp, yp), the second partial image data is Pimg2 (xp, yp), and the image data ratio is Rimg (xp, yp), the image data ratio Rimg is expressed by Equation 4. It can be calculated according to

Note that xp indicates a coordinate value in a direction corresponding to the x direction in the partial image data, and yp indicates a coordinate value in a direction corresponding to the y direction in the partial image data.

In step S2-3, a correction function Fn calculation process approximating the image data ratio Rimg is performed. In the present embodiment, the correction function Fn is calculated by the least square method. That is, N-th order polynomial (Nth-order function) coefficients having an image height (x and / or y) as a variable are calculated by least square approximation, and these coefficients have an image height (pixel position) as a variable. A correction function Fn expressed by the following function is adopted. The calculation of the correction function Fn can use a spline method in addition to the least square method. N may be any value as long as it is a natural number of 1 or more. As the value of N is increased, the amount of calculation for the correction function calculation process increases. Therefore, an appropriate N may be adopted by comparing the required accuracy with the amount of calculation. Since the light quantity balance can be corrected with sufficient accuracy even when N = 5, a natural number of 5 or less may be adopted as N. Since the image data ratio Rimg is an odd function, it is preferable from the viewpoint of calculation amount to adopt an odd natural number as N.

ＲｉｍｇとＦｎの差分二乗和が最小となるように、ａ（ｙ）とｂ（ｙ）を、ｙ座標毎に算出すればよい。また、係数ａ（ｙ）とｂ（ｙ）をｙ座標毎に算出するのではなく、ｙ座標に依存しない定数として算出しても構わない。 More specifically, the case where N = 1 is described as an example. In the image sensor 101 of the present embodiment, the first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 are juxtaposed in the x direction. Therefore, in the distance calculation device 110 according to the present embodiment, the relative positional deviation (image) between the A image (the subject image constituting the first image data) and the B image (the subject image constituting the second image data). Deviation occurs in the x direction. Considering the above, the loss of light quantity balance mainly occurs in the x direction. Therefore, in this embodiment, the correction function Fn can be expressed as Equation 5 as a function in the x direction.

What is necessary is just to calculate a (y) and b (y) for every y coordinate so that the sum of squared differences between Rimg and Fn is minimized. Further, the coefficients a (y) and b (y) may be calculated as constants that do not depend on the y coordinate instead of being calculated for each y coordinate.

最小二乗法の手法としては公知の手法を用いることができ、例えばＱＲ分解法や特異値分解を用いることができる。また、光ショットノイズ等のノイズの影響を低減するために、画像データ比Ｒｉｍｇに公知の平滑化フィルタを施してから補正関数Ｆｎを算出しても構わない。平滑化フィルタとしては、移動平均フィルタやガウシアンフィルタを用いることができる。 As another form of the correction function, the correction function Fn may be expressed as a function in the x direction and the y direction. For example, when the correction function is expressed by a cubic function, it is expressed as Equation 6 and each coefficient may be calculated using a known least square method. In this case as well, the correction function Fn may be calculated in a format independent of the y coordinate.

As a method of the least square method, a known method can be used. For example, a QR decomposition method or a singular value decomposition can be used. In order to reduce the influence of noise such as light shot noise, the correction function Fn may be calculated after applying a known smoothing filter to the image data ratio Rimg. As the smoothing filter, a moving average filter or a Gaussian filter can be used.

  FIG. 8D is a diagram for explaining steps S2-2 and S2-3. A solid line 831 in FIG. 8D indicates the image data ratio Rimg within the range 830 calculated according to Equation 4, and a broken line 841 indicates a correction function within the range 830 calculated in step S2-3 with N = 3. Fn. The horizontal axis is the image height in the x direction. Although the image data ratio is a data string having variations, a correction function in which the influence of variations is reduced can be obtained by approximating the image data ratio with an N-order function.

In step S2-4, a process of correcting the first image data using the correction function Fn is performed. The first corrected image data is image data obtained by correcting the first image data using the correction function Fn, and is hereinafter represented as Cimg1 (x, y). Similarly, the first image data is represented as Img1 (x, y). In the present embodiment, the first corrected image data Cimg1 is expressed by Equation 7
As shown in FIG. 5, the calculation is performed by multiplying the first image data Img1 by a function (1 + 1 / Fn: first image data correction function) obtained by adding 1 to the reciprocal of the correction function Fn.

In step S2-5, the second image data is corrected using the correction function Fn. The second corrected image data is image data obtained by correcting the second image data using the correction function Fn, and is hereinafter represented as Cimg2 (x, y). Similarly, the second image data is represented as Img2 (x, y). In the present embodiment, the second corrected image data Cimg2 is multiplied by a function (1 + Fn: second image data correction function) obtained by adding 1 to the second image data Img2 and the correction function Fn as shown in Expression 8. It is calculated by processing.

  FIG. 8E is a diagram for explaining steps S2-4 and S2-5. A solid line 851 in FIG. 8E indicates the first corrected image data Cimg1 within the range 830 calculated according to Equation 7, and a broken line 852 indicates the second corrected image data within the range 830 calculated according to Equation 8. Cimg2 is shown. The horizontal axis is the image height in the x direction. From a comparison between FIG. 8E and FIG. 8B, the use of the light amount balance correction process of the present embodiment can correct the collapse of the light amount balance. In step S3 (FIG. 3), the image shift amount is calculated using the first corrected image data Cimg1 and the second corrected image data Cimg2 in which the light quantity balance is corrected in this way.

  In the light intensity balance correction processing from step S2-1 to step S2-5 described above, the ratio (image data ratio) between the first partial image data and the second partial image data extracted from the first image data and the second image data. ) To calculate a correction function. When attention is paid only to the local region, the image data ratio is a ratio caused by the image shift depending on the subject distance in addition to the collapse of the light amount balance between the first image data and the second image data. However, a plurality of subjects (such as a person and a background) are arranged in the photographed image. Each subject is arranged at a different distance, and the image data ratio of the present embodiment is a data string having variations due to having various spatial frequency components. In the light quantity balance correction process of the present embodiment, the influence of image shift depending on the subject distance included in the image data ratio is reduced by calculating a correction function approximating the image data ratio. As a result, the collapse of the light amount balance is corrected based on the first image data and the second image data without acquiring data for correcting the light amount balance in advance. Alternatively, a correction function approximating the image data ratio may be calculated by focusing on a wide area within the image data ratio instead of the local area. In this way, not only the influence of image shift depending on the subject distance included in the image data ratio but also the influence of noise included in the first image data and the second image data can be reduced. In order to mitigate the effect of image shift depending on the subject distance included in the image data ratio, the first data range is wider than the reference area 420 used in the image shift amount calculation processing in step S3 described above. It is desirable to make it. More preferably, it is desirable that the first data range is wider than a range obtained by combining the reference region 420 and a predetermined image shift amount search range. In either case, it is possible to reduce the influence of noise caused by local computation.

In the light quantity balance correction process of the present embodiment, the correction function is calculated based on the first image data and the second image data, so that it is not necessary to acquire correction data in advance. Accordingly, even when the optical axis of the image forming optical system 120 is deviated from a predetermined position, the loss of light quantity balance can be corrected by the optical camera shake correction mechanism. Further, since it is not necessary to measure a correction coefficient for correcting the light amount balance after the digital camera 100 is assembled, the adjustment process at the time of manufacturing can be reduced, and the digital camera 100 can be manufactured at a lower cost. Can do.

  In the image shift amount calculation processing in step S3, when the corresponding point search is performed while sequentially moving the attention point 410, the reference regions overlap between adjacent attention points. In the subject calculation procedure of the present embodiment, since the image shift amount calculation processing in step S3 is performed after the light amount balance is corrected in step S2, the subject distance is corrected without performing the light amount balance correction in the overlapping region a plurality of times. Can be calculated. Therefore, the subject distance can be calculated with a small amount of calculation.

  In the subject distance calculation procedure of the present embodiment, the image shift amount calculation process is performed in step S3 (FIG. 3) based on the first correction image data and the second correction image data, thereby causing the light amount balance to be lost. This reduces the erroneous calculation of the amount of image shift.

<Modification of light intensity balance correction process>
Specific details of the light amount balance correction processing can be changed from the method described above. For example, the method for calculating the corrected image data based on the correction function and the method for calculating the correction function itself can be modified. First, a modified example of the corrected image data calculation process based on the correction function will be described, and then a modified example of the correction function calculation process will be described.

(Modification of corrected image data calculation method)
In the light amount balance correction process of the present embodiment, the first image data is corrected in step S2-4, and the second image data is corrected in step S2-5. As a modification of the light amount balance correction process of the present embodiment, only the first image data correction by the first image data correction process may be performed, and the second image data correction process may be omitted. In the first image data correction process in this case, the first corrected image data Cimg1 is calculated by multiplying the first image data Img1 and the inverse of the correction function Fn as shown in Equation 9.

  When the first corrected image data is calculated by the first image data correction process (Formula 9) instead of the processes of steps S2-4 and S2-5, the first corrected image data and the second corrected image data are calculated. Based on the image data (without correction), an image shift amount calculation process is performed in step S3. In the first image data correction process (Formula 9), the light quantity balance is corrected by matching the light quantity of the first image data with the second image data.

As a modification of the present embodiment, only the second image data correction by the second image data correction process may be performed, and the first image data correction process may be omitted. In the second image data correction process in this case, the second corrected image data Cimg2 is calculated by a multiplication process of the second image data and the correction function Fn as shown in Expression 10.

When the second corrected image data is calculated by the second image data correction process (Equation 10) instead of the processes of steps S2-4 and S2-5, the first image data (no correction) is used. Based on the second corrected image data, an image shift amount calculation process is performed in step S3. In the second image data correction process (Equation 10), the light quantity balance is corrected by matching the light quantity of the second image data with the first image data.

  In the above modification, only one of the first image data and the second image data is corrected, and the light quantity balance is corrected by adjusting the light quantity to the other light quantity. By performing the light amount balance correction process on only one of the image data, the amount of calculation required for correcting the light amount balance can be further reduced.

(Modification 1 of correction function calculation method)
Hereinafter, another embodiment of the light amount balance correction process, which is partially different from the light amount balance correction process described with reference to FIG. 7, will be described with reference to FIG. Steps S2-1 to S2-5 in FIG. 9 overlap with the description using FIG. 7, and therefore, steps S2-7 to S2-9, which are newly added processes, will be described below.

  Step S2-7 is a process (correlation calculation process) for calculating a correlation indicating the similarity between the correction function Fn and the image data ratio Rimg. The degree of correlation only needs to be a value representing the degree of correlation between two data. For example, a correlation coefficient that is a value obtained by dividing the covariance of the correction function Fn and the image data ratio Rimg by each standard deviation is used. Can do. Similarly to the description of step S2-3, in the distance calculation device 110 of the present embodiment, the correlation coefficient may be calculated for each y coordinate in consideration of the occurrence of image shift in the x direction.

  Step S2-8 is a process (correlation degree determination process) for comparing the correlation degree calculated in step S2-7 with a predetermined correlation degree determination threshold value to determine the level of the correlation degree. If it is determined that the correlation between the correction function Fn and the image data ratio Rimg is smaller than the correlation determination threshold (corresponding to the second threshold), the process proceeds to step S2-9, and the correlation determination threshold or more (first If it is determined that it is equal to or greater than the threshold of 2, the process proceeds to step S2-4.

  Step S2-9 performs processing for raising the order of the correction function Fn. If it is determined in step S2-8 that the degree of correlation is low, it is estimated that the approximation error of the correction function Fn is large. Therefore, after changing the order N of the correction function Fn to a larger value in step S2-9, the calculation process of the correction function Fn in step S2-3 is performed again. For example, a process of raising N to N + 1 is performed. Here, the increase amount of N may be a fixed value (for example, 1 or 2), but it is desirable to change the carry amount of the order according to the degree of correlation calculated in step S2-7. By changing the amount of increase in accordance with the degree of correlation, the number of executions of step S2-3 necessary for reducing the approximation error of the correction function Fn can be reduced, and the amount of calculation can be reduced. The following method can be adopted as a specific example of such a method. If the degree of correlation is smaller than 1/2 of the threshold value for determining the degree of correlation in step S2-8, the order N is incremented by 4 (for example, when the order N is 1, the order N is changed to 5). Step S2-3 is executed again). On the other hand, when the degree of correlation is not less than ½ of the threshold value for determining the degree of correlation in step S2-8 and smaller than the threshold value, the order N is incremented by 2 (for example, when the order N is 1). Change the order N to 3 and re-execute step S2-3).

  By further performing the processing from step S2-7 to step S2-9, the approximation error of the correction function is reduced, and the light intensity balance between the first image data and the second image data can be corrected more accurately. . As a result, the subject distance calculation error in the subject distance calculation procedure of the present embodiment can be reduced.

(Modification 2 of the correction function calculation method)
Hereinafter, another embodiment of the light amount balance correction process, which is partially different from the light amount balance correction process described with reference to FIG. 7, will be described with reference to FIG. Steps S2-1 to S2-5 in FIG. 10 are the same as those described with reference to FIG. 7, so step S2-10, which is a newly added process, will be described below.

  In step S2-10, a process of removing an error value (approximate point calculation data generation process) from which the influence of noise such as light shot noise or the effect of image shift due to subject distance is large is removed from the image data ratio Rimg. Do as follows. First, a differential filter is applied to the image data ratio Rimg to calculate a differential image data ratio. Then, only the data string within the image data ratio corresponding to the data position where the absolute value of the differential image data ratio is equal to or less than the predetermined approximate point determination threshold (less than the first threshold) is extracted, and a new image data ratio is generated.

  By further performing the process of step S2-10, the approximation error of the correction function is reduced, and the light quantity balance between the first image data and the second image data can be corrected with higher accuracy. As a result, the subject distance calculation error in the subject distance calculation procedure of the present embodiment can be reduced. In step S2-3 shown in FIG. 10, a correction function may be calculated by a spline method (spline interpolation). That is, the image data ratio after removing the error value may be divided into a plurality of regions in the x direction, and an Nth order polynomial representing an Nth order spline curve may be used as the correction function Fn.

  It should be noted that in the light quantity balance correction processing described with reference to FIG. 7, both the processing from step S2-7 to step S2-9 shown in FIG. 9 and step S2-10 shown in FIG. 10 may be performed. Absent. In that case, you may make it process by the flow shown in FIG. That is, instead of step S2-9, step S2-9 'in which the change of the approximate point discrimination threshold is added may be performed in addition to increasing the order of the correction function Fn. After performing Step S2-9 ', the process returns to either Step S2-10 or Step S2-3.

  In the light quantity balance correction processing described with reference to FIGS. 10 and 11, an error value is detected and removed by applying a differential filter to the image data ratio. Since the error value is caused by various noises (such as light shot noise) and image deviation that occur during image data acquisition, the correction function can be generated more accurately by removing the error value. As a result, the light quantity balance between the first image data and the second image data can be favorably corrected, and the subject distance can be calculated with high accuracy.

<Another Form of Image Sensor 101>
As another form of the image sensor 101 in this embodiment, as shown in FIG. 12A, two types of pixel groups 150 and 1250 may be arranged in a checkered pattern. The pixel group 150 is a pixel group in which a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162 are arranged in the x direction. By applying the above distance calculation processing to the image data obtained from the pixel group 150, the distance of the subject whose luminance changes in the x direction can be calculated. The pixel group 1250 is a pixel group in which a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162 are arranged in a pixel group 1250 arranged in the y direction. By applying the above distance calculation processing to the image data obtained from the pixel group 1250, the distance of the subject whose luminance changes in the y direction can be calculated.

  As another form of the image sensor 101, two types of ranging pixel groups may be employed as shown in FIG. The first ranging pixel group is a pixel group including pixels (green pixel 150G1, red pixel 150R1, blue pixel 150B1) having only the first photoelectric conversion unit 161. The second ranging pixel group is a pixel group including pixels (green pixel 150G2, red pixel 150R2, blue pixel 150B2) having only the second photoelectric conversion unit 162. As shown in FIG. 12B, the area of the first photoelectric conversion unit 161 is larger than that of the second photoelectric conversion unit 162. Difference data between the image data generated by the plurality of first photoelectric conversion units 161 in each pixel and the image data generated by the plurality of second photoelectric conversion units 162 in each pixel is the first data described above. By using image data, the distance to the subject can be calculated.

  It is sufficient that a relative positional deviation corresponding to the distance of the subject occurs between the image data generated by the first photoelectric conversion unit 161 and the image data generated by the second photoelectric conversion unit 162. Even in this form, the distance to the subject can be calculated.

  As another form of the image sensor 101, the first photoelectric conversion unit (161) of the first ranging pixel group shown in FIG. 12B passes through the first pupil region of the imaging optical system. You may comprise so that only a 1st light beam may be received. In this case, the image data generated by the first photoelectric conversion unit 161 is the first image data.

  It is not necessary to arrange the ranging pixels on the entire surface of the image sensor 101. The ranging pixels are arranged only in a part of the image sensor 101, and normal imaging pixels are arranged in other areas. Good. The imaging pixel receives and photoelectrically converts a light beam (first and second light beams) that passes through the entire exit pupil of the imaging optical system 120 to generate captured image data (third photoelectric conversion unit). Is provided. In this case, the ornamental image may be generated only from image data (captured image data) obtained from the photographic pixels, or may be generated from image data obtained from the photographic pixels and the ranging pixels. Since the photo-electric conversion unit has a larger volume of the photoelectric conversion unit than the distance measurement pixel, an image signal having a high S / N ratio can be obtained. By setting the area ratio between the area occupied by the ranging pixels and the area occupied by the imaging pixels as necessary, both the calculation accuracy of the subject distance and the image quality of the ornamental image can be achieved.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described in detail with reference to the drawings.

  In the following description, a digital camera will be used as an example of an imaging apparatus including the distance calculation device of the present invention, but the application of the present invention is not limited to this. For example, the distance calculation device of the present invention can be applied to a digital distance measuring device.

  In the description with reference to the figures, even if the figure numbers are different, in principle, the same reference numerals are given to the parts indicating the same parts, and overlapping explanations are avoided as much as possible.

  In FIG. 13, reference numeral 1300 denotes a digital camera provided with the distance calculation device 1310 of the present embodiment. The digital camera 1300 includes an imaging optical system 120, an image sensor 101, a distance calculation unit 102, an image generation unit (not shown), a lens drive control unit (not shown), a correction function storage memory unit 1301, and an imaging condition acquisition unit 1302. Are arranged inside the camera casing 130. The distance calculation device 1310 includes an imaging optical system 120, an image sensor 101, a distance calculation unit 102, a correction function storage memory unit 1301, and an imaging condition acquisition unit 1302.

  In the correction function storage memory unit 1301, correction functions calculated at the time of past shooting are stored in association with shooting conditions at the time of shooting.

  The imaging optical system 120 is a photographing lens of the digital camera 1300 and has a function of forming an image of a subject on the image sensor 101 that is an imaging surface. The imaging optical system 120 includes a plurality of lens groups (not shown), a diaphragm (not shown), and an optical camera shake correction mechanism (not shown), and has an exit pupil 103 at a position away from the image sensor 101 by a predetermined distance. The optical camera shake correction mechanism is composed of a correction lens having a gyro mechanism, and corrects the shake of the optical axis by operating the correction lens in a direction to cancel the camera shake. The imaging condition acquisition unit 1302 acquires imaging condition information such as focal length information, aperture value information, and an optical axis shake correction amount (a shift amount of the optical axis from a set value) of the imaging optical system 120. .

<Detailed description of light quantity balance correction processing>
Hereinafter, the light amount balance correction processing of the present embodiment will be described in detail with reference to FIG. Note that steps S2-1 to S2-5 in FIG. 14 are the same as those described with reference to FIG. 7, and therefore, in the following description, steps S2-11 to S2-14 will be described.

  In step S2-11, processing for acquiring shooting condition information from the shooting condition acquisition unit 1302 is performed.

  Step S2-12 is processing for determining whether or not the correction function Fn already stored in the correction function storage memory unit 1301 can be used from the photographing condition information. If the shooting condition when the first image data and the second image data are acquired differs from the shooting condition when the correction function Fn stored in the correction function storage memory unit 1301 is generated, a new correction is made. As it is necessary to generate the function Fn, the process proceeds to step S2-1. On the other hand, if the shooting conditions are substantially the same when the correction function Fn stored in the correction function storage memory unit 1301 is generated, the process proceeds to step S2-13. As the photographing condition information, for example, a focal length, an aperture value, and an optical axis shift amount can be used.

  In step S2-13, the correction function Fn is read from the correction function storage memory unit 1301.

  In step S2-14, the correction function Fn calculated in step S2-3 is stored in the correction function storage memory unit 1301 together with the shooting conditions.

  The distance calculation device 1310 according to the present embodiment performs the light amount balance correction processing described with reference to FIG. 14 in the distance calculation unit 102. According to this, when there are the same shooting conditions in the past, the correction function generated at that time is read and reused to reduce the processing for calculating the correction function and calculate the subject distance in a shorter time. Is possible.

  In the present embodiment, the light intensity balance correction process of FIG. 14 is used. However, the processes of steps S2-11 to S2-14 are added to the light intensity balance correction process described with reference to FIGS. It does not matter if it is a process. In either case, the subject distance can be calculated in a shorter time by reusing the correction function.

  The specific implementation of the distance calculation device in the first and second embodiments described above can be implemented either by software (program) or by hardware. For example, each process may be realized by storing a computer program in a memory of a computer (microcomputer, CPU, MPU, FPGA, or the like) built in the imaging apparatus or the image processing apparatus, and causing the computer program to execute the computer program. . It is also preferable to provide a dedicated processor such as an ASIC that implements all or part of the processing of the present invention by a logic circuit. The present invention is also applicable to a server in a cloud environment.

For example, the present invention can be implemented by a method including steps executed by a computer of a system or apparatus that implements the functions of the above-described embodiments by reading and executing a program recorded in a storage device. . For this purpose, the program is stored in the computer from, for example, various types of recording media that can serve as the storage device (ie, computer-readable recording media that holds data non-temporarily). Provided to. Therefore, the computer (including devices such as CPU and MPU), the method, the program (including program code and program product), and the computer-readable recording medium that holds the program in a non-temporary manner are all present. It is included in the category of the invention.

DESCRIPTION OF SYMBOLS 102 Distance calculation part 103 Exit pupil 110 Distance calculation apparatus 120 Imaging optical system

Claims (16)

A first based ku first image data to the light beam passing through the first pupil area of the image forming optical system, unlike the first pupil region of the imaging optical system and eccentric to the first direction and based Ku second image data to the second light flux passing through the second pupil area are, the image data acquisition means for acquiring,
Image data ratio calculating means for calculating an image data ratio that is a ratio of at least a part of the first image data and the second image data along the first direction ;
Correction function calculating means for calculating a correction function in the first direction based on the image data ratio;
Correction means for correcting at least one of the first image data and the second image data based on the correction function ;
An image data processing apparatus comprising: The image data ratio calculation means calculates a ratio between the first partial image data included in the first image data and the second partial image data included in the second image data, and the image data ratio To calculate,
The image data processing apparatus according to claim 1. The correction function calculation means uses only data included in the image data ratio at a position where the absolute value of the differential image data ratio obtained by applying a differential filter to the image data ratio is equal to or less than a predetermined first threshold value. To calculate the correction function,
The image data processing apparatus according to claim 1 . The correction function is an N-order function (N is a natural number).
The image data processing apparatus according to claim 1 . Calculating a correlation indicating a correlation between the correction function and the image data ratio;
Determining whether the correlation is greater than or equal to a second threshold;
If it is determined that the degree of correlation between the correction function and the image data ratio is lower than the second threshold, the correction function calculation means calculates the correction function by increasing the value of the order N. Do again,
The image data processing apparatus according to claim 4 . When it is determined that the degree of correlation between the correction function and the image data ratio is lower than the second threshold, the order when the correction function is calculated again according to the value of the degree of correlation Set N,
The image data processing apparatus according to claim 5 . Correction function storage means for storing a predetermined correction function determined in advance in association with the imaging conditions, and imaging condition acquisition means for acquiring the imaging conditions when the first image data and the second image data are acquired When,
Further comprising
When the predetermined correction function corresponding to the shooting condition acquired by the shooting condition acquisition unit is stored in the correction function storage unit, the correction unit performs correction using the stored correction function. If not, perform correction using the correction function calculated by the correction function calculation means,
Image data processing apparatus according to any one of claims 1 to 6. The area of the first pupil region is larger than the area of the second pupil region,
The first pupil region includes the second pupil region;
Image data processing apparatus according to any one of claims 1 to 7. And an image data processing apparatus according to any one of claims 1 to 7,
A distance calculation unit that calculates an image shift amount, which is a relative positional shift amount between the first image data and the second image data, using the image data corrected by the correction unit;
A distance calculation device comprising: An image data processing device according to claim 8 ,
Using the image data corrected by the correction means, the second image data and a relative relationship between the third image data that is difference data between the first image data and the second image data. Distance calculating means for calculating an image misalignment amount that is a large misalignment amount;
A distance calculation device comprising: The distance calculating means calculates a defocus amount or a distance to the subject based on the image shift amount.
The distance calculation apparatus according to claim 9 or 10 . An imaging optical system;
First image data based on a first light flux passing through a first pupil region of the imaging optical system and a second image based on a second light flux passing through a second pupil region of the imaging optical system An image sensor for acquiring data;
The image data processing device according to any one of claims 1 to 8 , or the distance calculation device according to any one of claims 9 to 11 ,
An imaging apparatus comprising: The imaging device photoelectrically converts the first light flux to generate the first image data, and photoelectrically converts the second light flux to generate the second image data. Ranging pixels including a second photoelectric conversion unit, and imaging pixels including a third photoelectric conversion unit that photoelectrically converts the first and second light beams to generate captured image data.
The imaging device according to claim 12 . The imaging element photoelectrically converts the first light flux and first ranging pixels including a first photoelectric conversion unit that generates the first image data, and photoelectrically converts the second light flux. A second ranging pixel including a second photoelectric conversion unit that generates the second image data, and a third photoelectric conversion unit that generates captured image data by photoelectrically converting the first and second light beams. A photographic pixel comprising
The imaging device according to claim 12 . The first image data based on the first light flux passing through the first pupil region of the imaging optical system and the first image data different from the first pupil region of the imaging optical system and decentered in the first direction. Image data acquisition step for acquiring second image data based on the second light flux passing through the two pupil regions;
An image data ratio calculating step of calculating an image data ratio that is a ratio of at least a part of the first image data and the second image data along the first direction ;
A correction function calculating step of calculating a correction function in the first direction based on the image data ratio;
A correction step of correcting at least one of the first image data and the second image data based on the correction function ;
Including an image data processing method. A computer program for causing a computer to execute each step of the image data processing method according to claim 15 .

Images