JP2017194591A - Distance measurement device, imaging apparatus, and distance measurement method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a distance measurement device for an imaging apparatus having an anti-vibration mechanism capable of accurately obtaining distance information.SOLUTION: The distance measurement device includes: image blur correction means; distance information acquisition means that acquires a piece of distance information based on a first image formed by a light flux passing through a first pupil area in a focusing optical system and a second image which is formed on the basis of a light flux passing through a second pupil area of the focusing optical system; and distance information correction means that corrects the distance information on the basis of an image blur correction amount calculated by the image blur correction means.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a distance measurement technique, and more particularly to a distance measurement technique used for a digital still camera, a digital video, or the like.

  As a distance detection technique applicable to a digital still camera or a video camera, a distance detection technique based on a phase difference method is known. This method acquires optical images (hereinafter referred to as “A image” and “B image”, respectively) formed by light beams that have passed through different pupil regions. The image displacement amount (also called parallax), which is the relative positional displacement amount between the A and B images, is calculated, and based on the baseline length, which is the center of gravity interval that forms the A and B images on the lens pupil. The distance to the subject can be calculated by converting the defocus amount through the conversion factor.

  At this time, it is known that the conversion coefficient changes due to various factors in the peripheral angle of view. Causes of changes include, for example, pupil shape changes and chromatic aberration characteristics due to the effects of vignetting (also referred to as “vignetting”) caused by vignetting of the lens frame of the imaging optical system, and spectral sensitivity to the incident angle of the ranging pixels. Characteristics, camera manufacturing errors, etc. In the first place, the conversion coefficient also changes in accordance with the defocus amount itself. Japanese Patent Application Laid-Open No. H10-228561 describes a method for correcting fluctuations in a conversion coefficient.

  In recent years, various image stabilization functions have been installed in digital still cameras and video cameras in order to prevent image blur caused by camera shake. For example, Patent Document 2 discloses a technique for suppressing image blur by shifting a part of a lens system. Specifically, since the amount of light and the state of distortion change due to the shift of the lens system, Patent Document 2 performs distortion correction according to position information of the image blur correction optical system.

JP2015-212772A JP 2012-231262 A JP 2013-222183 A

  An imaging device (digital still camera or video camera) provided with a distance measuring device is preferably portable. The distance map of the intended composition can be obtained by carrying the camera and photographing from an appropriate position. In addition, by photographing from near, a high-density and high-precision distance map can be obtained due to the geometric principle and the characteristics of the digital system.

  Portable cameras are vulnerable to camera shake, so it is desirable to have a vibration isolation mechanism. However, if image correction is performed in accordance with the amount of light and image distortion that vary according to the method described in Patent Document 2, fluctuations in image brightness, distortion, and the like can be suppressed, while ranging is possible. The conversion factor concerned will fluctuate. That is, there is a problem that an error occurs in distance measurement.

  The present invention has been made in consideration of the above-described problems, and an object of the present invention is to provide a technique that enables accurate distance measurement even when a vibration isolation mechanism is effective in an imaging apparatus.

The first aspect of the present invention is:
Image blur correction means;
Based on the first image based on the light beam passing through the first pupil region of the imaging optical system and the second image based on the light beam passing through the second pupil region of the imaging optical system, the distance Distance information acquisition means for acquiring information;
Distance information correction means for correcting the distance information based on an image blur correction amount by the image blur correction means;
It is a distance measuring device provided with.

The second aspect of the present invention is:
Based on the first image based on the light beam passing through the first pupil region of the imaging optical system and the second image based on the light beam passing through the second pupil region of the imaging optical system, the distance A distance information acquisition step for acquiring information;
An acquisition step of acquiring an image blur correction amount by the image blur correction means;
A distance information correction step of correcting the distance information based on the image blur correction amount;
Is a distance measuring method.

  According to the present invention, distance measurement can be performed with high accuracy even in an imaging apparatus in which a vibration isolation mechanism is effective.

1 is a schematic configuration diagram of a distance measuring device according to Embodiment 1. FIG. It is a figure which shows the basic processing in Example 1. It is a figure explaining the anti-vibration function of the distance measuring device which concerns on Example 1. FIG. It is a figure explaining the geometric deformation | transformation of the conversion factor map in Example 1. FIG. It is a figure which shows the ranging process of the arithmetic processing part in Example 1. It is a figure explaining the conversion factor map of the vector element in Example 2. FIG. It is a figure which shows the ranging process of the arithmetic processing part in Example 2. 3 is a top view of the main part of the image sensor 103. FIG. It is a figure explaining the change of the pupil transmittance distribution by the vignetting in Example 3. It is a figure which shows the ranging process of the arithmetic processing part in Example 3. It is a figure explaining the conversion coefficient in Example 3, and an image correction coefficient map. It is a figure explaining the image correction filter coefficient in Example 3. FIG.

(Embodiment 1)
The present embodiment is an imaging apparatus including a distance measuring device that obtains distance information. The distance information is a value corresponding to the distance between a certain reference point and the subject. Typically, the distance (absolute distance) between the distance measuring device and the subject corresponds to the distance information. In general, the distance information may be another value obtained from the distance between the imaging device and the subject. Another example of the distance information is a distance between the focus position of one image and the subject, and a distance between the intermediate position of the focus position of two images and the subject. The distance information may be either a distance on the image plane side or a distance on the object side. The distance information may be represented by a distance in real space, or may be represented by an amount that can be converted into a real space distance, such as a defocus amount or a parallax amount. Thus, the distance information is a value that changes depending on the distance (absolute distance) between the imaging device and the subject. Therefore, the distance information can also be referred to as a distance dependent value.

  Hereinafter, this embodiment will be described in detail with reference to the drawings.

<Configuration of distance measuring device>
FIG. 1A is a schematic diagram illustrating an imaging device 100 including a distance measuring device according to the present invention. The imaging apparatus 100 includes an imaging optical system 101, a distance measuring device 102, an attitude sensor 140, and an image blur correction unit 150. The distance measuring device 102 includes an arithmetic processing unit 104 and a memory 105. The imaging optical system 101 includes an image blur correction member 1011 (not shown in FIG. 1A).

The arithmetic processing unit 104 is configured by a logic circuit such as an ASIC (Application Specific Integrated Circuit). In another embodiment, the arithmetic processing unit 104 includes a general-purpose processor such as a CPU (Central Processing Unit) and a memory that stores a program, and the general-purpose processor executes the following process by executing the program. The arithmetic processing unit 104 functions as a distance information acquisition unit and a distance information correction unit as a whole. More specifically, as shown in FIG. 5A, the arithmetic processing unit 104 includes an image acquisition unit 11, an image shift amount calculation unit (first acquisition unit) 12, an image blur correction amount acquisition unit 13, and a conversion unit. A coefficient calculation unit (third acquisition unit) 14 and a distance calculation unit (second acquisition unit) 15 are provided as functional units. Specific descriptions of these functional units will be described later.

  FIG. 1B is a diagram illustrating a configuration of the imaging element 103. The image sensor 103 is configured by juxtaposing a large number of ranging pixels (hereinafter also simply referred to as “pixels”) 110R, 110G, and 110B arranged on the xy plane.

  FIG. 1C shows a cross-sectional configuration diagram of the ranging pixels 110R, 110G, and 110B. The ranging pixel includes a micro lens 111, color filters 122R, 122G, 122B, photoelectric conversion units 110Ra, 110Rb, 110Ga, 110Gb, 110Ba, 110Bb, and a waveguide 113. Hereinafter, the photoelectric conversion units 110Ra, 110Ga, and 110Ba may be collectively referred to as the photoelectric conversion unit 110a, and the photoelectric conversion units 110Rb, 110Gb, and 110Bb may be collectively referred to as the photoelectric conversion unit 110b.

  Each pixel of the image sensor 103 is given spectral characteristics according to the wavelength band to be detected by the color filters 122R, 122G, and 122B, and mainly acquires red light, green light, and blue light, respectively. The ranging pixels 110R, 110G, and 110B are arranged on the xy plane by a known color arrangement pattern (for example, a Bayer array) (not shown). The substrate 124 is made of a material that absorbs a wavelength band to be detected, such as Si. A photoelectric conversion portion is formed in at least a partial region of the substrate 124 by ion implantation or the like. Each pixel includes a wiring (not shown).

  The light fluxes that have passed through the first pupil region 131a and the second pupil region 131b, which are different regions in the exit pupil 130, are incident on the photoelectric conversion unit 110a and the photoelectric conversion unit 110b, respectively. 2 signal is output. A photoelectric conversion unit that acquires a first signal that forms an A image is referred to as an A pixel, and a photoelectric conversion unit that acquires a second signal that forms a B image is referred to as a B pixel. A signal acquired by each photoelectric conversion unit is transmitted to the calculation processing unit 104, and distance measurement calculation processing is performed.

<Ranging calculation basic flow>
An example of a basic flowchart of the distance measurement calculation process is shown in FIG.

[Image shift amount calculation]
In the image shift amount calculation processing in step S201, the image acquisition unit 11 reads the A image (first signal image) and the B image (second signal image) from the memory, and the image shift amount calculation unit 12 reads the A image. An image shift amount that is a relative positional shift amount between the image B and the B image is calculated. A known method can be used to calculate the image shift amount. The correlation value S (k) is calculated from the image signal data A (i) and B (i) of the A image and the B image using the equation (1), for example.

  In Expression (1), S (j) is a correlation value indicating the degree of correlation between two images in the image shift amount j, i is a pixel number, and j is a relative image shift amount of the two images. p and q indicate the target pixel range used for calculating the correlation value S (j). The correlation value S (j) means that the lower the value, the higher the correlation.

  The image shift amount calculation unit 12 calculates a correlation value S (j) for a predetermined range j, and determines an image shift amount j that gives a minimum value as an image shift amount between the A image and the B image at the position i. Note that the method of calculating the image shift amount is not limited to the method of this embodiment, and other known methods may be used.

[Defocus amount calculation]
In the distance calculation process in step S202, the distance calculation unit 15 calculates the defocus amount from the image shift amount. An image of the subject 106 is formed on the image sensor 103 via the imaging optical system 101. FIG. 1A shows a state in which a light beam that has passed through the exit pupil 130 is focused on an image plane 107 that is different from the imaging surface, that is, a defocused state. Note that defocusing refers to a state in which the imaging surface 107 does not coincide with the imaging surface (light receiving surface) and is displaced in the direction of the optical axis 108. The defocus amount indicates the distance between the imaging surface of the image sensor 103 and the imaging surface 107. The distance measuring device according to the present embodiment detects the distance of the subject 106 based on the defocus amount.

The image shift amount r indicating the relative positional shift between the A image and the B image that depends on the first signal and the second signal acquired by the photoelectric conversion units 110a and 110b of the pixel 110, and the defocus amount ΔL , Formula (2).

  In Expression (2), W is the base line length, and L is the distance from the image sensor (imaging surface) 103 to the exit pupil 130. The baseline length W corresponds to the center-of-gravity interval of the pupil sensitivity distribution obtained by projecting the sensitivity distribution with respect to the incident angle of the pixel, which will be described later, onto the exit pupil 130 plane.

Expression (2) can be written as Expression (3) using the conversion coefficient K.

  A coefficient for converting the image shift amount into the defocus amount is hereinafter referred to as a “conversion coefficient”. The conversion coefficient refers to, for example, the baseline length W. Thereafter, the correction of the baseline length W is equivalent to the correction of the conversion coefficient. In the present embodiment, the conversion coefficient calculated by the conversion coefficient calculation unit 14 is used for distance calculation. A method for determining (correcting) the conversion coefficient will be described later.

  Note that the defocus amount calculation method is not limited to the above method, and other known methods may be used.

  When the photographic lens is a zoom lens, when the exit pupil position and the sensor pupil position change due to a change in zoom state, the shape of the transmittance distribution (received light amount distribution) on the exit pupil changes. This change depends on the image height. Therefore, the value of the base line length W, which is the distance between the centers of gravity of the transmittance distribution on the pupil, and the conversion coefficient K related to the base line length W are also different values depending on the image height. The base line length W or the conversion coefficient K corresponding to the change in the transmittance distribution is stored in the memory 105 as a two-dimensional table corresponding to the pixel position as shown in FIG.

<Configuration of vibration isolation mechanism>
The suppression of image blur by the image blur correction member 1011 will be described. In the present embodiment, the image blur correction unit 150 translates a part of the lenses (correction member 1011) of the imaging optical system 101 as shown in FIG. As a result, the position of the optical image on the image sensor moves in parallel, and image shake due to camera shake, so-called shake, can be suppressed.

  Instead of providing the correction member 1011 in the imaging optical system 101, a mechanical drive component such as a piezo element that translates the imaging element 103 in a plane perpendicular to the optical axis of the imaging optical system 101 (corresponding to a correction member) May be provided to suppress image blurring. Alternatively, for example, an image stabilization mechanism that controls the range of an image output as an image signal from the image sensor 103 and electronically prevents vibration due to camera shake may be provided.

  The camera shake is detected by the attitude sensor 140. The attitude sensor 140 is constituted by a gyro, for example, and is attached to an arbitrary axis orthogonal to the optical axis of the optical system. The posture sensor 140 measures posture change information and transmits the measured information to the image blur correction unit 150. For example, when the attitude sensor 140 is a rotation sensor, the attitude sensor 140 is attached to each axis in the yaw direction and the pitch direction of the imaging apparatus, and measures an attitude change due to rotation around each axis. The posture sensor 140 may be a sensor that measures the parallel movement of the imaging device such as an acceleration sensor, or a sensor that calculates the amount of image blur due to the parallel movement of the imaging device. In addition, the posture sensor 140 may be a sensor that calculates the amount of image blur due to relative rotation and parallel movement of the imaging device from the image vector.

  The image blur correction unit 150 controls the position and posture of the correction member 1011 so as to cancel the image blur. The movement amount of the correction member 1011 is calculated based on the posture and / or position change of the imaging apparatus measured by the posture sensor 140 or image blur caused by the posture and / or position change. The correction member 1011 is moved by driving a unit including the correction member 1011 with an electromagnetic coil or the like. The image blur correction unit 150 has a function of detecting position and orientation information of the correction member 1011. The image blur correction unit 150 includes a Hall element, an encoder, or the like in the configuration, and measures position and orientation information of the correction member 1011 based on the position of the lens assembly unit that performs optical image stabilization. Specifically, the image blur correction unit 150 monitors the position of the image center moved by the parallel movement of the correction member 1011. The image center is a point on the image where a light beam passing on the optical axis of the camera intersects the imaging element surface in a state where the optical axis of the lens is not decentered. The image center position on the image sensor surface moves as the correction member 1011 moves. In addition, the parallel movement amount of the optical image and the parallel movement amount of the image center are naturally equivalent. When high accuracy is not required, the image blur correction unit 150 may directly output control information of the correction member 1011.

<Anti-Vibration>
First, FIG. 3A shows the state of optical image stabilization by the parallel movement of the lens portion of the optical system. An optical image from the subject 106 is formed on the surface of the image sensor 103 by the optical system 101. A broken line indicates an optical path when the correction member 1011 does not perform image blur correction. On the other hand, a solid line indicates an optical path when the correction member 1011 moves upward by image blur correction. The subject image is formed at the center of the image sensor 103 by image blur correction. In other words, the state shown by the solid line in FIG. 3A is the image of the subject (dotted line) formed on the lower peripheral portion of the image sensor without image blur correction, as shown in FIG. 3B. Is formed in the center of the image sensor by image blur correction (movement of the correction member). FIG. 3B shows a change in the position of the optical image on the image sensor due to image blur correction. The broken line illustration is the position of the image before image blur correction, and the solid line illustration is the position of the image after image blur correction. Further, the x mark 301 is the position of the image center before image blur correction, and the black circle 302 is the position of the image center after image blur correction. The center of the image is a point on the image where a light beam passing on the optical axis of the camera intersects with the imaging element surface in a state where the optical axis of the lens is not decentered. The image center moves according to the movement of the correction member 1011. In other words, the image center moves in accordance with the movement of the image by the image blur correction even if it matches the center coordinates of the image when the image blur correction is not performed.

  FIG. 3C illustrates a case where the correction member 1011 is included in the image sensor 103 (when the correction member 1011 moves the image sensor 103). The correction member 1011 is moved by the image blur correction, and thereby the optical image is moved to the center of the imaging region as in FIG. FIG. 3D shows a captured image 303 in the situation shown in FIG. A cross mark 304 indicates the position of the image center when no image blur correction is performed, and a black circle mark 305 indicates the position of the image center when image blur correction is performed. Although the correction member 1011 is moved in the opposite direction to the case of FIG. 3A, the effect obtained as shown in FIG. 3D is the same as that of FIG. That is, whether the correction member 1011 is included in the imaging optical system 101 or the image sensor 103 does not cause a substantial difference in the correction effect. Therefore, in the following description, it is not distinguished whether the correction member 1011 is included in the imaging optical system 101 or the imaging element 103.

  The movement of the optical image with respect to the image sensor 103 causes a change in the correspondence between the pixel and the conversion coefficient. That is, if there is image blur correction, the conversion coefficient map in the case where there is no image blur correction as shown in FIG. 4A cannot be used as it is. Therefore, the conversion coefficient calculation unit 14 changes the conversion coefficient map based on the image blur correction amount (the movement amount of the optical image), and acquires an appropriate conversion coefficient for the pixel for calculating the defocus amount.

<Conversion factor determination process>
When the image blur correction (anti-vibration mechanism) is used as described above, it is necessary to adjust the conversion coefficient map in the distance calculation process. Hereinafter, distance calculation processing (distance information acquisition processing) in consideration of image blur correction will be described with reference to a flowchart shown in FIG.

  In step S501 (pixel selection processing), the arithmetic processing unit 104 selects the position on the image sensor of the pixel for which the distance is calculated. In the following description, the pixel selected in step S501 is referred to as a target pixel.

  In step S <b> 502 (image blur correction amount acquisition processing), the image blur correction amount acquisition unit 13 acquires the image blur correction amount from the image blur correction unit 150. As described above, the image blur correction amount is information related to the amount of image movement with respect to the image sensor. The image blur correction amount may be a movement amount of the correction member 1011.

In step S503 (conversion coefficient determination process), the conversion coefficient calculation unit 14 determines a conversion coefficient (a coefficient for converting an image shift amount into a defocus amount as shown in Expression (3)) for the target pixel. The conversion coefficient calculation unit 14 corrects the conversion coefficient map according to the optical image moved by the image blur correction based on the image blur correction amount. The conversion coefficient calculation unit 14 determines the conversion coefficient of the target pixel from the corrected conversion coefficient map. The process of step S503 can be regarded as a conversion coefficient correction process for correcting the conversion coefficient, or can be regarded as a distance information correction process for correcting the defocus amount (distance information).

  It is preferable that the conversion coefficient map correction method varies depending on the amount of image blur correction. In the present embodiment, the processing is different for each case where the image blur correction amount is (1) small, (2) slightly large, and (3) large. In the following, the processing in each case will be described first, and then the threshold value of the image blur correction amount serving as a reference for case classification will be described.

[1. When image blur correction amount is small]
When the image blur correction amount is small, the conversion coefficient map relating to the state where the image blur correction amount is 0 may be translated based on the movement amount of the pixel on the imaging surface. FIG. 4A shows a conversion coefficient map for a state where the image blur correction amount is 0, and FIG. 4B shows a corrected conversion coefficient map. In this example, the image blur correction amount (x, y) = (1,0). In this case, if the corrected conversion coefficient map is K ′ (x, y), K ′ (x, y) = K (x + 1, y).

  Since the image blur correction amount changes continuously, the movement amount of the conversion coefficient map can be a subpixel value. In that case, the conversion coefficient map may be recreated using interpolation. At this time, the transform coefficient calculation unit 14 realizes re-creation of the map by backward mapping that samples and interpolates the input map data based on the output map coordinates so that the output map data is not defective. The interpolation method is not particularly limited. The conversion coefficient calculation unit 14 performs an interpolation process using pixel values in the vicinity of the calculated sampling coordinates on the input map according to the interpolation method. As the interpolation processing, for example, bilinear interpolation processing that performs linear interpolation using 4 neighborhoods, bicubic interpolation processing that performs cubic interpolation using 16 neighborhoods, and the like can be used.

Bilinear interpolation will be described as a typical example with reference to FIG. A black circle 601 indicates a corresponding point (sampling point) of a certain pixel on the output map on the input map. The x and y coordinates of sampling points are generally non-integer. The conversion coefficient calculation unit 14 obtains output map data by bilinear interpolation using peripheral four input map data (data indicated by white circles) around the sampling coordinates. More specifically, the transform coefficient calculation unit 14 has four neighbors from the memory, that is, ([x], [y]), ([x + 1], [y]), ([x], [y + 1]), ([x + 1 ], [Y + 1]) are read out. [X] represents the integer part of the value x. Then, the conversion coefficient calculation unit 14 calculates a map value of the output map according to the following formula.

  Here, K (x, y) is an input map (that is, a conversion coefficient map with an image blur correction amount of 0), and K ′ (x, y) is an output map (that is, a conversion coefficient considering image blur correction). Map).

  The conversion coefficient map can be interpolated by sequentially performing the above processing. However, actually, it is not necessary to recreate the entire conversion coefficient map, and it is sufficient to obtain only the conversion coefficient of the position of the target pixel acquired in step S501 (pixel selection processing).

[2. When image blur correction amount is slightly large]
When the image blur correction amount is slightly increased, the movement of the correction member 1011 causes the influence of the eccentric distortion on the parallel movement of the optical image. Therefore, the image does not simply move in parallel, but geometric deformation such as enlargement / reduction or tilting also occurs. However, if the movement of the optical image due to image blur correction is not so large, the change in the amount of light is small. Therefore, the conversion coefficient corresponding to the movement of the optical image is approximated by geometric conversion (generally projective conversion) of the conversion coefficient map. You can recreate the map. That is, the conversion coefficient calculation unit 14 deforms the conversion coefficient map when the image blur correction amount is 0 in accordance with the translational movement or enlargement / reduction / tilting movement of the optical image.

[3. When image blur correction amount is large]
When the amount of image blur correction is larger, the change in the amount of light due to geometric deformation of the optical image cannot be ignored. In this case, a conversion coefficient map for the image height is created and held in advance for each of the x and y parameters of the image blur correction amount as well as the zoom state, subject distance, and aperture value of the optical system 101. Is possible. The conversion coefficient calculation unit 14 may acquire a conversion coefficient map that matches the conditions and determine a conversion coefficient to be applied to the target pixel.

  Alternatively, the conversion coefficient map may be deformed according to the image blur correction amount. For this purpose, for example, the change in the conversion coefficient at each image position accompanying the change in the image blur correction amount is held as a coefficient (fitting parameter) of the hyperplane fitting function. The conversion coefficient calculation unit 14 can also obtain a conversion coefficient map corresponding to the image blur correction amount by applying a fitting function to the conversion coefficient map and deforming it.

  In addition, a conversion coefficient map is created and stored in advance for a plurality of image blur correction amounts, and the above processing (1) or (2) is performed based on the conversion coefficient map closest to the actual image blur correction amount. Thus, the conversion coefficient applied to the target pixel may be determined. That is, a conversion coefficient map corresponding to the position in the image plane is stored in the memory for a plurality of predetermined image blur correction amounts, and the conversion coefficient calculation unit 14 converts the conversion coefficient closest to the actual image blur correction amount. A conversion coefficient to be applied to the target pixel may be determined by translating or projecting the map.

[Case classification criteria]
As described above, it is preferable to change the processing according to the magnitude of the image blur correction amount. Here, the threshold value of the amount of image blur, which is a criterion for classification, differs depending on the imaging optical system 101, and may be determined by an appropriate method. For example, when various image blur correction amounts are applied to various subjects with known distances, the distance information is obtained by the above three processes. The image blur correction amount at which the distance measurement error by the first method is greater than or equal to the allowable range can be determined as a first threshold value that distinguishes between the case where the image blur correction amount is small and the case where the image blur correction amount is slightly large. . Similarly, an image blur correction amount at which the distance measurement error according to the second method is greater than or equal to an allowable range is set to a second threshold value that distinguishes between “a little image blur correction amount” and “a large image blur correction amount”. Can be determined as

  The conversion coefficient calculation unit 14 uses the first method when the image blur correction amount is equal to or smaller than the first threshold, the second method when the image blur correction amount is larger than the first threshold and equal to or smaller than the second threshold, and the second method. When it is larger than the threshold value, the conversion coefficient is acquired according to the third method. It is technically synonymous to change the process when the image blur correction amount is equal to or smaller than the threshold value and when the image blur correction amount is smaller than the threshold value and when the image blur correction amount is equal to or greater than the threshold value. Accordingly, instead of determining whether or not the image blur correction amount is equal to or smaller than the threshold value, it may be determined whether or not the image blur correction amount is smaller than the threshold value.

  The image shift amount calculation process in step S504 is the same process as step S201 in FIG. In step S505 (distance calculation processing), the distance calculation unit 15 calculates a defocus amount from the image shift amount by using the value of the conversion coefficient acquired in step S503, using equation (3).

The conversion coefficient map for the state where the image blur correction amount is 0 can be determined as follows, for example. First, it is at a known distance in a predetermined distance measurement condition such as the state of the imaging optical system (focal length, aperture, focus distance, defocus amount) and the position of the distance measurement pixel on the image sensor 103 (image height). Shoot various subjects. A conversion coefficient map is calculated from a signal acquired from each subject using an image shift amount and a predetermined calculation formula. Further, the conversion coefficient is adjusted so that the actual distance of each subject is close to the distance obtained from the image shift amount and the conversion coefficient. Alternatively, the conversion coefficient map may be determined by performing a similar method by numerical simulation based on the design values of the imaging optical system 101 and the distance measuring device 102.

(Embodiment 2)
The image shift amount may not be zero even at the best focus position where the defocus amount should be zero due to the influence of the tilt of the image sensor surface, spherical aberration, axial chromatic aberration, and the like. Further, the best focus position may vary depending on the RGB pixel and the image height. In the distance calculation processing, this influence can be corrected by adopting the coefficient M as shown in Equation (5).

  In this case, the correction value M also varies depending on the image height. That is, in order to obtain the defocus amount with a small error in consideration of the shift of the best focus position due to the image height, it is necessary to store the coefficient K related to the baseline length and the correction amount M of the best focus position in the memory 105. There is. In this embodiment, the two-element vector parameter composed of the coefficient K and the coefficient M is stored in the memory 105 as a two-dimensional table corresponding to the pixel position as shown in FIG. That is, in this embodiment, the vector parameter (K, M) corresponds to the conversion coefficient, and the table T (x, y) = (K (x, y), M) in which the coefficient K and the coefficient M are stored for each pixel position. (X, y)) corresponds to the conversion coefficient map.

  FIG. 7 shows a flowchart of the distance measurement calculation process in this embodiment. As shown in FIG. 7, the processing content is the same as that of the first embodiment.

  In the present embodiment, in step S603 (conversion coefficient determination process), a process for determining the conversion coefficient (K, M) corresponding to the target pixel selected in step S501 is performed. Specific processing is substantially the same as that of the first embodiment. That is, the conversion coefficient map when the image blur correction amount is 0 may be corrected according to the image blur correction amount acquired in step S502. Also in this case, as in the first embodiment, it is preferable to correct the conversion coefficient map by different processing depending on the magnitude of the image blur correction amount.

  In step S605 (distance calculation processing), the distance calculation unit 15 calculates the defocus amount from the image shift amount by the equation (5) using the value of the conversion coefficient acquired in step S603.

  The conversion coefficient map (K, M) corresponding to the state in which the image blur correction amount is 0 can be determined as follows, for example. First, it is at a known distance in a predetermined distance measurement condition such as the state of the imaging optical system (focal length, aperture, focus distance, defocus amount) and the position of the distance measurement pixel on the image sensor 103 (image height). Shoot various subjects. A conversion coefficient map (K, M) is calculated from a signal acquired from each subject using an image shift amount and a predetermined calculation formula. The conversion coefficient (K, M) is adjusted so that the actual distance of each subject is close to the distance obtained from the image shift amount and the conversion coefficient (K, M). Alternatively, the conversion coefficient map (K, M) may be determined by performing a similar method by numerical simulation based on the design values of the imaging optical system 101 and the distance measuring device 102.

(Embodiment 3)
In the first and second embodiments, it is basically assumed that there is no vignetting or aberration caused by the lens barrel. However, because of the demand for downsizing of the imaging lens, so-called vignetting occurs in the normal imaging optical system so that the luminous flux is generated by the lens frame, and all the luminous flux that has passed through the aperture is not guided to the imaging device. Due to this influence, the value of the baseline length W, which is the distance between the centers of gravity of the transmittance distribution on the pupil, and the associated conversion coefficient K also become different values depending on the image height. In addition, as the defocus calculation pixel becomes a peripheral pixel, the image difference between the A image and the B image increases, and the necessity to consider this increases.

  FIG. 8 is a top view of the main part of the image sensor 103. Here, an imaging element region 1032 and an imaging element region 1033 are defined with a line segment 1031 passing through the center of the imaging element 103 and perpendicular to the direction connecting the central points of the photoelectric conversion unit 1101 and the photoelectric conversion unit 1102 as a boundary. In the imaging element region 1032 (second imaging element region), the photoelectric conversion unit 1102 that receives the light beam that has passed through the second pupil region is the first photoelectric conversion unit, and the photoelectric conversion unit 1101 is the second photoelectric conversion unit. It is said. On the other hand, in the region 1033 (first image sensor region), the photoelectric conversion unit 1101 that receives the light beam that has passed through the first pupil region is the first photoelectric conversion unit, and the photoelectric conversion unit 1102 is the second photoelectric conversion unit. And

  FIG. 9A shows a pupil transmittance distribution on the exit pupil 130 corresponding to the photoelectric conversion unit 1102 in the ranging pixel 110 arranged in the vicinity of the center of the image sensor 103, and corresponds to the second pupil region. . The blacker the transmittance is higher, and the whiter the transmittance is lower. Similarly, FIG. 9B shows a pupil transmittance distribution on the exit pupil 130 corresponding to the photoelectric conversion unit 1101 in the ranging pixel 110 arranged in the vicinity of the center of the image sensor 103.

  FIG. 9C shows the transmittance distribution in the x-axis cross section of the ranging pixel 110 near the center of the image sensor 103, where the horizontal axis is the x-axis coordinate and the vertical axis is the transmittance. A solid line is a transmittance distribution corresponding to the photoelectric conversion unit 1102 (corresponding to the second pupil region), and a dotted line is a transmittance distribution corresponding to the photoelectric conversion unit 1101 (corresponding to the first pupil region). The pupil transmittance distribution depends on the positional relationship between the photoelectric conversion unit, the microlens and the exit pupil, and the light propagation state such as aberration, diffraction, and light scattering / absorption in the optical path from the incident surface of the image sensor to the photoelectric conversion unit. Determined.

  FIG. 9D shows the pupil transmittance distribution on the exit pupil 130 corresponding to the photoelectric conversion unit 1102 in the distance measuring pixel 110 in the imaging element region 1032 and corresponds to the second pupil region. FIG. 9E shows a pupil transmittance distribution on the exit pupil 130 corresponding to the photoelectric conversion unit 1101 in the ranging pixel 110 in the imaging element region 1032 and corresponds to the first pupil region. FIG. 9F shows the transmittance distribution in the x-axis cross section of the ranging pixel 110 in the imaging element region 1032.

  In general, at the peripheral image height, so-called vignetting occurs where the light flux is generated by the lens frame due to a demand for downsizing the imaging lens, and all the light flux that has passed through the aperture is not guided to the imaging device. Since vignetting occurs from one side on the pupil, the amount of change in transmission efficiency varies depending on the shape of the original transmittance distribution. As shown in FIG. 9D, when the region 130a where vignetting occurs is a region where the original transmittance is high, the amount of decrease in transmission efficiency is large. On the other hand, as shown in FIG. 9E, when the region 130a where vignetting occurs is a region where the original transmittance is low, the amount of decrease in transmission efficiency is small. Therefore, in the ranging pixel 110 of the imaging element region 1032, as the image height increases, the transmittance distribution of the second pupil region is less than that of the first pupil region. The shape changes greatly due to defects. Therefore, the position of the center of gravity changes, and the base line length changes in the direction of shortening.

FIG. 9G represents a pupil transmittance distribution on the exit pupil 130 corresponding to the photoelectric conversion unit 1102 in the distance measuring pixel 110 in the imaging element region 1033, and corresponds to the second pupil region. FIG. 9H represents a pupil transmittance distribution on the exit pupil 130 corresponding to the photoelectric conversion unit 1101 in the distance measuring pixel 110 in the imaging element region 1033, and corresponds to the first pupil region. FIG. 9I illustrates the transmittance distribution in the x-axis cross section for the distance measuring pixels in the image sensor region 1033.

  Similar to the image sensor region 1032, in the ranging pixel 110 of the image sensor region 1033, as the image height increases, the transmittance distribution of the first pupil region is compared with the transmittance distribution of the second pupil region. In contrast to the case where there is no failure, the shape changes greatly due to the defect. Therefore, the position of the center of gravity changes, and the base line length changes in the direction of shortening.

  As described above, the center-of-gravity interval, which is the length between the center-of-gravity positions of the pupil sensitivity distribution of each photoelectric conversion unit, varies depending on the image height (pixel position), and the value of the baseline length W also varies depending on the image height. Become. Therefore, in order to perform distance measurement with high accuracy, it is necessary to use the value of the base line length corresponding to the image height in the distance calculation processing using Expression (2) or Expression (5).

  Thus, there is a correspondence between the amount of change in the received light amount distribution value according to the position of the pixel on the image sensor, the baseline length W, or a related conversion factor.

<Image correction>
If there is vignetting of the light beam due to the frame of the imaging lens or the like, the aperture functions of the optical systems that respectively form the A image and the B image are different, so the A image and the B image have different shapes, and the ranging accuracy deteriorates. . Therefore, image correction processing is performed using a known method. A distance measuring method using basic image correction processing when image blur correction is OFF is disclosed in Japanese Patent Laid-Open No. 2013-222183 (Patent Document 3).

  The image correction filter is created based on an image correction function. For example, the image correction function is a line image distribution function corresponding to the first pupil region or the second pupil region. Using the A and B images and the image correction filters La ′ and Lb ′, an A ′ image and a B ′ image whose image shapes are corrected are created. A ′ [x] is created by convolution integration of the image correction filter Lb ′ [x] with the one-dimensional signal A [x] of the A image. Similarly, for the B image, B ′ [x] is created by convolution integration of the image correction filter La ′ (x) with the one-dimensional signal B [x] of the B image. The image correction filter is composed of, for example, filter coefficients (f1,..., F9) of 9 × 1 multidimensional vectors, and is pre-calculated according to the image height.

  However, since the corresponding image height of the image correction filter also moves with the movement of the optical image by image blur correction, the map must be corrected by performing geometric transformation according to the image blur correction amount in the same manner as the conversion coefficient. FIG. 10 shows an example of a flowchart of the distance measurement calculation process for the image correction process in consideration of the image blur correction.

  The pixel selection process in step S701 and the image stabilization correction amount acquisition process in step S702 are the same as those in step S501 and step S503 in the first embodiment, and thus description thereof is omitted.

  In the conversion coefficient / image correction coefficient acquisition process in step S703, the conversion coefficient is acquired by a process substantially equivalent to step S504 in the first embodiment or step S604 in the second embodiment. In the present embodiment, the image correction filter coefficient map associated with the image height is further geometrically deformed based on the image blur correction amount, and the image correction filter coefficient corresponding to the selected pixel is acquired from the converted image correction filter coefficient map. . In order to reduce the storage capacity in the memory, a change in the filter coefficient corresponding to the image blur correction amount may be separately stored as a polynomial coefficient or the like, and the filter coefficient may be obtained by calculation using this polynomial. Thereby, the number of stored image correction filter coefficient maps corresponding to the image blur correction amount can be limited.

  It should be noted that the conversion coefficient and the filter coefficient when the image blur correction amount is 0 are stored in the memory 105 in advance for each imaging condition (F value, exit pupil distance) of the imaging optical system 101. FIG. 11 is a diagram showing how the conversion coefficient and the filter coefficient are stored in the memory 105 as a two-dimensional table corresponding to the pixel position. Here, T1 = (f1,..., Fn) represents a vector of filter coefficients, and T2 = (K, M) represents a vector of conversion coefficients. In step S703, the two-dimensional table corresponding to the pixel position is geometrically deformed according to the image blur correction amount, as in the second embodiment.

  In the image correction process in step S704, an A ′ image and a B ′ image, which are images obtained by correcting the A image and the B image, are created using the image correction filter obtained in step S703. The image correction filter is created based on an image correction function. For example, the image correction function is a line image distribution function corresponding to the first pupil region or the second pupil region. Using the A and B images and the image correction filters La ′ and Lb ′, an A ′ image and a B ′ image whose image shapes are corrected are created. A ′ [x] is created by convolution integration of the image correction filter Lb ′ [x] with the one-dimensional signal A [x] of the A image. Similarly, for the B image, B ′ [x] is created by convolution integration of the image correction filter La ′ (x) with the one-dimensional signal B [x] of the B image.

  The image shift amount calculation process in step S705 is the same as step S502 in the first embodiment except that the A image and the B image (that is, the A ′ image and the B ′ image) after the image correction process are used. The distance calculation process in step S706 is equivalent to the distance calculation process in step S505 of the first embodiment or S605 of the second embodiment, and the value of the conversion coefficient acquired in the conversion coefficient acquisition process in step S705 is used to calculate the equation (2). Alternatively, the ranging calculation process according to the equation (5) is performed.

  The conversion coefficient map (K, M) corresponding to the state in which the image blur correction amount is 0 can be determined as follows, for example. First, it is at a known distance in a predetermined distance measurement condition such as the state of the imaging optical system (focal length, aperture, focus distance, defocus amount) and the position of the distance measurement pixel on the image sensor 103 (image height). Shoot various subjects. A conversion coefficient map (K, M) is calculated from a signal acquired from each subject using an image shift amount and a predetermined calculation formula. The conversion coefficient (K, M) is adjusted so that the actual distance of each subject is close to the distance obtained from the image shift amount and the conversion coefficient (K, M). Alternatively, the conversion coefficient map (K, M) may be determined by performing a similar method by numerical simulation based on the design values of the imaging optical system 101 and the distance measuring device 102.

  An image correction filter corresponding to a state where the image blur correction amount is 0 can be created in advance by numerical simulation based on the design values of the imaging optical system 101 and the distance measuring device 102. An image correction filter is created in advance under predetermined distance measurement conditions such as the state of the imaging optical system (focal length, aperture, focus distance, defocus amount) and the position of the distance measurement pixel on the image sensor 103 (image height). That's fine.

  An example of the image correction filter is shown in FIG. In FIG. 12A, a filter Fy1301 (first filter) has a cell number of 1 in the x direction and Ay in the y direction (Ay is an integer of 2 or more), and a cell 1303 has a cell value (filter value). Have In FIG. 12B, the filter Fx1302 (second filter) has the number of cells Ax in the x direction and 1 (Ax is an integer of 2 or more) in the y direction, and the cell 1304 has a cell value (filter value). Have Each cell corresponds to one pixel of the image signal (S101 or S102).

(Embodiment 4)
In the first to third embodiments, the conversion coefficient is corrected according to the image blur correction amount. In this embodiment, not the conversion coefficient but the image shift amount is corrected according to the image blur correction amount. Since the defocus amount is obtained from the image shift amount and the conversion coefficient, the same effect can be obtained by correcting the image shift amount according to the image blur correction amount instead of correcting the conversion coefficient according to the image blur correction amount. It is done.

  The distance measuring apparatus 102 according to the present embodiment holds a correction coefficient map in which the correction coefficient for correcting the image shift amount is stored for each pixel position in the memory 105.

  The basic flow of ranging calculation processing in the present embodiment is the same as that in the first embodiment. Therefore, the distance measurement processing in the present embodiment will be described in comparison with the first embodiment (the flowchart in FIG. 5B). The pixel selection process (S501) and the image blur correction amount acquisition process (S502) are the same as those in the first embodiment. In the conversion coefficient acquisition process (S503), the value of the conversion coefficient included in the conversion coefficient map prepared in advance is used as it is. In the image shift amount calculation process (S504), first, as in the first embodiment, the image shift amount is obtained as a shift amount that minimizes the correlation value S (k). Thereafter, the correction coefficient map is adjusted according to the image blur correction amount, and the image shift amount is corrected using the correction coefficient included in the adjusted correction coefficient map. The adjustment process of the correction coefficient map is the same as the adjustment process of the conversion coefficient map in the first embodiment. In the distance calculation process (S505), the defocus amount is calculated according to the equation (2) or (3) using the conversion coefficient acquired in step S503 and the corrected image shift amount acquired in step S504. The

(Other embodiments)
The object of the present invention can also be achieved by the following embodiments. That is, a storage or recording medium storing software program codes for realizing the functions of the above-described embodiment (functions of arithmetic means, etc.) is supplied to the distance measuring device. The computer (or CPU, MPU, etc.) of the calculation unit reads the program code stored in the storage medium and executes the above function. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the program and the storage medium storing the program constitute the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. The case where the functions of the above-described embodiment are realized by the processing is also included in the present invention. Furthermore, it is assumed that the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion card or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Included in the invention. When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

  Considering application to a digital camera or the like, the present invention is a so-called imaging that performs distance detection using an imaging unit of an imaging device rather than a distance measurement device (used in a single-lens reflex camera or the like) that performs distance detection exclusively. It can be regarded as a device suitable for surface distance measurement. As described above, the arithmetic unit in the distance measuring device of the present invention can be configured using an integrated circuit in which semiconductor elements are integrated, and includes an IC, LSI, system LSI, micro processing unit (MPU), central processing unit. (CPU) or the like. When the arithmetic unit is configured by a micro processing unit, a central processing unit (CPU) or the like, the arithmetic unit can be regarded as a computer. By installing the program of the present invention in a computer of an imaging apparatus including a predetermined imaging optical system, a predetermined imaging unit, and a computer, the imaging apparatus can detect a distance with high accuracy. Can do. The program of the present invention can be distributed through the Internet in addition to the recording medium.

  Using the distance detected by the present invention, a distance distribution (distance map) corresponding to an image obtained by the imaging apparatus can be generated. Also, since the amount of blur of the subject in the image depends on the defocus amount, by performing processing based on the distance distribution on the obtained image, arbitrary blur addition processing and post-shooting refocus processing (optional Image processing such as processing for focusing on the position can be appropriately performed.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

14. Conversion coefficient calculation unit 15 Distance calculation unit 102 Distance measuring device

Claims (12)

Image blur correction means;
Based on the first image based on the light beam passing through the first pupil region of the imaging optical system and the second image based on the light beam passing through the second pupil region of the imaging optical system, the distance Distance information acquisition means for acquiring information;
Distance information correction means for correcting the distance information based on an image blur correction amount by the image blur correction means;
A distance measuring device comprising: The distance information acquisition means uses first acquisition means for acquiring an image shift amount between the first image and the second image, and the image shift amount from the image shift amount using a conversion coefficient corresponding to a pixel position. Second acquisition means for acquiring distance information;
The distance information correction unit includes a third acquisition unit that acquires the conversion coefficient based on the image blur correction amount.
The distance measuring device according to claim 1. Storage means for storing a conversion coefficient map corresponding to a position in the image plane for a predetermined image blur correction amount;
The third acquisition means acquires the conversion coefficient by geometrically converting the conversion coefficient map according to the image blur correction amount.
The distance measuring device according to claim 2. The third acquisition means translates the conversion coefficient map when the image blur correction amount is smaller than the first threshold, and converts the conversion coefficient map when the image blur correction amount is larger than the first threshold. Perform projective transformation on the coefficient map,
The distance measuring device according to claim 3. When the image shift amount is r and the distance information is ΔL, the second acquisition unit acquires the distance information by ΔL = K × r,
The conversion coefficient is a coefficient K corresponding to the position of the pixel.
The distance measuring device according to any one of claims 2 to 4. When the image shift amount is r and the distance information is ΔL, the second acquisition unit acquires the distance information by ΔL = K × r + M,
The conversion coefficient includes a coefficient K and a coefficient M according to the position of the pixel.
The distance measuring device according to any one of claims 2 to 4. Image correction means for determining an image correction filter based on an image blur correction amount by the image blur correction means and performing image correction processing of the first image and the second image using the image correction filter; Prepared,
The second acquisition unit acquires the distance information from the first image and the second image after the image correction processing by the image correction unit.
The distance measuring device according to any one of claims 2 to 6. Storage means for storing a filter coefficient map of an image correction filter corresponding to a position in the image plane with respect to a predetermined image blur correction amount;
The image correction means geometrically transforms the filter coefficient map according to the image blur correction amount, and performs the image correction process using an image correction filter obtained from the converted filter coefficient map.
The distance measuring device according to claim 7. The image blur correcting means moves a part of the lens or imaging element of the imaging optical system,
The image blur correction amount is a movement amount of the partial lens or the image sensor by the image blur correction unit.
The distance measuring device according to any one of claims 1 to 8. An image sensor;
The distance measuring device according to any one of claims 1 to 9,
An imaging apparatus comprising: Based on the first image based on the light beam passing through the first pupil region of the imaging optical system and the second image based on the light beam passing through the second pupil region of the imaging optical system, the distance A distance information acquisition step for acquiring information;
An acquisition step of acquiring an image blur correction amount by the image blur correction means;
A distance information correction step of correcting the distance information based on the image blur correction amount;
A distance measuring method including: The program for making a computer perform each step of the method of Claim 11.

Images