JP7504579B2 - Image capture device, image capture device control method and program - Google Patents

Description

The present invention relates to an imaging device, a control method for an imaging device, and a program.

There is an autofocus (hereinafter referred to as AF) function that automatically detects and adjusts focus. Patent Document 1 discloses a technology that calculates the lens drive amount for driving the lens to focus and drives the lens to readjust the focus when an angle change in the optical axis of the lens due to a change in the camera's attitude is detected. In Patent Document 1, when the composition is changed by panning the camera mounted on a tripod after focusing on a desired subject, a correction value for the distance to the subject is calculated based on the angle change of the camera to calculate the lens drive amount.

Japanese Patent Application Laid-Open No. 4-175738

However, Patent Document 1 does not take into consideration changes in the internal state of the imaging device that occur when the attitude of the imaging device changes, and if a change in the internal state causes a change in the focal length of the imaging optical system, a shift in focus on the subject will occur. Changes in the internal state of the imaging device that occur when the attitude of the imaging device changes include, for example, changes in the lens spacing of the imaging optical system, changes in the lens tilt, and changes in the mounting position of the imaging element. Compared to the horizontal movement described in Patent Document 1, the internal state of the imaging device changes significantly due to the influence of gravity when the imaging device is held facing upward or downward.

On the other hand, in a focusing operation using AF, even if a change in the internal state occurs due to a change in the attitude of the imaging device, it is possible to focus on the subject if the focusing operation is performed in the changed state. However, there is a risk that measurement values such as the distance to the subject, which are calculated based on the internal state of the imaging device, such as the position of the focus lens, will differ even for subjects located at the same distance due to changes in the internal state caused by the attitude of the imaging device.

The present invention provides an imaging device that can obtain measurement values with high accuracy according to the orientation of the imaging device.

In order to solve the above problems, the imaging device of the present invention comprises a measurement means for acquiring the amount of drive of the focusing lens when focusing , an attitude detection means for detecting the attitude of the imaging device when focusing , a correction information acquisition means for acquiring the amount of change in the drive amount of the focusing lens corresponding to the detected attitude of the imaging device as correction information for correcting errors in the drive amount of the focusing lens due to changes in the internal state of the imaging device caused by the attitude of the imaging device , and a correction means for correcting the drive amount of the focusing lens when focusing based on the correction information and calculating a subject distance, which is the distance to the focused subject, based on the corrected drive amount of the focusing lens .

The present invention provides an imaging device that can obtain measurement values with high accuracy according to the orientation of the imaging device.

FIG. 2 is a diagram illustrating the functional configuration of the imaging device according to the first embodiment. 1A and 1B are diagrams illustrating changes in the internal state of an imaging device depending on its attitude. 11 is a flowchart showing a process of calculating a distance to a subject. FIG. 11 is a diagram illustrating the functional configuration of an imaging device according to a second embodiment. 13 is a diagram illustrating a configuration of an image sensor according to a second embodiment. FIG. 1A and 1B are diagrams for explaining the principle of distance measurement using an imaging surface phase difference distance measurement method. 11A and 11B are diagrams illustrating the relationship between the amount of parallax and the amount of defocus. 11 is a flowchart showing a process for generating a depth image. 10 is a flowchart showing a process of correcting focal length and imaging magnification.

In each embodiment, the present invention is described as being applied to a digital camera that can acquire depth information indicating the distance from the imaging device to a focused subject, as an example of an imaging device. However, the present invention is not limited to this, and can be applied to any imaging device that can acquire the distance to a subject. Also, in each embodiment, the present invention is described as being applied to an imaging device in which a lens device and a main body are integrated, as an example of an imaging device, but the present invention is not limited to this, and an imaging device with interchangeable lenses in which the lens device is detachably attached to the main body may be used.

First Embodiment
1 is a diagram for explaining the functional configuration of an imaging device 100 according to the first embodiment. The imaging device 100 includes an imaging optical system 10, an imaging element 11, a control unit 12, a measurement unit 13, an image processing unit 14, a storage unit 15, an input unit 16, a display unit 17, a communication unit 18, and a sensor 19. In the following description, a direction parallel to an optical axis 103 of the imaging optical system 10 (left-right direction in FIG. 1) is referred to as a z-direction or depth direction. A direction perpendicular to the optical axis 103 and parallel to the vertical direction of the imaging element 11 is referred to as an x-direction (up-down direction in FIG. 1). A direction perpendicular to the optical axis 103 and the x-axis and parallel to the horizontal direction of the imaging element 11 is referred to as a y-direction.

The imaging optical system 10 forms an optical image of a subject on an image sensor 11. The imaging optical system 10 has a number of lenses and an aperture arranged on an optical axis 103, including a zoom lens 104 that changes the shooting magnification, a focusing lens 105 used for focus adjustment, and a shift lens 102 that corrects image blur caused by camera shake or the like. The exit pupil 101 of the imaging optical system 10 is located at a predetermined distance from the image sensor 11.

The image sensor 11 is, for example, a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor) sensor. The image sensor 11 photoelectrically converts the subject image formed on the imaging surface via the imaging optical system 10, and outputs an image signal related to the subject image.

The control unit 12 is, for example, a CPU (Central Processing Unit) or a microprocessor, and controls the operation of each block of the imaging device 100. The control unit 12 performs, for example, AF (automatic focusing) during imaging, changing the focus position, changing the F-number (aperture), capturing images, measuring the subject distance, and controlling the memory unit 15, input unit 16, display unit 17, and communication unit 18.

The measurement unit 13 calculates the distance to the focused subject. The measurement unit 13 includes a lens drive information acquisition unit 130, an attitude detection unit 131, a correction information acquisition unit 132, and a correction unit 133. The lens drive information acquisition unit 130 acquires drive information for each lens in the imaging optical system 10 via the control unit 12. For example, the lens drive information acquisition unit 130 acquires from the control unit 12 the drive amount when the focusing lens 105 is driven by a focusing operation.

The attitude detection unit 131 detects attitude information of the imaging device 100 at the time of shooting based on the output of the sensor 19 installed in the imaging device 100. The correction information acquisition unit 132 acquires distance correction information required for distance correction according to the attitude of the imaging device 100. Specifically, the correction information acquisition unit 132 acquires distance correction information corresponding to the attitude information of the imaging device 100 detected by the attitude detection unit 131 from the storage unit 15, which stores distance correction information for each attitude of the imaging device 100 in advance. The correction unit 133 calculates subject distance information based on the distance correction information acquired by the correction information acquisition unit 132.

The image processing unit 14 performs various image processing on the image signal output from the imaging element 11. For example, the image processing unit 14 performs processes such as noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction on the image signal output from the imaging element 11. The image data (captured image) output from the image processing unit 14 is accumulated in a memory (not shown) and displayed on the display unit 17. The image data output from the image processing unit 14 is also saved in the storage unit 15. The image processing unit 14 may be configured using a logic circuit, or may be configured with a central processing unit (CPU) and a memory that stores a processing program.

The storage unit 15 is a recording medium on which captured image data, intermediate data generated during the operation of each block, parameters referenced in the operation of the image processing unit 14 and the imaging device 100, etc. are recorded. The storage unit 15 may be any type of recording medium that can be read and written at high speed and has a large capacity, so long as it ensures processing performance acceptable for implementing processing; for example, a flash memory is preferable.

The input unit 16 is a user interface that detects information input or setting change operation input made to the imaging device 100, such as a dial, button, switch, or touch panel. When the input unit 16 detects an operation input from the user, it outputs a corresponding control signal to the control unit 12. The display unit 17 is a display device such as a liquid crystal display or an organic EL. The display unit 17 is used to check the composition during shooting by displaying the captured image in real time, check various settings, notify message information, etc. In addition, by using a touch panel for the display unit 17, it can have both a display function and an input function.

The communication unit 18 is a communication interface that realizes sending and receiving information to and from an external device. The communication unit 18 sends captured images, shooting information, distance information, and the like to the external device. The sensor 19 is a type of sensor that monitors the state of the imaging device 100. The sensor 19 includes, for example, an acceleration sensor, a gyro sensor, a temperature sensor, and the like. An acceleration sensor is generally used to detect the attitude of the imaging device 100.

The change in the internal state of the imaging device 100 depending on the orientation during shooting will be explained using FIG. 2. FIG. 2 is a diagram explaining the change in the internal state depending on the orientation of the imaging device. FIG. 2(A) shows the internal state of the imaging device 100 when it is held horizontally, FIG. 2(B) shows the internal state of the imaging device 100 when it is held facing downward, and FIG. 2(C) shows the internal state of the imaging device 100 when it is held facing upward. In the following, it is assumed that a gap occurs when the zoom lens 104 is attached. Also, the subject side of each lens is referred to as the first surface, and the imaging device 100 side is referred to as the second surface.

Figure 2 (A) is a cross-sectional view showing the internal state of the imaging device 100 when it is held horizontally. In the following, the state in which the imaging device 100 is held horizontally is taken as the reference posture. The distance between the second surface of the zoom lens 104 and the first surface of the focusing lens 105 when it is held horizontally is taken as L.

Figure 2 (B) is a cross-sectional view showing the internal state when the imaging device 100 is held facing downward (-z direction). In the state of Figure 2 (B), the detected attitude information is -90° when the horizontal direction is 0°. The distance between the second surface of the zoom lens 104 and the first surface of the focusing lens 105 in the downward state is Ld. Because there is a gap in the attachment of the zoom lens 104, the lens 104 moves parallel in the -z direction due to gravity. Therefore, the lens distance Ld in the downward state is longer than the lens distance L in the reference attitude.

On the other hand, FIG. 2(C) is a cross-sectional view showing the internal state when the imaging device 100 is held facing upward (+z direction). In the state of FIG. 2(C), the detected attitude information is 90° when the horizontal direction is 0°. The distance between the second surface of the zoom lens 104 and the first surface of the focusing lens 105 in the upward facing state is set to Lu. Because there is a gap in the attachment of the zoom lens 104, the zoom lens 104 moves parallel in the +z direction due to gravity. Therefore, the lens distance Ld in the upward facing state is shorter than the lens distance L in the reference attitude.

The change in lens spacing is determined by the effects of friction, stress, and gravity in the posture during shooting, and friction and stress depend on the lens barrel design of the imaging optical system 10. In addition, changes in lens spacing cause slight fluctuations in measured values such as focal length and focal length based on the state of the imaging optical system 10. The amount of change in measured values due to changes in lens spacing depends on the optical design of the imaging optical system 10 and how much each lens has moved. Note that while the above explanation uses one lens as an example, in reality position fluctuations occur in multiple lenses. The fluctuations in each lens differ depending on tolerances and manufacturing errors. In addition, the sensitivity of each lens to changes in measured values due to position fluctuations also differs depending on the optical design of the imaging optical system 10.

In this way, the change in lens spacing and the accompanying fluctuation in the measurement value depend in a complex way on the optical design and lens barrel design of the imaging optical system 10. However, as a general trend, when the change in attitude from the reference attitude (horizontal) is small, there is no significant effect, but when the change in attitude becomes large, there is a tendency for sudden fluctuations to occur. Therefore, the amount of change in the measurement value can also be approximated by a cubic function that uses the attitude of the imaging device 100 as a variable.

When AF is performed in a posture different from the reference posture of the imaging device 100, even if there is a slight change in the focal length, the focusing operation is not hindered because the focusing lens 105 is moved so that a good captured image can be obtained. However, when the distance to a focused subject is calculated using the movement state of the focusing lens 105 and the optical design information in the reference posture, an error will occur in the calculated subject distance if the state of the imaging optical system 10 is different from the reference state. Therefore, when measuring the distance to a subject using the imaging device 100 or when measuring the dimensions of a subject using the focal length of the imaging device 100, the influence of the change in internal state due to the change in posture of the imaging device 100 cannot be ignored.

The change in state of the imaging optical system 10 has been described as an internal state change accompanying a change in the posture of the imaging device 100, but this is not limited thereto, and for example, the position of the imaging element 11 may also fluctuate. In addition, if the imaging device 100 is an interchangeable lens camera, a gap may occur in the attachment portion of the imaging optical system 10 to the main body, which may be a cause of an internal state change due to posture.

The process of calculating the distance to a focused subject will be described with reference to the flowchart of Fig. 3. Fig. 3 is a flowchart showing the process of calculating the distance to a subject.
In step S301, the lens drive information acquisition unit 130 acquires lens drive information at the time of focusing. Specifically, the lens drive information acquisition unit 130 acquires the drive amount when the focusing lens 102 installed in the imaging optical system 10 is driven by the focusing operation from the control unit 12. Here, the drive amount of the focusing lens 102 acquired by the lens drive information acquisition unit 130 is, for example, the number of drive pulses when the focusing lens 102 is driven by a drive device such as a stepping motor. Note that the number of pulses may be an absolute value or a relative value, and subsequent processing can be performed according to the acquired pulse value.

In step S302, the attitude detection unit 131 detects attitude information of the imaging device 100 at the time of shooting based on the output of the acceleration sensor of the sensor 19 installed in the imaging device 100. The attitude information is the amount of angular change in the roll, pitch, and yaw directions from the reference attitude of the imaging device 100. Note that, although the reference attitude of the imaging device 100 in this embodiment is the horizontal direction, this is not limited to this.

The internal state of the imaging device 100 changes significantly under the influence of gravity when the imaging device is held facing upward or downward compared to horizontal movement. For this reason, it is particularly useful to obtain the degree of tilt in the up-down direction (pitch direction) at which the image was captured. In this embodiment, we will explain the attitude information of the imaging device 100 by taking the pitch angle in the up-down direction relative to the horizontal direction as an example.

In step S303, the correction information acquisition unit 132 acquires distance correction information corresponding to the attitude information of the image capture device 100. Specifically, the correction information acquisition unit 132 acquires distance correction information corresponding to the attitude information (angle in the pitch direction) of the image capture device 100 detected in step S302 from the distance correction information stored in advance in the storage unit 15. The distance correction information is information necessary for distance correction, and in this embodiment, it is the amount of change in the drive amount of the focusing lens 102 corresponding to the change in attitude. For example, when the focusing lens 102 is driven using a stepping motor, the amount of change in drive pulse corresponding to the change in attitude becomes the distance correction information.

The storage unit 15 stores in advance distance correction information for each posture information of the imaging device 100, that is, the amount of change in the drive pulse due to the posture change of the imaging device 100, in the form of a table or a function. The amount of change in the drive pulse as distance correction information stored in advance in the storage unit 15 is a value calculated as the difference between the drive pulse value in the reference posture and the drive pulse value in a posture different from the reference posture. More specifically, the drive pulse value (control pulse) of the focusing lens 102 during the focusing operation on a subject (e.g., a chart) at a specific distance in the reference posture of the imaging device 100 is acquired. Then, the posture of the imaging device 100 is changed in the vertical direction while the subject distance is kept constant, and the drive pulse value (control pulse) of the focusing lens 102 when the focusing operation is performed is acquired. The difference (relative pulse number) between the drive pulse value in the acquired reference posture and the drive pulse value corresponding to the posture change of the imaging device 100 is the amount of change in the drive pulse. The calculated drive pulse change amount may be saved as a table for use, or only the coefficients of a function that fits the relationship between the calculated drive pulse change amount and the posture change amount of the imaging device 100 may be saved for use. When only the function coefficients are saved, the amount of data to be saved can be reduced compared to when a table is saved.

In addition to obtaining the correction information experimentally as described above, the correction information may be calculated by simulation taking into account the relationship between the lens mounting tolerance and the pulses of the stepping motor and the lens movement amount. If the amount of pulse change associated with the change in posture changes depending on the focusing distance, the distance between the subject and the imaging device 100 is changed and the amount of pulse change is obtained in the same manner. If the number of pulses of the focusing lens 102 is large, instead of obtaining the amount of pulse change at all focusing distances, the amount of pulse change may be obtained at multiple distances and the amount of pulse change at other distances may be calculated by interpolation. In addition, when using an imaging optical system with a variable focal length, it is preferable to obtain the amount of pulse change at each of the focal lengths that can be set. If it is difficult to obtain the amount of pulse change at all focal lengths, the amount of pulse change may be obtained at a predetermined focal length interval and the amount of pulse change at focal lengths within the predetermined focal length interval may be calculated by interpolation.

If the imaging optical system 10 attached to the imaging device 100 is replaceable, it is preferable to store correction information for multiple different imaging optical systems in the storage unit 15. Also, if the imaging optical system 10 has a storage unit, the correction information may be stored in the storage unit of the imaging optical system 10, and the correction information acquisition unit 13 may be configured to read the correction information from the imaging optical system 10. In this configuration, even if the imaging optical system 10 does not have distance correction information stored in the storage unit 15, it is possible to acquire the correction information of the imaging optical system 10. Note that, in this embodiment, an example in which distance correction information is stored in the storage unit 15 has been described, but if the imaging optical system 10 of the imaging device 100 is fixed, the correction information may be stored in advance in a non-volatile memory (not shown) in the measurement unit 13.

In step S304, the correction unit 133 calculates the corrected subject distance. Specifically, the correction unit 133 calculates the number of drive pulses when shooting in the reference posture from the number of drive pulses when focusing obtained in step S301 and the amount of change in drive pulse corresponding to the posture change of the imaging device 100 obtained in step S303. Then, the correction unit 133 calculates the subject distance corresponding to the number of drive pulses when shooting in the reference posture as the corrected subject distance.

The calculation of the subject distance based on the number of drive pulses is performed by using a table or function of the relationship between the number of drive pulses and the subject distance that has been stored in advance. The subject distance information table showing the relationship between the number of drive pulses and the subject distance is created by calculating the relationship between the number of drive pulses (drive position) of the focusing lens 102 in the imaging optical system 10 when focusing in the reference posture and the focusing distance using the design information of the imaging optical system 10. From this subject distance information table or function and the corrected number of drive pulses (number of drive pulses in the reference posture), it is possible to output a distance in which the distance calculation error due to changes in the internal state caused by the posture of the imaging device 100 has been corrected.

As described above, the imaging device of this embodiment can acquire correction information (change in driving pulse) according to the camera attitude when focusing, and calculate measurement values such as distance information based on the correction information. Therefore, it is possible to correct distance measurement errors to the subject caused by changes in the internal state of the imaging device depending on the attitude of the imaging device, making it possible to perform highly accurate distance measurement independent of the attitude.

Second Embodiment
In the first embodiment, a mode in which one distance at a subject position focused by an AF operation or the like is output is described. In the present embodiment, an example in which the present invention is applied to a mode in which a depth image of an area similar to that of a captured image is output by performing distance measurement on an imaging surface is described.

Figure 4 shows a block diagram showing the functional configuration of the imaging device 200 according to this embodiment. Figure 4 is a diagram for explaining the functional configuration of the imaging device in the second embodiment. The same components as in the first embodiment are given the same reference numerals and their explanation will be omitted. Below, the imaging element 20 and image processing unit 24, which are differences from the first embodiment, will be explained.

The image sensor 20 is an image sensor that has a distance measurement function using an image plane phase difference distance measurement method. Therefore, the image sensor 20 can generate and output distance information indicating the distance from the image sensor to the subject (subject distance) in addition to the captured image.

The detailed configuration of the imaging element 20 will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining the configuration of the imaging element 20. FIG. 5(A) is a diagram for explaining the imaging element 20 and the pixel 210. FIG. 5(B) is a diagram for explaining the pixel 210. The imaging element 20 has a plurality of pixels 210 arranged two-dimensionally in the row direction and the column direction. Each pixel 210 is arranged in a 2-row x 2-column array in which a plurality of pixels 210 are connected to each other, to which red (R), green (G), and blue (B) color filters are applied. Therefore, an image signal indicating any one of the color information of R, G, or B is output from the pixel 210. Note that in this embodiment, an example in which the color filters are arranged in a Bayer array will be described as an example, but the implementation of the present invention is not limited to this.

Each pixel 210 of the image sensor 20 of this embodiment, which has a distance measurement function using the image plane phase difference distance measurement method, has multiple photoelectric conversion units. The multiple photoelectric conversion units are arranged side by side in the I-I' cross section of FIG. 5(A) in the horizontal direction of the image sensor 20. As shown in FIG. 5(B), each pixel 210 has a light guide layer 213 and a light receiving layer 214. The light guide layer 213 has one microlens 211 and multiple color filters 212. The light receiving layer 214 has a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216. In other words, the pixel 210 is configured to have multiple photoelectric conversion units for one microlens.

The microlens 211 is configured to efficiently guide the light beam incident on the pixel to the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216. The color filter 212 passes light of a specific wavelength band, and passes only light of one of the wavelength bands of red (R), green (G), or blue (B), and guides it to the downstream first photoelectric conversion unit 215 and second photoelectric conversion unit 216.

The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 convert the received light into an analog image signal. The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 generate a pair of image signals from light beams passing through different pupil regions of the imaging optical system 10, and the pair of image signals are used for distance measurement. That is, a first image signal composed of an image signal output from the first photoelectric conversion unit 215 of each pixel 210 of the imaging element 20, and a second image signal composed of an image signal output from the second photoelectric conversion unit 216 are used for distance measurement.

The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 each partially receive the light beam that enters the pixel through the microlens 211. Therefore, the two types of image signals finally obtained are pupil division images related to the light beams that have passed through different regions of the exit pupil 101 of the imaging optical system 10. Here, the image signal obtained by combining the image signals photoelectrically converted by the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 in each pixel 210 is equivalent to the image signal for viewing output from the single photoelectric conversion unit in an embodiment in which only one photoelectric conversion unit is provided in the pixel.

By having such a structure, the image sensor 20 of this embodiment is capable of outputting an image signal for viewing and an image signal for distance measurement (two types of pupil-split images). Note that in this embodiment, an example is described in which all pixels 210 of the image sensor 20 are provided with two photoelectric conversion units and are configured to be able to output high-density depth information, but this is not limited to this. For example, the number of photoelectric conversion units provided in each pixel 210 may be two or more. Also, a configuration in which some, but not all, pixels 210 have multiple photoelectric conversion units may be used.

Here, the principle of deriving the subject distance based on the pupil division image group output from the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 will be described with reference to Figs. 6 and 7. Fig. 6(A) is a schematic diagram showing the light flux received by the first photoelectric conversion unit 215 of the pixel 210 of the image sensor 20. Fig. 6(B) is a schematic diagram showing the light flux received by the second photoelectric conversion unit 216.

The microlens 211 is arranged so that the exit pupil 101 and the light receiving layer 214 are optically conjugate. The light beam that passes through the exit pupil 101 of the imaging optical system 10 is collected by the microlens 211 and guided to the first photoelectric conversion unit 215 or the second photoelectric conversion unit 216. At this time, the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 mainly receive the light beam that passes through different pupil regions. The first photoelectric conversion unit 215 receives the light beam that passes through the first pupil region 510, and the second photoelectric conversion unit 216 receives the light beam that passes through the second pupil region 520.

The multiple first photoelectric conversion units 215 included in the image sensor 20 mainly receive the light beam that has passed through the first pupil region 510 and output a first image signal. At the same time, the multiple second photoelectric conversion units 216 included in the image sensor 20 mainly receive the light beam that has passed through the second pupil region 520 and output a second image signal. From the first image signal, it is possible to obtain the intensity distribution of the image formed on the image sensor 20 by the light beam that has passed through the first pupil region 510. Similarly, from the second image signal, it is possible to obtain the intensity distribution of the image formed on the image sensor 20 by the light beam that has passed through the second pupil region 520.

The relative positional shift amount (so-called parallax amount) between the first image signal and the second image signal is a value according to the defocus amount. The relationship between the parallax amount and the defocus amount will be explained using FIG. 7. FIG. 7 is a diagram explaining the relationship between the parallax amount and the defocus amount. In FIG. 7, a first light beam 511 indicates a light beam passing through the first pupil region 510, and a second light beam 521 indicates a light beam passing through the second pupil region 520.

Figure 7 (A) shows the state when in focus. When in focus, the first light beam 511 and the second light beam 521 converge on the image sensor 20. At this time, the parallax amount between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is 0.

7B shows a state where the image is defocused in the negative direction of the z axis. At this time, the parallax amount between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not 0, but has a negative value. FIG. 7C shows a state where the image is defocused in the positive direction of the z axis. At this time, the parallax amount between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not 0, but has a positive value. From a comparison of FIG. 7B and FIG. 7C, it can be seen that the direction of the positional deviation is reversed depending on whether the defocus amount is positive or negative. It can also be seen that the positional deviation occurs according to the imaging relationship (geometric relationship) of the imaging optical system 10 depending on the defocus amount. The parallax amount, which is the positional deviation between the first image signal and the second image signal, can be detected by a region-based matching method described later.

The image processing unit 24 includes an image generating unit 240 and a depth generating unit 241. The image generating unit 240 synthesizes a pair of image signals obtained from the imaging element 20 to generate an image for viewing. The depth generating unit 241 generates a depth image from the pair of image signals obtained from the imaging element 20.

Here, the process for generating a depth image will be described with reference to the flowchart of FIG. 8. FIG. 8 is a flowchart showing the process for generating a depth image. In step S2401, the depth generation unit 241 performs a light amount correction process on the first image signal and the second image signal. At the peripheral angle of view of the imaging optical system 10, the shape of the first pupil region 510 and the second pupil region 520 differs due to vignetting, causing a loss of light amount balance between the first image signal and the second image signal. Therefore, the depth generation unit 241 performs light amount correction on the first image signal and the second image signal using, for example, a light amount correction value stored in advance in a memory (not shown), and aligns the light amount of the first image signal and the second image signal.

In step S2402, the depth generation unit 241 performs a process to reduce noise generated in the first image signal and the second image signal. The noise is, for example, noise generated during photoelectric conversion in the image sensor 20. Specifically, the depth generation unit 241 achieves noise reduction by applying a filter process to the first image signal and the second image signal. In general, the higher the spatial frequency is, the lower the S/N ratio is, and the more noise components there are relatively. Therefore, the depth generation unit 241 applies a low-pass filter, which reduces the pass rate as the spatial frequency is higher, to the first image signal and the second image signal. Note that the light quantity correction in step S2401 may not produce a favorable result due to manufacturing errors in the imaging optical system 10, so it is more preferable for the depth generation unit 241 to apply a band-pass filter in step S2402 that blocks direct current components and has a low pass rate for high-frequency components.

In step S2403, the depth generation unit 241 generates a depth image. The depth generation unit 241 calculates the amount of parallax between the images of the first image signal and the second image signal, converts it into a defocus amount, and generates two-dimensional information in which the defocus amount is a pixel value as a depth image.

Specifically, in order to calculate the amount of parallax, the depth generation unit 241 first sets a point of interest corresponding to the representative pixel information and a matching area centered on the point of interest in the first image signal. The matching area is, for example, a rectangular area such as a square area with one side having a predetermined length centered on the point of interest. Next, the depth generation unit 241 sets a reference point in the second image signal and sets a reference area centered on the reference point. The reference area has the same size and shape as the above-mentioned matching area. While sequentially moving the reference point, the depth generation unit 241 derives the degree of correlation between the image included in the matching area of the first image signal and the image included in the reference area of the second image signal, and identifies the reference point with the highest degree of correlation as the corresponding point corresponding to the point of interest in the second image signal. The relative positional shift amount between the corresponding point identified by matching in this way and the point of interest becomes the amount of parallax at the point of interest. The depth generation unit 241 calculates the amount of parallax while sequentially changing the point of interest according to the representative pixel information, thereby deriving the amount of parallax at multiple pixel positions determined by the representative pixel information. For simplicity, in this embodiment, in order to obtain depth information at the same resolution as the image for viewing, it is assumed that the pixel positions (pixel groups included in the representative pixel information) at which the amount of parallax is calculated are set to be the same number as the image for viewing. Methods such as NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), and SAD (Sum of Absolute Difference) may be used as a method for deriving the degree of correlation.

The calculated parallax amount can be converted into a defocus amount, which is the distance from the image sensor 20 to the focal point of the image pickup optical system 10, by using a predetermined conversion coefficient. Here, assuming that the predetermined conversion coefficient K and the defocus amount are ΔL, the parallax amount can be converted into the defocus amount by the following formula 1.
ΔL=K×d (Equation 1)
The depth generation unit 241 generates two-dimensional information with the calculated defocus amount as a pixel value, and stores the information in a memory (not shown) as a depth image.

The correction process in this embodiment is performed in a similar manner to the correction process flowchart in the first embodiment (Figure 3), but the information used for distance correction is different. In the first embodiment, an example was described in which correction was performed using the number of drive pulses for the focusing lens 105, but in this embodiment, correction is performed based on the shift in focus position (defocus amount) on the image side.

The distance correction process in this embodiment is similar to the focal length correction process in the first embodiment shown in FIG. 3, but differs in that the change in the drive amount of the focusing lens 105 according to the attitude of the imaging device 200 is stored as the defocus amount. In step S301, the lens drive information acquisition unit 130 acquires from the control unit 12 the drive amount when the focusing lens 105 installed in the imaging optical system 10 is driven by the focusing operation. The lens drive information acquisition unit 130 calculates the image plane distance from the principal point of the imaging optical system 10 to the image plane based on the acquired drive amount.

In step S302, the attitude detection unit 131 detects attitude information of the imaging device 200 at the time of shooting, i.e., at the time of focusing, from the output of the acceleration sensor, gyro sensor, etc. of the sensor 19. As attitude information, the attitude detection unit 131 obtains the roll, pitch, and yaw tilt angles from the reference attitude of the imaging device 200, particularly the tilt angle in the vertical direction at which shooting was performed.

In step S303, the correction information acquisition unit 132 acquires distance correction information required for distance correction, which has been acquired in advance from the storage unit 15. The distance correction information in this embodiment is a value obtained by converting the amount of change in drive amount due to a change in attitude when the focusing lens 105 is driven into a defocus amount. The storage unit 15 may store the change in defocus amount due to attitude acquired in advance as a table as distance correction information, or may approximate the amount of defocus amount due to an attitude angle and hold its coefficient. When a function is used as distance correction information, it is preferable to approximate the amount of defocus amount due to a trigonometric function or a cubic function for the pitch angle, since the amount of defocus generally tends to change rapidly when the attitude is facing upward or downward.

In step S304, the correction unit 133 calculates the corrected image-side subject distance (image-side distance + subject defocus amount) using the image-side distance at the time of focusing acquired in step S301, the defocus amount generated by the depth generation unit 241, and the defocus amount acquired in step S303 due to the attitude of the imaging device 200. Then, the correction unit 133 converts the calculated image-side subject distance into the object-side subject distance by using the lens formula shown in the following Equation 2 in geometric optics.
1/A+1/B=1/F (Equation 2)
Here, A represents the distance from the subject to the principal point of the imaging optical system 10 (subject distance), B represents the subject distance from the principal point of the imaging optical system 10 to the image side, and F represents the focal length of the imaging optical system 10. In the lens formula, the value of B is the subject distance on the image side calculated in step S304. The subject distance A on the object side can be calculated based on the setting of the subject distance B on the image side and the focal length F during imaging.
If the defocus amount generated by the depth generation unit 241 is not used, the corrected image plane distance is calculated using the image plane distance at the time of focus obtained in step S301 and the defocus amount associated with the attitude of the imaging device 200 obtained in step S303.

In this embodiment, even in depth image generation using the imaging plane phase difference ranging method, distance measurement errors due to the attitude of the imaging device can be corrected, making it possible to generate subject distance and depth images independent of changes in the attitude of the imaging device. The correction unit 133 then converts the calculated image plane distance into the subject distance on the object side by using the lens formula shown in Equation 2 in geometric optics.

Third Embodiment
In the first and second embodiments, examples of correcting the distance to the subject have been described. However, in addition to the distance to the subject, deviations occur due to changes in the attitude of the imaging device. For example, due to changes in the attitude of the imaging device, a slight change in the focal length of the imaging optical system 10 occurs due to changes in the lens spacing caused by gaps that occur when lenses are attached in the imaging optical system 10. Although a slight change in the focal length does not interfere with normal photography, it also causes errors in the calculation of the imaging magnification. In addition, in generating a depth image with a configuration such as the imaging element 20 in the second embodiment, by making the focal length in Equation 2 a more accurate value, it becomes possible to calculate the subject distance more accurately. Therefore, in this embodiment, a process of correcting the focal length and the imaging magnification will be described.

The focal length correction process will be described with reference to the flowchart in FIG. 9(A). FIG. 9(A) is a flowchart showing the focal length correction process. Processes similar to those in the first embodiment are given the same reference numerals and their description will be omitted. Note that the configuration of the imaging device in this embodiment may be the same as that in the first embodiment or the second embodiment. The imaging device 100 in the first embodiment will be used as an example below.

In step S901, the correction information acquisition unit 132 acquires focal length correction information from the storage unit 15. The focal length correction information is information about the change in focal length associated with a change in attitude, and is a value that represents the focal length in a reference attitude (e.g., horizontal) of the imaging device 100 and the change in focal length when the attitude of the imaging device 100 is changed. The focal length correction information is calculated, for example, from a minute change in the angle of view due to the attitude from a chart placed at a specific distance, and is stored in advance in the storage unit 15.

In step S902, the correction unit 133 calculates a reference focal length from the lens drive information acquired in step S301, and corrects the calculated focal length using the focal length correction information acquired in step S901. The corrected focal length is linked to the captured image as auxiliary information of the image and stored in the storage unit 15. In addition, by using the corrected focal length in the calculation of Equation 2 in the second embodiment, the subject distance can be corrected with higher accuracy.

Next, the magnification correction process will be described with reference to the flowchart in FIG. 9(B). FIG. 9(A) is a flowchart showing the magnification correction process. In the magnification correction process, corrections are made to the subject distance and the focal length to calculate a corrected magnification. Processes similar to those in the first embodiment (FIG. 3) and the focal length correction process (FIG. 9(A)) are given the same reference numerals and will not be described again.

In step S903, the correction unit 133 or a magnification calculation unit (not shown) in the measurement unit 13 calculates the shooting magnification M from the corrected subject distance A' calculated in step S304 and the corrected focal length F' calculated in step S902. The shooting magnification M is calculated by M = A'/F'. Based on the calculated shooting magnification, it becomes possible to perform subject dimension measurement in a dimension measurement unit (not shown) in the image processing unit 14. In the dimension measurement, the size of the subject on the image plane is calculated from the pixel size of the image sensor 10, and by multiplying it by the shooting magnification, it becomes possible to calculate the subject dimension on the object side.

In this embodiment, it is possible to correct the focal length and imaging magnification of the imaging optical system in accordance with changes in the attitude of the imaging device, and it is possible to calculate a more accurate focal length and imaging magnification corresponding to the change in attitude. Furthermore, the correction of the focal length and imaging magnification is effective when performing dimensional measurements from the captured image, and it is possible to measure dimensions from the captured image independently of the attitude of the imaging device.

Fourth Embodiment
In the first to third embodiments, correction of one-dimensional information such as the subject distance and the focal length has been described, but in this embodiment, correction of two-dimensional information accompanying a change in the posture of the imaging device 100 will be described. When a tilt occurs in the lens in the imaging optical system 10 due to a change in the posture of the imaging device 100, one-sided blur may occur in the captured image according to the distribution of the amount of blur on the image plane. Therefore, as correction information, the amount of blur or the amount of defocus at each position on the image plane according to the change in the posture of the imaging device 100 is measured in advance and stored in the storage unit 15. By using the correction information according to the change in the posture of the imaging device 100, it is possible to obtain information on how much blur occurs in which area of the image simply by detecting the posture of the imaging device 100 at the time of shooting. Then, by applying a sharpness filter corresponding to the blur to each area in the captured image, it is possible to reduce the effect of one-sided blur in the captured image. In addition, a correction filter corresponding to the change in the posture of the imaging device 100 may be stored in advance as correction information.

According to this embodiment, it is possible to correct the effect of one-sided blur that occurs in a captured image when the image capture device changes its posture, and it is possible to obtain a captured image in which the effect of one-sided blur caused by the posture of the image capture device is reduced, regardless of the posture of the image capture device.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

REFERENCE SIGNS LIST 100 Imaging device 10 Imaging optical system 11 Imaging element 12 Control unit 13 Measurement unit 130 Lens drive information acquisition unit 131 Attitude detection unit 132 Correction information acquisition unit 133 Correction unit 14 Image processing unit 15 Storage unit 19 Sensor

Claims (9)

A measuring means for acquiring a driving amount of the focusing lens when focusing;
an attitude detection means for detecting an attitude of the imaging device when focusing;
a correction information acquisition means for acquiring a change in a driving amount of a focusing lens according to the detected attitude of the imaging device as correction information for correcting an error in a driving amount of a focusing lens due to a change in an internal state of the imaging device caused by the attitude of the imaging device;
and a correction means for correcting a driving amount of the focusing lens at the time of focusing based on the correction information, and calculating a subject distance, which is a distance to a focused subject, based on the corrected driving amount of the focusing lens. a measuring means for acquiring an image plane distance from a principal point of the imaging optical system to an image plane based on a driving amount of the focusing lens during focusing;
an attitude detection means for detecting an attitude of the imaging device when focusing;
a correction information acquisition means for acquiring a change in defocus amount obtained by converting a change in a driving amount of a focusing lens corresponding to the detected attitude of the imaging device, as correction information for correcting an error in an image plane distance due to a change in an internal state of the imaging device caused by the attitude of the imaging device;
a correction unit that corrects the image plane distance at the time of focusing based on the correction information, and calculates a subject distance, which is the distance to a focused subject, based on the corrected image plane distance. A measuring means for acquiring a driving amount of the focusing lens when focusing;
an attitude detection means for detecting an attitude of the imaging device when focusing;
a correction information acquisition means for acquiring a change in focal length of the imaging optical system corresponding to the detected attitude of the imaging device as correction information for correcting an error in focal length of the imaging optical system due to a change in an internal state of the imaging device caused by an attitude of the imaging device;
a correction unit that calculates a focal length of the imaging optical system when in focus based on the driving amount of the focusing lens when in focus, and corrects the calculated focal length based on the correction information. A measuring means for acquiring a driving amount of the focusing lens when focusing;
an attitude detection means for detecting an attitude of the imaging device when focusing;
a first correction information acquisition means for acquiring a change in a driving amount of a focusing lens according to the detected attitude of the imaging device as correction information for correcting an error in a driving amount of a focusing lens due to a change in an internal state of the imaging device caused by an attitude of the imaging device;
a first correction means for correcting the driving amount of the focusing lens at the time of focusing based on the acquired change amount of the driving amount of the focusing lens and the correction information, and calculating a corrected subject distance which is a distance to a focused subject based on the corrected driving amount of the focusing lens;
a second correction information acquisition means for acquiring a change in focal length of the imaging optical system corresponding to the detected attitude of the imaging device as correction information for correcting an error in focal length of the imaging optical system due to a change in an internal state of the imaging device caused by an attitude of the imaging device;
a second correction means for calculating a focal length of the imaging optical system when in focus based on a driving amount of the focusing lens when in focus, and correcting the calculated focal length based on the acquired amount of change in the focal length to calculate a corrected focal length;
and a magnification calculation unit that calculates a photographing magnification based on the corrected object distance and the corrected focal length. 5. The imaging apparatus according to claim 4 , further comprising a dimension measuring means for calculating a subject dimension from the photographing magnification calculated by the magnification calculating means and a dimension of the subject image on an image plane. 6. The imaging device according to claim 1, wherein the correction information is stored in a storage device as a table corresponding to the orientation of the imaging device or as coefficients of a function having the orientation of the imaging device as a variable. 7. The imaging apparatus according to claim 1, wherein the attitude of the imaging apparatus detected by the attitude detection means is an angle in either a roll direction or a pitch direction with respect to a reference attitude of the imaging apparatus. A control method for an imaging device, comprising:
acquiring a driving amount of the focusing lens when focusing;
detecting the attitude of the imaging device when focusing;
acquiring a change in a driving amount of a focusing lens according to the detected attitude of the imaging device as correction information for correcting an error in a driving amount of a focusing lens due to a change in an internal state of the imaging device caused by the attitude of the imaging device;
and correcting the amount of drive of the focusing lens at the time of focusing based on the correction information, and calculating an object distance, which is the distance to the focused object, based on the corrected amount of drive of the focusing lens. A program for causing a computer to function as each of the means of the imaging apparatus according to any one of claims 1 to 7 .

Images