JP2021013179A - Imaging device and distance measuring device - Google Patents

Abstract

To provide an imaging device with high distance resolution and a wide distance measurable range.SOLUTION: An imaging device comprises: a light source unit which emits first and second modulation light to an object; an imaging element including first and second pixels which receive the first and second modulation light reflected by the object to generate a signal; a first phase difference measuring unit which measures a first phase difference between a phase of the first modulation light emitted from the light source unit and a phase of the first modulation light received by the first pixel by an output signal of the first pixel; a second phase difference measuring unit which measures a second phase difference between a phase of the second modulation light emitted from the light source unit and a phase of the second modulation light received by the second pixel by an output signal of the second pixel; a distance calculating unit which calculates the distance to the object according to the first and second phase differences; and an image data generating unit which generates image data of the object on the basis of the output signal of the first pixel and the output signal of the second pixel.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device and a ranging device.

A light flight time measurement method (TOF; Time of Flight) is known in which the distance to an object is measured by receiving the reflected light of the light emitted to the object (for example, Patent Document 1). This technique has a problem that the range that can be measured becomes shorter as the distance resolution is increased.

Japanese Unexamined Patent Publication No. 2008-89346

The imaging device according to claim 1 generates a signal by receiving a light source unit that emits first and second modulated light to an object and the first and second modulated light reflected by the object. The phase of the first modulated light emitted from the light source unit by the image pickup element having the first and second pixels and the output signal of the first pixel, and the first received by the first pixel. The first phase difference measuring unit that measures the first phase difference with the phase of the modulated light of 1, and the phase of the second modulated light emitted from the light source unit by the output signal of the second pixel. , A second phase difference measuring unit that measures the second phase difference with the phase of the second modulated light received by the second pixel, and the object by the first and second phase differences. A distance calculation unit for calculating the distance of the object, and an image data generation unit for generating image data of the object based on the output signal of the first pixel and the output signal of the second pixel are provided.
The distance measuring device according to claim 18 receives a light source unit that emits first and second modulated light to an object and the first and second modulated light reflected by the object and receives a signal. The phase of the first modulated light emitted from the light source unit and the light received by the first pixel are received by the image pickup element having the first and second pixels to be generated and the output signal of the first pixel. The phase of the first modulated light that measures the first phase difference with the phase of the first modulated light and the phase of the second modulated light emitted from the light source unit by the output signal of the second pixel. And a second phase difference measuring unit that measures the second phase difference with the phase of the second modulated light received by the second pixel, and the object by the first and second phase differences. It is provided with a distance calculation unit for calculating the distance to.

It is sectional drawing which shows typically the structure of the image pickup apparatus which concerns on 1st Embodiment. It is a schematic diagram which shows the structure of an image sensor. It is a circuit diagram which shows the structure of an image sensor. It is a figure which shows the modulated light schematically. It is a figure which shows typically the relationship between the modulated light and the reflected light. It is a figure which shows typically the relationship between the modulated light and the reflected light. It is a timing chart which shows the shooting operation. It is explanatory drawing explaining the operation of the 1st detection part, the 2nd detection part, the distance measuring part, and the image creation part. It is explanatory drawing which shows that the control signal can be approximated by a sine wave. It is explanatory drawing of the calculated phase θ _a and the true phase Θ. It is sectional drawing which shows typically the structure of the image pickup apparatus which concerns on 2nd Embodiment. It is a schematic diagram which shows the structure of a pixel. It is a circuit diagram which shows the structure of an image sensor. It is a timing chart which shows the shooting operation. It is a figure which shows the modulated light schematically. It is a schematic diagram which shows the modification of the pixel layout. It is a circuit diagram which shows the structure of the image pickup device which concerns on modification 4. It is explanatory drawing of the modification 4.

(First Embodiment)
FIG. 1 is a cross-sectional view schematically showing the configuration of the image pickup apparatus according to the first embodiment. The image pickup device 1 is a device having a function of photographing the subject 2 and creating depth map data of the subject 2 together with the image data of the subject 2. In the present embodiment, the depth map data is map data in which distance values representing the distance between the portion and the image pickup apparatus 1 (that is, the depth of the portion) are arranged in a two-dimensional manner for each portion of the subject 2. .. When the image pickup device 1 is used, not only the image data of the subject 2 but also the depth map data can be obtained at the same time in one shooting. In other words, the image data and the depth map data are created from the same photoelectric conversion signal.

The image pickup device 1 includes a light source unit 10, an optical system 11, an image pickup element 12, a control unit 13, a ROM 14, a display unit 15, and a storage medium 16. The light source unit 10 emits modulated light 3, which will be described later, to the subject 2. The modulated light 3 has a transmission first modulated light 3a and a transmitting second modulated light 3b. The modulated light 3 is reflected by the surface of the subject 2, returns to the image pickup apparatus 1 as reflected light 4, and is incident on the image pickup surface of the image pickup device 12 via the optical system 11. The reflected light 4 includes a reception first modulation light 4a which is a reflected light of the transmission first modulation light 3a and a reception second modulation light 4b which is a reflected light of the transmission second modulation light 3b. The optical system 11 is, for example, an imaging optical system composed of a plurality of lenses. In FIG. 1, the optical system 11 is simplified and shown as a single lens.

The control unit 13 is composed of a microcomputer (not shown) and peripheral circuits. The control unit 13 controls each unit of the image pickup apparatus 1 by reading and executing a control program stored in advance in the ROM 14 which is a non-volatile storage medium. The display unit 15 is a display device such as a liquid crystal monitor. The storage medium 16 is a rewritable storage medium such as a memory card.

Although details will be described later, a photoelectric conversion signal is output from the image sensor 12. The control unit 13 creates image data and depth map data based on the photoelectric conversion signal. The control unit 13 stores the created image data and depth map data in the storage medium 16, and displays an image based on these data on the display unit 15.

The control unit 13 includes a first detection unit 13a, a second detection unit 13b, a distance measurement unit 13c, an image creation unit 13d, and a light emission control unit 13e, depending on the software form. Each of these units is realized by software when the control unit 13 executes a predetermined control program stored in the ROM 14. The operation of each of these parts will be described in detail later.

FIG. 2A is a plan view schematically showing the configuration of the image sensor 12, and FIG. 2B is an enlarged schematic view of a part of the region 20a of the image pickup surface 20. As shown in FIG. 2B, a plurality of pixels 30 are arranged two-dimensionally on the imaging surface 20.

There are two types of pixels 30, a first pixel 30a and a second pixel 30b. In the present embodiment, as shown in FIG. 2B, the first pixel 30a and the second pixel 30b are arranged alternately. That is, the second pixel 30b exists on the top, bottom, left, and right of the first pixel 30a, and the first pixel 30a exists on the top, bottom, left, and right of the second pixel 30b. Therefore, substantially the same number of first pixels 30a and second pixels 30b are uniformly (uniformly) arranged on the imaging surface 20.

As shown in FIG. 2A, the image pickup device 12 includes a horizontal scanning circuit 21, a vertical scanning circuit 22, a first transfer control circuit 23a, and a second transfer control circuit 23b. The vertical scanning circuit 22 controls a plurality of pixels 30 included in the image pickup device 12 line by line. The horizontal scanning circuit 21 sequentially reads output signals from the pixels 30 for one line. The first transfer control circuit 23a transfers the photoelectric conversion signal to the floating diffusion at the same timing for the plurality of first pixels 30a. The second transfer control circuit 23b transfers the photoelectric conversion signal to the floating diffusion at the same timing for the plurality of second pixels 30b.

Although the details will be described later, a control pulse signal output from the first transfer control circuit 23a toward the first pixel 30a and a control pulse output from the second transfer control circuit 23b toward the second pixel 30b. The signals are different pulse signals.

FIG. 3 is a circuit diagram showing the configuration of the image pickup device 12. Note that FIG. 3 shows only a circuit diagram of one pixel 30. Both the first pixel 30a and the second pixel 30b have the configuration shown in FIG. Hereinafter, the pixel 30 will be described, but this description is a description of both the first pixel 30a and the second pixel 30b.

The pixel 30 includes a microlens (not shown), a photoelectric conversion unit 32 covered with the microlens, a first reading unit 331, a second reading unit 332, a third reading unit 333, and a fourth reading unit 334. To be equipped with. The microlens concentrates the incident light (luminous flux from the optical system 11) on the photoelectric conversion unit 32. The photoelectric conversion unit 32 photoelectrically converts the incident light to generate an electric charge according to the amount of light. That is, the photoelectric conversion unit 32 photoelectrically converts the incident light and outputs a photoelectric conversion signal. The first reading unit 331, the second reading unit 332, the third reading unit 333, and the fourth reading unit 334 read the photoelectric conversion signal from the photoelectric conversion unit 32.

The first reading unit 331 includes a transfer transistor TX1, a reset transistor RST1, a floating diffusion FD1, an amplification transistor AMP1, and a row selection transistor SEL1. The transfer transistor TX1 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion unit 32) output by the photoelectric conversion unit 32 to the floating diffusion FD1 based on the signal φTX1. The reset transistor RST1 resets the floating diffusion FD1 based on the signal φRST. The amplification transistor AMP1 outputs a signal corresponding to the amount of electric charge stored in the floating diffusion FD1. The row selection transistor SEL1 outputs the signal output from the amplification transistor AMP1 to the vertical signal line V1 based on the signal φSEL. In the following description, a signal output to the vertical signal line V1, referred to as the output signal N _1.

The second reading unit 332 includes a transfer transistor TX2, a reset transistor RST2, a floating diffusion FD2, an amplification transistor AMP2, and a row selection transistor SEL2. Based on the signal φTX2, the transfer transistor TX2 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion unit 32) output by the photoelectric conversion unit 32 to the floating diffusion FD2. The reset transistor RST2 resets the floating diffusion FD2 based on the signal φRST. The amplification transistor AMP2 outputs a signal corresponding to the amount of electric charge stored in the floating diffusion FD2. The row selection transistor SEL2 outputs the signal output from the amplification transistor AMP2 to the vertical signal line V2 based on the signal φSEL. In the following description, a signal output to the vertical signal lines V2, referred to as the output signal N _2.

The third reading unit 333 includes a transfer transistor TX3, a reset transistor RST3, a floating diffusion FD3, an amplification transistor AMP3, and a row selection transistor SEL3. Based on the signal φTX3, the transfer transistor TX3 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion unit 32) output by the photoelectric conversion unit 32 to the floating diffusion FD3. The reset transistor RST3 resets the floating diffusion FD3 based on the signal φRST. The amplification transistor AMP3 outputs a signal corresponding to the amount of electric charge stored in the floating diffusion FD3. The row selection transistor SEL3 outputs the signal output from the amplification transistor AMP3 to the vertical signal line V3 based on the signal φSEL. In the following description, a signal output to the vertical signal line V3, referred to as the output signal N _3.

The fourth reading unit 334 includes a transfer transistor TX4, a reset transistor RST4, a floating diffusion FD4, an amplification transistor AMP4, and a row selection transistor SEL4. Based on the signal φTX4, the transfer transistor TX4 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion unit 32) output by the photoelectric conversion unit 32 to the floating diffusion FD4. The reset transistor RST4 resets the floating diffusion FD4 based on the signal φRST. The amplification transistor AMP4 outputs a signal according to the amount of electric charge stored in the floating diffusion FD4. The row selection transistor SEL4 outputs the signal output from the amplification transistor AMP4 to the vertical signal line V4 based on the signal φSEL. In the following description, a signal output to the vertical signal lines V4, referred to as the output signal N _4.

In this embodiment, the signal φSEL is input to the pixel 30 independently for each row. In the following description, the signal φSEL input to the pixel 30 on the nth line is referred to as a signal φSEL (n). For example, the signal φSEL (1) indicates a signal input to the pixel 30 on the first line. On the other hand, for the signal φRST, the same signal is input to all the pixels 30. For the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4, different signals are input to the first pixel 30a and the second pixel 30b. Each signal will be described in detail later.

FIG. 4 is a diagram schematically showing the modulated light 3 emitted from the light source unit 10. The horizontal axis of FIG. 4 represents time, and the vertical axis represents light intensity. The modulated light 3 is visible light in which the first transmitted modulated light 3a and the second transmitted modulated light 3b are superimposed. Transmitting the first modulated light 3a is the amplitude A _a, light modulated as a sine wave of period T _a. Frequency _{f a} of the transmitting first modulated light 3a is 1 / _{T a.} In the following description, the amplitude _{A a} transmission first modulated beams 3a, period _{T a,} the frequency _{f a,} the first amplitude _{A a,} respectively, the first period _{T a,} referred to as a first frequency _{f a.}

The second transmitted modulated light 3b is light modulated as a sine wave having an amplitude _Ab and a period T _b . The frequency fb of the transmission second modulated light 3b is 1 / T _b . In the following description, the amplitude A _b , the period T _b , and the frequency f _b of the transmission second modulated light 3 _b are referred to as the second amplitude _Ab , the second period T _b , and the second frequency f _b , respectively. The first period T _a is shorter than the second period T _b (for example, _Ta ≈ T _b / 7). In other words, the first frequency _{f a} is higher than the second frequency _{f b.}

FIG. 5 is a diagram schematically showing the relationship between the modulated light 3 emitted from the light source unit 10 and the reflected light 4 incident on the image pickup surface of the image pickup device 12. As schematically shown in FIG. 1, the reflected light 4 includes a first received modulated light 4a and a second received modulated light 4b. The first received modulated light 4a and the second received modulated light 4b enter the image sensor 12 after a time corresponding to the distance of the subject 2 has elapsed from the time when the modulated light 3 is emitted. In the following description, the time corresponding to the distance of the subject 2 is referred to as a delay time. Due to the delay time, a phase difference θa is generated between the transmission first modulation light 3a and the reception first modulation light 4a, and a phase difference between the transmission second modulation light 3b and the reception second modulation light 4b. θb is generated. As shown in FIG. 5, if the delay time is less than the first period T _a, the delay time represented by the delay time and the phase difference θb representing the phase difference θa are the same.

FIG. 6 is a diagram schematically showing the relationship between the modulated light 3 emitted from the light source unit 10 and the reflected light 4 incident on the image pickup surface of the image pickup device 12 when the subject 2 is farther away. As shown in FIG. 6, if the delay time is the first period T _a or more, the phase difference theta _a is apparently smaller than the true phase difference theta. In other words, Θ = 2πn × θ _a (n is an integer greater than or equal to 0). That is, if the delay time is the first period T _a or more, the delay time represented by the phase difference theta _a and the phase difference theta _b represents delay time are different values.

FIG. 7 is a timing chart showing a shooting operation. The control unit 13 causes the image sensor 12 to reset the pixels 30 during the shooting operation. The vertical scanning circuit 22 outputs a signal φRST, which is a control pulse signal, at time t1 based on the signal from the control unit 13. As described above, the signal φRST is output for all pixels 30. As a result, all the pixels 30 are reset.

At the time t2 when the reset is completed, the light emission control unit 13e causes the light source unit 10 to start emitting the modulated light 3. Synchronized with the emission of the modulated light 3 by the light source unit 10, at time t2, the first transfer control circuit 23a and the second transfer control circuit 23b are pulse signals for control, signal φTX1, signal φTX2, signal φTX3, and The output of the signal φTX4 is started. In the light emission control unit 13e, the position p1 at which the transmission first modulated light 3a is maximized coincides with the pulse center position p2 of the signal φTX1 output from the first transfer control circuit 23a to the first pixel 30a. In addition, the light source unit 10 is controlled. In the light emission control unit 13e, the position p3 at which the transmission second modulated light 3b is maximized coincides with the pulse center position p4 of the signal φTX1 output from the second transfer control circuit 23b to the second pixel 30b. In addition, the light source unit 10 is controlled. That is, the light emission control unit 13e synchronizes the modulated light 3 with the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4.

The first transfer control circuit 23a outputs a signal φTX1, a signal φTX2, a signal φTX3, and a signal φTX4 to the first pixel 30a. The first transfer control circuit 23a divides the period from time t2 to time t3 after only period _{T a} of the transmitting the first modulated light 3a into four periods, in their four periods, signal .phi.TX1, signal FaiTX2, signal Turn on φTX3 and the signal φTX4. The first transfer control circuit 23a repeatedly outputs the above-mentioned signal φTX1, signal φTX2, signal φTX3, and signal φTX4 during the exposure period Te.

The second transfer control circuit 23b outputs the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4 to the second pixel 30b. Second transfer control circuit 23b divides the period from time t2 to time t4 after only period _{T b} of the transmission second modulated light 3b into four periods, in their four periods, signal .phi.TX1, signal FaiTX2, signal Turn on φTX3 and the signal φTX4. The second transfer control circuit 23b repeatedly outputs the above-mentioned signal φTX1, signal φTX2, signal φTX3, and signal φTX4 during the exposure period Te.

At time t5, which is only the exposure period Te after time t2, the light emission control unit 13e causes the light source unit 10 to stop the emission of the modulated light 3. At time t5, the first transfer control circuit 23a and the second transfer control circuit 23b stop the output of the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4 in synchronization with the emission stop of the modulated light 3. At time t5, the electric charges generated by the photoelectric conversion unit 32 are accumulated in the floating diffusion FD1, the floating diffusion FD2, the floating diffusion FD3, and the floating diffusion FD4 of the pixel 30, respectively.

In other words, the charge photoelectrically converted from the reflected light 4 (including the first received modulated light 4a and the second received modulated light 4b) incident on the photoelectric conversion unit 32 during the period from time t2 to time t5 is a fine period. Each is individually transferred to the floating diffusion FD1 to FD4. For example, the electric charge converted photoelectric by the photoelectric conversion unit 32 is transferred to the floating diffusion FD1 while the signal φTX1 is on.

After time t5, the horizontal scanning circuit 21 and the vertical scanning circuit 22 start reading the output signals N _{1 to} N ₄ . The output signals N _{1 to} N ₄ are read out line by line. First, at time t5, the vertical scanning circuit 22 outputs a signal φSEL (1), which is a control pulse signal, to the pixel 30 in the first row. As a result, the output signals N _{1 to} N ₄ from each of the pixels 30 in the first line are output to the vertical signal lines V1 to V4. While the signal φSEL (1) is being output, the horizontal scanning circuit 21 outputs the output signals N _{1 to} N ₄ from each of the pixels 30 in the first row to the control unit 13 one by one. When the output signals N _{1 to} N ₄ are output to the control unit 13 for all the pixels 30 in the first row, the vertical scanning circuit 22 then outputs the signal φSEL (2) to the pixels 30 in the second row. To do. As a result, the output signals N _{1 to} N ₄ from each of the pixels 30 in the second line are output to the vertical signal lines V1 to V4. While the signal φSEL (2) is being output, the horizontal scanning circuit 21 outputs the output signals N _{1 to} N ₄ from each of the pixels 30 in the second row to the control unit 13 one by one. The horizontal scanning circuit 21 and the vertical scanning circuit 22 repeat the above operations for all the rows to output the output signals N _{1 to} N ₄ from all the pixels 30 to the control unit 13.

Next, the operations of the first detection unit 13a, the second detection unit 13b, the distance measurement unit 13c, and the image creation unit 13d will be described. FIG. 8 is an explanatory diagram illustrating the operations of the first detection unit 13a, the second detection unit 13b, the distance measurement unit 13c, and the image creation unit 13d. The modulated light 3 emitted to the subject 2 is reflected by the subject 2 and is incident on the first pixel 30a and the second pixel 30b as the reflected light 4 together with the background light. The first pixel 30a and the second pixel 30b output output signals N _{1 to} N ₄ . The first detection unit 13a calculates z _a , θ _a , and A _a, which will be described later, from the output signals N _{1 to} N _{4 of} the first pixel 30 a. The second detection unit 13b, _{z b,} calculates the theta _{b, A b,} which will be described later, from the output signal _N 1 to N ₄ of the second pixel 30b. The ranging unit 13c creates depth map data of the subject 2 based on the calculation results of the first detection unit 13a and the second detection unit 13b. Image creating unit 13d, the output signal _N 1 to N ₄ of the first pixel 30a and a second pixel 30b, to create the image data of the object 2. Hereinafter, these operations will be described in detail.

As described above, the light source unit 10 emits the modulated light 3 in which the transmission first modulation light 3a and the transmission second modulation light 3b are superimposed to the subject 2. The modulated light 3 is reflected by the surface of the subject 2 to become reflected light 4, and is incident on the first pixel 30a and the second pixel 30b of the image pickup device 12. However, since the subject 2 is also irradiated with background light from a light source such as the sun in addition to the modulated light 3, the reflected light 4 includes the received first modulated light 4a and the received second modulated light 4b. Reflected light of various background lights is also included.

The first pixel 30a photoelectrically converts the reflected light 4 and outputs the above-mentioned output signals N _{1 to} N ₄ . Similarly, the second pixel 30b also photoelectrically converts the reflected light 4 and outputs the above-mentioned output signals N _{1 to} N ₄ .

The first detection unit 13a calculates an output value z _a, which will be described later, based on the output signals N _{1 to} N ₄ output from the first pixel 30 a. The output value z _a is a complex numbered value of the photoelectric conversion result of the received first modulated light 4a included in the reflected light 4. From the output value z _a , the first amplitude A _a of the received first modulated light 4a included in the reflected light 4 and the phase difference θ _{a of} the received first modulated light 4a with respect to the transmitted first modulated light 3 _a are calculated. .. However, although the details will be described later, the phase difference θ _a calculated here may be different from the true phase difference Θ of the received first modulated light 4a with respect to the transmitted first modulated light 3a. Part 13c is corrected separately.

The first detection unit 13a calculates an output value z _a for each of the first pixels 30a based on the output signals N _{1 to} N ₄ output from the respective first pixels 30 a. The first detection unit 13a calculates _a plurality of output values za in each first pixel 30a.

The second detection unit 13b calculates an output value z _b, which will be described later, based on the output signals N _{1 to} N ₄ output from the second pixel 30 _b . The output value z _b is a complex numbered value of the photoelectric conversion result of the received second modulated light 4b included in the reflected light 4. From the output value z _b, a second amplitude A _b of the received second modulated light 4b, a phase difference theta _b for transmitting the second modulated light 3b of the received second modulated light 4b is calculated to be contained in the reflected light 4 ..

The second detection unit 13b calculates an output value z _b for each second pixel 30b based on the output signals N _{1 to} N ₄ output from the respective second pixels 30 _b . The second detection unit 13b calculates a plurality of output values z _b in each of the second pixels 30 _b .

The distance measuring unit 13c corrects the phase difference θ _a detected by the first detecting unit 13a by performing an operation described later using the output value z _b calculated by the second detecting unit 13 _b , and is the true position. Calculate the phase difference Θ. Then, the distance L (depth L) of the subject portion corresponding to one pixel 30 from the imaging device 1 is calculated by performing the calculation described later using this true phase difference Θ. The distance measuring unit 13c creates depth map data in which the depth L of each subject portion is two-dimensionally mapped by calculating the depth L for all the pixels 30.

Image creating unit 13d, and the output signal _N 1 to N ₄ output from the first pixel 30a, on the basis of an output signal _N 1 to N ₄ output from the second pixel 30b, create image data of the object 2 To do. Although details will be described later, the image creating unit 13d, based on the output signal N ₁ to N ₄ output from the first pixel 30a in to determine the pixel value corresponding to the first pixel 30a. Further, the image creating unit 13d, based on the output signal _N 1 to N ₄ output from the second pixel 30b that determines the pixel values corresponding to the second pixel 30b. In this way, the image creation unit 13d determines the pixel value of the pixel 30 for each pixel 30 based on the output signals N _{1 to} N ₄ output from the pixel 30.

Next, _a method of calculating the output value za by the first detection unit 13a will be described. First detecting unit 13a, a first pixel 30a in a period of time t2~t5 is based on the output signal _N 1 to N ₄ output result of the photoelectric conversion, the reflected light 4 photoelectric conversion result from the received first modulated light 4a photoelectric conversion result, i.e. to detect a photoelectric conversion result of the incident light having a first frequency f _a. The output signal _Ni (i = 1 to 4) can be expressed as the following equation (1).

Here, η is the sensitivity of the image sensor 12, and φ _R (t) is the intensity of the reflected light 4. _{Further, g} i (t) is a signal φTXi illustrated in FIG. 7 (φTX1~φTX4). For example, g ₁ (t) is a function that becomes 1 when the signal φTX1 is on and 0 at other times. From the above equation (1), N ₁ −N ₃ becomes as in the following equation (2).

As shown in FIG. 9, g ₁ (t) −g ₃ (t) is a signal that oscillates in the section of +1 to -1 with 0 as the center. This period of time t2~t3 can be approximated by cos2πf _a t (sine wave or period _{T a)} sine wave having a period. Further, since the reflected light 4 is light obtained by adding the received first modulated light 4a, the received second modulated light 4b, and other background light, it can be expressed by the following equation (3).

The above equation (3) is an equation expressing light obtained by superimposing k types of light, one of which is the received first modulated light 4a, the other one is the received second modulated light 4b, and the rest. Is the background light. In the above equation (3), f _k (f ₀ , f ₁ , f ₂ , ...) Is the frequency of each light contained in the reflected light 4, and _Ak (A ₀ , A ₁ , A ₂ , ...) Is each. Is the amplitude of light. Further, Δt is the difference between the time when the modulated light 3 is projected from the light source unit 10 and the time when the reflected light 4 is incident on the first pixel 30a, that is, the delay time.

If g ₁ (t) -g ₃ (t) is approximated by a sine wave and the above equation (3) is introduced, N _{1 −} N ₃ can be expressed by the following equation (4).

Here, as expressed by the following equation (5), the result obtained by integrating the product of each other (sine wave of the sine wave and the frequency f _p of the frequency f _k) sine wave having a different frequency can be regarded as zero.

Applying this to the above equation (4), of the term for a number of light included in the reflected light 4, light only the term relates to an optical (i.e. receiving the first modulated light 4a) the remaining (other first frequency f _a Since the term related to is 0), N _{1 to} N ₃ is finally expressed by the following equation (6).

That is, when the output signal N ₃ is subtracted from the output signal N _1, only the photoelectric conversion result of the received first modulated light 4a can be extracted from the photoelectric conversion result of the reflected light 4. When the operations on the output signal N ₁ and the output signal N ₃ described above are performed in the same manner for the output signal N ₂ and the output signal N ₄ , N _2- N ₄ is expressed by the following equation (7). To.

Here, the complex number of the above equations (6) and (7) is the output value z _a . The output value z _a is represented by the following equation (8).

That is, the output value z _a is a value obtained by extracting only the value corresponding to the received first modulated light _{4 a} from the received output of the reflected light 4 of the first pixel 30 a, that is, the output signals N _{1 to} N ₄ .

The output value z _a first detection unit 13a detects, if Atehamere the following equation (9) can calculate the distance L of the object part corresponding to the first pixel 30a. In the following equation (9), c is the speed of light. Further, argz _a is an argument of the output value z _a , that is, a phase difference θ _a between the transmission first modulation light 3a and the reception first modulation light 4a.

However, only the output value z _a, which is detected by the first detector 13a, it is impossible to accurately calculate the distance L. This is because, as described with reference to FIG. 6, the delay time Δt is increased, because the phase difference theta _{a (i.e.} declination argz a output value z _a) is not determined to a single value. Assuming that the delay time Δt is sufficiently small, it can be assumed that the phase difference θ _a is less than 2π, as shown in FIG. On the other hand, when the possibility that the delay time Δt is large must be considered, that is, the possibility that the phase difference θ _a is 2π or more must be considered, the phase difference θ _a obtained by calculation must be considered. However, it is not possible to determine how many cycles the deviation is. In other words, the true phase difference Θ between the transmission first modulation light 3a and the reception first modulation light 4a may be the phase difference θ _a obtained by calculation plus an integral multiple of 2π. ..

FIG. 10 is an explanatory diagram of the calculated phase difference θ _a and the true phase Θ, and is a graph in which the horizontal axis is the distance L and the vertical axis is the phase. As the distance L increases from 0, that is, as the subject 2 moves away from the image pickup apparatus 1, the calculated phase difference θ _a increases. When the distance L is a certain value L1, the calculated phase difference θ _a is apparently reset from 2π to 0, and increases again from 0 as the distance L increases.

Therefore, for example, when the calculated phase difference θ _a is π, the true phase difference Θ is any of π, 3π, 5π, ..., And the distance L is L1 / 2, 3 × L1 / 2. It can only be determined that it is either 5 × L1 / 2, ...

Therefore, the distance measuring unit 13c of the present embodiment not only detects the received first modulated light 4a from the reflected light 4 by the first detecting unit 13a, but also detects the received second modulated light 4b by the second detecting unit 13b. As a result, the distance L is calculated. Specifically, the ranging unit 13c corrects the phase difference θ _a using the output value z _b calculated by the second detection unit 13 _b , and obtains the true phase difference Θ. Then, the distance L is calculated by substituting the corrected true phase difference Θ into the above equation (9) instead of the pre-corrected phase difference θ _a .

First, a description method of calculating the output value _{z b} by the second detector 13b. Content of operation in which the second detection unit 13b performs, except that the content of g i _(t) are different, the same as the method of calculating the output value z _a by the first detection unit 13a described above. As described above, since the second pixel 30 b is driven based on the second period T _b , the output value z _b is derived from the light receiving output of the reflected light 4 of the second pixel 30 _{b, that} is, the output signals N _{1 to} N ₄ . It is a value obtained by extracting a value corresponding to the received second modulated light 4b.

As described above, the second period _{T b,} longer than the first period _{T a.} Therefore, as shown in FIG. 10, the declination argz _b (phase difference θ _b ) calculated by the second detection unit 13b does not change from 2π to 0 up to a distance L2 longer than L1 (continuously). is there). That is, if the second period T _b is set sufficiently long, even if the true phase difference Θ is not known only by the phase difference θ _a detected by the first detection unit 13a, it is detected by the second detection unit 13b. The true phase difference Θ can be obtained by using the phase difference θ _b together.

For example, in FIG. 10, when the phase difference θ _a is π, the distance L is any one of L1 / 2, 3 × L1 / 2, 5 × L1 / 2, and so on, as described above. That is, there are a plurality of candidates for the distance L derived from the phase difference θ _a . Here, if the phase difference θ _b is a value of around π / 2, the distance L is 3 × L1 / from those plurality of candidates (L1 / 2, 3 × L1 / 2, 5 × L1 / 2, ...). It is specified by one value of 2.

Next, the specific calculation contents by the distance measuring unit 13c will be described. The distance measuring unit 13c first obtains an integer n indicating how many cycles the phase difference θ _a is deviated from the true phase difference Θ by the following equation (10).

In the above equation (10), round (x) is a function that returns the integer closest to x (a function that rounds x). For example, when the distance L is less than the distance L1 shown in FIG. 10 (when the true phase difference Θ is less than 2π), the phase difference θ _a is equal to the true phase difference Θ, and n is 0. When the phase difference θ _a deviates from the true phase difference Θ by one cycle, n becomes 1, and the true phase difference Θ is the phase difference θ _a plus 2π.

The distance measuring unit 13c obtains the true phase difference Θ by the following equation (11). After that, the distance measuring unit 13c calculates the distance L to the subject by applying the true phase difference Θ to the following equation (12).

As described above, by detecting the received first modulated light 4a and the received second modulated light 4b, it is possible to take a wide range in which the distance can be measured. For example, the first frequency _{f a} 300 MHz, when the second frequency _{f b} and 3 MHz, the seek distance L by detecting only the reception first modulated light 4a, measurable distance L is maximum 1m approximately, the distance The maximum resolution is about 1 cm. Further, when only the received second modulated light 4b is detected and the distance L is obtained, the distance L that can be measured is about 100 m at the maximum, but the distance resolution is about 1 m at the maximum. In the present embodiment, the distance measuring unit 13c detects both the received first modulated light 4a and the received second modulated light 4b to obtain the distance L, thereby expanding the distance that can be measured to a maximum of 100 m and the distance resolution. Keep up to 1 cm.

As shown schematically in FIG. 2B, since the first pixel 30a and the second pixel 30b are at different positions, the output value z _a and the output value z _b can be simultaneously set for the same subject position. You can't get it. Therefore, the distance measuring unit 13c has an output value z _a calculated based on the first pixel 30 a and four output values z _b calculated based on the second pixel 30 b existing on the top, bottom, left, and right of the first pixel 30 a. The distance L corresponding to the first pixel 30a is calculated using the average value. Further, the distance measuring unit 13c is an average of the output value z _b calculated based on the second pixel 30 _b and the four output values z _a calculated based on the first pixel 30 a existing on the top, bottom, left and right of the second pixel 30 b. The value is used to calculate the distance L corresponding to the second pixel 30b. By doing so, it is possible to create depth map data having an number of elements (resolution) equal to the number of pixels 30.

Next, the image creation unit 13d will be described. As shown in FIG. 7, the electric charges generated by the photoelectric conversion unit 32 during the exposure period Te are dispersed and accumulated in the floating diffusion FD1 to FD4. Therefore, by adding the output signals N _{1 to} N ₄ , a photoelectric conversion signal based on the exposure period Te can be obtained. The image creation unit 13d obtains the pixel value (luminance value) of the pixel 30 by adding the output signals N _{1 to} N ₄ for each pixel 30. The image creation unit 13d creates image data by performing this for all the pixels 30.

For convenience of explanation, it is assumed that the pixel 30 does not have a color filter in the present embodiment. That is, the photoelectric conversion signal output from each pixel 30 does not have color information. Therefore, the image data created by the image creation unit 13d is a monochrome image having only the luminance information. However, it is also possible to provide a color filter for each pixel 30 so that the image creation unit 13d generates image data of a color image. For example, a pixel 30 provided with a color filter that passes only red component light, a pixel 30 provided with a color filter that passes only blue component light, and a pixel provided with a color filter that passes only green component light. 30 and 30 are arranged on the imaging surface 20 according to the so-called Bayer arrangement. The image creation unit 13d generates image data of a color image by executing a well-known demosaic process based on the pixel values obtained for each pixel 30.

Although the method of creating image data and depth map data in still image shooting has been described above, it is also possible to perform moving image shooting, that is, to create image data and depth map data for each frame. When shooting a moving image, the pixels 30 may be reset, exposed, and read out line by line.

According to the image pickup apparatus according to the first embodiment described above, the following effects can be obtained.
(1) The transmission first modulation light 3a and the transmission second modulation light 3b having different cycles are reflected by the subject 2 (object), and the first pixel 30a is used as the reception first modulation light 4a and the reception second modulation light 4b. And incident on the second pixel 30b. The first detection unit 13a (first phase difference measurement unit) determines the phase difference θ _a between the phase of the transmission first modulation light 3a and the phase of the reception first modulation light 4a based on the output signal of the first pixel 30a. Detect (measure). The second detection unit 13b (second phase difference measurement unit) determines the phase difference θ _b between the phase of the transmission second modulation light 3b and the phase of the reception second modulation light 4b based on the output signal of the second pixel 30b. Detect (measure). The distance measuring unit 13c calculates the distance L (distance value) with respect to the subject 2 based on the detected (measured) phase differences thereof. Since this is done, it is possible to perform high-precision and wide-measurement range distance measurement by removing the influence of the background light by performing the modulated light emission and the photographing once.

(2) The image sensor 12 transmits the output signal of the first pixel 30a to the output signals N _{1 to} N ₄ (first to _fourth) with a predetermined phase delay during one cycle of the transmission first modulated light 3a. Read sequentially as a signal). The first detection unit 13a measures the phase difference θ _a based on the difference between the output signal N ₁ and the output signal N ₃ and the difference between the output signal N ₂ and the output signal N ₄ . The image sensor 12 uses the output signal of the second pixel 30b as output signals N _{1 to} N ₄ (fifth to eighth signals) with a predetermined phase delay during one cycle of the transmission second modulated light 3b. Read sequentially. The second detection unit 13b measures the phase difference θ _b based on the difference between the output signal N ₁ and the output signal N ₃ and the difference between the output signal N ₂ and the output signal N ₄ . Since this is done, the influence of the background light can be removed from the photoelectric conversion signal.

(3) Each of the first pixel 30a and the second pixel 30b has a microlens and a photoelectric conversion unit 32. The image sensor 12 sequentially reads the output signals N _{1 to} N ₄ from the photoelectric conversion unit 32 of the first pixel 30a. The image sensor 12 sequentially reads the output signals N _{1 to} N ₄ from the photoelectric conversion unit 32 of the second pixel 30b. Since this is done, the light incident on one pixel 30 can be effectively used.

(4) The image sensor 12 reads the output signals N _{1 to} N ₄ from the photoelectric conversion unit 32 of the first pixel 30a, respectively, in the first reading unit 331, the second reading unit 332, the third reading unit 333, and the fourth reading unit. It has a part 334. The image sensor 12 includes a first reading unit 331, a second reading unit 332, a third reading unit 333, and a fourth reading unit 334 that read output signals N _{1 to} N ₄ from the photoelectric conversion unit 32 of the second pixel 30b, respectively. Have. Since this is done, the output signals N _{1 to} N ₄ can be easily obtained.

(5) The image creating unit 13d (the image data generating unit) is added to the sum and the sum signal obtained by adding the output signal _N 1 to N ₄ of the first pixel 30a, the output signal _N 1 to N ₄ of the second pixel 30b Based on the signal, image data related to the image of the subject 2 by the optical system 11 (imaging optical system) is generated. Since this is done, image data based on ordinary visible light can be obtained at the same time as depth map data.

(6) The first pixel 30a and the second pixel 30b are distributed substantially uniformly over the entire image pickup surface 20 of the image pickup device 12, respectively, and the distance measuring unit 13c includes the first pixel 30a and the first pixel 30a located in the vicinity of each other. For the second pixel 30b, the distance L is calculated based on the phase differences θ _a and θ _b measured by the first detection unit 13a and the second detection unit 13b, respectively. Since this is done, it is possible to create depth map data having an number of elements equal to the number of pixels.

(7) The distance measuring unit 13c is a depth map representing a map relating to the distance L at the positions of the first pixel 30a and the second pixel 30b based on the distance L for each first pixel 30a and the distance L for each second pixel 30b. Create data. Since this is done, it is possible to obtain not only the main part of the subject 2 but also the distance of each part of the subject 2.

(8) The light source unit 10 emits modulated light 3, which is visible light, to the subject 2. Since this is done, image data based on ordinary visible light can be obtained at the same time as depth map data.

(Second Embodiment)
FIG. 11 is a cross-sectional view schematically showing the configuration of the image pickup apparatus 100 according to the second embodiment. In the following description, the points different from those of the image pickup apparatus 1 according to the first embodiment will be mainly described, and the description of other parts will be omitted.

The image pickup device 100 has a configuration in which the image pickup device 12 of the image pickup device 1 according to the first embodiment is replaced with the image pickup element 112, and the control unit 13 is replaced with the control unit 113. The image sensor 112 has a pixel 130 described later instead of the pixel 30 described above. The control unit 113 has a first detection unit 13a, a second detection unit 13b, a distance measuring unit 13c, an image creation unit 13d, and a light emission control unit 13e, similarly to the control unit 13 according to the first embodiment. Further, it has a focus detection unit 13f and a focus adjustment unit 13g. The focus detection unit 13f and the focus adjustment unit 13g will be described in detail later.

FIG. 12 is a plan view schematically showing the structure of the pixel 130. As in the first embodiment, the pixel 130 of the present embodiment also has two types, the first pixel 130a and the second pixel 130b, as in the first embodiment (that is, FIG. 2). (As shown in (b)), the first pixel 130a and the second pixel 130b are arranged alternately. That is, the second pixel 130b exists on the top, bottom, left, and right of the first pixel 130a, and the first pixel 130a exists on the top, bottom, left, and right of the second pixel 130b. Therefore, substantially the same number of first pixels 130a and second pixels 130b are uniformly (uniformly) arranged on the imaging surface 20.

The pixel 130 has a first photoelectric conversion unit 132a and a second photoelectric conversion unit 132b instead of the photoelectric conversion unit 32. The shapes of the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b are substantially equal to those obtained by dividing the photoelectric conversion unit 32 into two equal parts by a straight line along the y direction in the xy plane. Therefore, the pair of light fluxes that have passed through the pair of regions of the exit pupil of the optical system 11 are incident on the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b, respectively.

FIG. 13 is a circuit diagram showing the configuration of the image pickup device 12. Note that FIG. 13 shows only a circuit diagram of one pixel 130. The pixel 130 includes a first reading unit 331, a second reading unit 332, a third reading unit 333, and a fourth reading unit 334. The first reading unit 331 and the second reading unit 332 read the photoelectric conversion signal from the first photoelectric conversion unit 132a. The third reading unit 333 and the fourth reading unit 334 read the photoelectric conversion signal from the second photoelectric conversion unit 132b. Since the configurations of the first reading unit 331, the second reading unit 332, the third reading unit 333, and the fourth reading unit 334 are the same as those of the first embodiment, the description thereof will be omitted.

FIG. 14 is a timing chart showing a shooting operation. The control unit 13 causes the image sensor 12 to reset the pixels 30 during the shooting operation. The vertical scanning circuit 22 outputs the signal φRST at time t11 based on the signal from the control unit 13. The signal φRST is output to the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b of all the pixels 130. As a result, the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b of all the pixels 30 are reset.

At the time t12 when the reset is completed, the light emission control unit 13e causes the light source unit 10 to start emitting the modulated light 3. At time t12, the first transfer control circuit 23a and the second transfer control circuit 23b start outputting the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4 in synchronization with the emission of the modulated light 3 by the light source unit 10. .. In the light emission control unit 13e, the position p5 at which the transmission first modulated light 3a is maximized coincides with the pulse center position p6 of the signal φTX1 output from the first transfer control circuit 23a to the first pixel 30a. In addition, the light source unit 10 is controlled. In the light emission control unit 13e, the position p7 at which the transmission second modulated light 3b is maximized coincides with the pulse center position p8 of the signal φTX1 output from the second transfer control circuit 23b to the second pixel 30b. In addition, the light source unit 10 is controlled. That is, the light emission control unit 13e synchronizes the modulated light 3 with the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4.

The first transfer control circuit 23a outputs a signal φTX1, a signal φTX2, a signal φTX3, and a signal φTX4 to the first pixel 130a. The first transfer control circuit 23a, a period from the time t12 to the time t13 after only period T _a of the transmitting first modulated light 3a, divides front half and two periods of the rear half. The first transfer control circuit 23a turns on the signal φTX1 in one of these two periods and turns on the signal φTX2 in the other. That is, the signal φTX1 is a signal having a phase different from that of the signal φTX2 by 1/2 period (180 degrees). The first transfer control circuit 23a outputs a signal obtained by delaying the signal φTX1 by 1/4 period (90 degrees) as the signal φTX3. The first transfer control circuit 23a outputs a signal obtained by delaying the signal φTX2 by 1/4 period (90 degrees) as the signal φTX4. The first transfer control circuit 23a repeatedly outputs the above-mentioned signal φTX1, signal φTX2, signal φTX3, and signal φTX4 during the exposure period Te.

In the first embodiment, the signal φTX1 and signal φTX2 includes a signal for turning on the first quarter period of one cycle of the transmission first modulated light 3a (period T _a), a signal for turning on the second quarter cycle Met. In contrast, the signal φTX1 and signal φTX2 in the present embodiment, a signal for turning on the first half period of one period of transmitting the first modulated light 3a (period T _a), on the second half cycle It is a signal to do. That is, in the present embodiment, the first reading unit 331 and the second reading unit 332 reverse the photoelectric conversion signal from the first photoelectric conversion unit 132a during one cycle (cycle _Ta ) of the transmission first modulated light 3a. Read alternately in phase.

In the first embodiment, the signal φTX3 and signal φTX4 includes a signal for turning on the third quarter cycle of one period of the transmission first modulated light 3a (period T _a), a signal for turning on the fourth quarter cycle of Met. In contrast, the signal φTX3 and signal φTX4 in the present embodiment, a signal for turning on the second and third quarter cycle of one period of the transmission first modulated light 3a (period T _a), first and second It is a signal that turns on in the quarter cycle of 4. That is, in the present embodiment, the third reading unit 333 and the fourth reading unit 334 reverse the photoelectric conversion signal from the second photoelectric conversion unit 132b during one cycle (period _Ta ) of the transmission first modulated light 3a. Read alternately in phase.

Second transfer control circuit 23b is a period from the time t12 to the time t14 after only period T _b of the transmission second modulated light 3b, divides front half and two periods of the rear half. The second transfer control circuit 23b turns on the signal φTX1 in one of those two periods and turns on the signal φTX2 in the other. That is, the signal φTX1 is a signal having a phase different from that of the signal φTX2 by 1/2 period (180 degrees). The second transfer control circuit 23b outputs a signal obtained by delaying the signal φTX1 by 1/4 period (90 degrees) as the signal φTX3. The second transfer control circuit 23b outputs a signal obtained by delaying the signal φTX2 by 1/4 period (90 degrees) as the signal φTX4. The second transfer control circuit 23b repeatedly outputs the above-mentioned signal φTX1, signal φTX2, signal φTX3, and signal φTX4 during the exposure period Te.

At time t15, which is only the exposure period Te after time t12, the light emission control unit 13e causes the light source unit 10 to stop the emission of the modulated light 3. At time t15, the first transfer control circuit 23a and the second transfer control circuit 23b stop the output of the signal φTX1, the signal φTX2, the signal φTX3, and the signal φTX4 in synchronization with the emission stop of the modulated light 3. At time t15, the electric charges generated by the photoelectric conversion unit 32 are accumulated in the floating diffusion FD1, the floating diffusion FD2, the floating diffusion FD3, and the floating diffusion FD4 of the pixel 130, respectively.

After time t15, the horizontal scanning circuit 21 and the vertical scanning circuit 22 start reading the output signals N _{1 to} N ₄ . The output signals N _{1 to} N ₄ are read out line by line. First, at time t5, the vertical scanning circuit 22 outputs a signal φSEL (1) to the pixel 130 in the first row. As a result, the output signals N _{1 to} N ₄ from each of the pixels 130 in the first line are output to the vertical signal lines V1 to V4. The output signals N ₁ and N ₂ are signals based on the electric charge photoelectrically converted by the first photoelectric conversion unit 132a. The output signals N ₃ and N ₄ are signals based on the electric charge photoelectrically converted by the second photoelectric conversion unit 132b. While the signal φSEL (1) is being output, the horizontal scanning circuit 21 outputs the output signals N _{1 to} N ₄ from each of the pixels 130 in the first row to the control unit 13 one by one. When the output signals N _{1 to} N ₄ are output to the control unit 13 for all the pixels 130 in the first row, the vertical scanning circuit 22 then outputs the signal φSEL (2) to the pixels 130 in the second row. To do. As a result, the output signals N _{1 to} N ₄ from each of the pixels 30 in the second line are output to the vertical signal lines V1 to V4. While the signal φSEL (2) is being output, the horizontal scanning circuit 21 outputs the output signals N _{1 to} N ₄ from each of the pixels 130 in the second row to the control unit 13 one by one. The horizontal scanning circuit 21 and the vertical scanning circuit 22 repeat the above operations for all the rows to output the output signals N _{1 to} N ₄ from all the pixels 130 to the control unit 13.

Next, _a method of calculating the output value za by the first detection unit 13a will be described. The first detection unit 13a is based on the output signals N ₁ and N ₂ output from the first pixel 130a and the output signals N ₃ and N ₄ output from the first pixel 130a, and the photoelectric conversion result of the reflected light 4 the photoelectric conversion result of the received first modulated light 4a from, that detects the photoelectric conversion result of the incident light having a first frequency f _a.

In the first embodiment, N _1- N ₃ is considered in the formula (2), but in this embodiment, N _1- N ₂ is considered instead. As shown in FIG. 15, since g ₁ (t) −g ₂ (t) can be approximated by a sine wave, the following equation (13) is derived by the same procedure as that described in the first embodiment. ..

Similarly, N _3- N ₄ is expressed by the following equation (14).

The output value z _a is expressed by the following equation (15).

Therefore, the ranging unit 13c can create the depth map data as in the first embodiment.

Next, the image creation unit 13d will be described. As shown in FIG. 12, the pixel 130 according to the present embodiment has a first photoelectric conversion unit 132a and a second photoelectric conversion unit 132b. The shapes of the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b are substantially equal to those obtained by dividing the photoelectric conversion unit 32 in the first embodiment into two equal parts. Therefore, by adding the output signal of the first photoelectric conversion unit 132a and the output signal of the second photoelectric conversion unit 132b, a signal corresponding to the output signal of the photoelectric conversion unit 32 according to the first embodiment can be obtained. .. The image creation unit 13d obtains the pixel value (luminance value) of the pixel 130 by adding the output signals N _{1 to} N ₄ for each pixel 130. The image creation unit 13d creates image data by performing this for all the pixels 130.

Next, the focus detection unit 13f will be described. The focus detection unit 13f includes a plurality of pixels 130 (the first pixel 130a and the second pixel 130b alternately arranged) arranged in a row in the x direction within a predetermined focus detection area provided on the imaging surface 20. ), A signal obtained by adding the output signal N ₁ and the output signal N ₂ and a signal obtained by adding the output signal N ₃ and the output signal N ₄ are created. The signal obtained by adding the output signal N ₁ and the output signal N ₂ is a signal based on the charge photoelectrically converted by the first photoelectric conversion unit 132a during the exposure period Te (FIG. 14). The signal obtained by adding the output signal N ₃ and the output signal N ₄ is a signal based on the electric charge photoelectrically converted by the second photoelectric conversion unit 132b during the exposure period Te. This pair of signals is a signal corresponding to a pair of luminous fluxes that have passed through a pair of regions of the exit pupil of the optical system 11. That is, this pair of signals is a signal for performing so-called phase difference type focus detection. As is well known, the phase difference type focus detection is performed by performing a correlation calculation on a pair of signals while changing the shift amount little by little to obtain the shift amount having the largest correlation value, that is, the phase difference. This is a focus detection method that detects the focus adjustment state. The focus detection unit 13f performs a well-known phase difference type focus detection on the created pair of signals, and detects the focus adjustment state (defocus amount) of the optical system 11.

The focus detection area can be provided at an arbitrary position on the imaging surface 20. Further, a plurality of candidate areas may be provided on the imaging surface 20 in advance, and any one of the candidate areas may be selected as the focus detection area. In the above description, an example of performing focus detection in the x direction has been described, but it is also possible to perform focus detection in a direction different from this (for example, the y direction).

Next, the focus adjusting unit 13g will be described. The focus adjustment unit 13g determines the focus position of the optical system 11 based on either the depth map data created by the distance measuring unit 13c or the focus adjustment state of the optical system 11 detected by the focus detection unit 13f. Adjust. The optical system 11 has a focus lens (not shown), and the focus adjusting unit 13g adjusts the focus of the optical system 11 by driving the focus lens in the optical axis direction using an actuator (not shown).

The focus adjusting unit 13g first examines the current aperture value (F value) of the optical system 11 when adjusting the focus. When the aperture value is equal to or less than a predetermined threshold value, that is, when the aperture diameter of the aperture (not shown) of the optical system 11 is large and the parallax of the two pupils divided by the pupil division is large, the focus adjustment unit 13g is focused. The detection unit 13f detects the focus adjustment state, and adjusts the focus based on the detected focus adjustment state.

On the other hand, when the current aperture value of the optical system 11 is larger than a predetermined threshold value, that is, when the aperture diameter of the aperture (not shown) of the optical system 11 is small and the parallax of the two pupils divided by the pupil division is small, the focal point is set. The adjustment unit 13g causes the distance measuring unit 13c to create depth map data covering at least the range of the focus detection area, and adjusts the focus based on the created depth map data. For example, if it is found from the depth map data that the distance (depth) of the subject corresponding to the focus detection area is 1 m, the focus adjustment unit 13 g is a focus lens (not shown) so as to focus on the position of 1 m. To drive.

According to the image pickup apparatus according to the second embodiment described above, the following effects can be obtained.
(1) The transmission first modulation light 3a and the transmission second modulation light 3b having different cycles are reflected by the subject 2 (object), and the first pixel 130a is used as the reception first modulation light 4a and the reception second modulation light 4b. And incident on the second pixel 130b. The first detection unit 13a (first phase difference measurement unit) determines the phase difference θ _a between the phase of the transmission first modulation light 3a and the phase of the reception first modulation light 4a based on the output signal of the first pixel 130a. Detect (measure). The second detection unit 13b (second phase difference measurement unit) determines the phase difference θ _b between the phase of the transmission second modulation light 3b and the phase of the reception second modulation light 4b based on the output signal of the second pixel 130b. Detect (measure). The distance measuring unit 13c calculates the distance L (distance value) with respect to the subject 2 based on the detected (measured) phase differences thereof. Since this is done, it is possible to perform distance measurement with high accuracy and a wide measurement range by performing one-time modulated light emission and photographing.

(2) the image pickup device 12, an output signal of the first pixel 130a, during one cycle of the transmission first modulated light 3a, with a predetermined phase delay, the output signal _N 1 to N ₄ (first to fourth Read sequentially as a signal). The first detection unit 13a measures the phase difference θ _a based on the difference between the output signal N ₁ and the output signal N ₂ and the difference between the output signal N ₃ and the output signal N ₄ . The image sensor 12 uses the output signal of the second pixel 130b as output signals N _{1 to} N ₄ (fifth to eighth signals) with a predetermined phase delay during one cycle of the transmission second modulated light 3b. Read sequentially. The second detection unit 13b measures the phase difference θ _b based on the difference between the output signal N ₁ and the output signal N ₂ and the difference between the output signal N ₃ and the output signal N ₄ . Since this is done, the influence of the background light can be removed from the photoelectric conversion signal.

(3) Each of the first pixel 130a and the second pixel 130b has a microlens 31, a first photoelectric conversion unit 132a, and a second photoelectric conversion unit 132b. The imaging device 12 reads an output signal _N 1, _{N 2} of the output signals _N 1 to N ₄ from the first photoelectric conversion portion 132a of the first pixel 130a, the output signal _N 1 to N from the second photoelectric conversion unit 132b output signal _N 3 of the _4, reads the _{N 4.} The imaging device 12 reads an output signal _N 1, _{N 2} of the output signals _N 1 to N ₄ from the first photoelectric conversion portion 132a of the second pixel 130b, the output signal _N 1 to N from the second photoelectric conversion unit 132b output signal _N 3 of the _4, reads the _{N 4.} Since this is done, the light incident on one pixel 30 can be divided into pupils and photoelectrically converted.

(4) The image sensor 12 has a first reading unit 331 and a second reading unit 332 that read output signals N ₁ and N ₂ from the first photoelectric conversion unit 132a of the first pixel 130a, respectively, and a second reading unit 130a of the first pixel 130a. It has a third reading unit 333 and a fourth reading unit 334 that read output signals N ₃ and N ₄ from the photoelectric conversion unit 132b, respectively. The image sensor 12 has a first reading unit 331 and a second reading unit 332 that read output signals N ₁ and N ₂ from the first photoelectric conversion unit 132a of the second pixel 130b, respectively, and a second photoelectric conversion unit of the second pixel 130b. It has a third reading unit 333 and a fourth reading unit 334 that read output signals N ₃ and N ₄ from 132b, respectively. Since this is done in this way, the output signals N _{1 to} N ₄ can be easily obtained.

(5) first reading unit 331 of the first pixel 130a reads during a half cycle of one cycle of the transmission first modulated light 3a from the first photoelectric conversion portion 132a output signal _{N 1} of the first pixel 130a .. The second reading unit 332 of the first pixel 130a reads the output signal N ₂ from the first photoelectric conversion unit 132a of the first pixel 130a during the remaining half cycle of one cycle of the transmission first modulated light 3a. The third reading unit 333 of the first pixel 130a is the output signal _{N 3} from the second photoelectric conversion unit 132b transmits the first modulated light 3a of the first pixel 130a at a predetermined time delayed from the read by the first reading unit 331 Read during half a cycle of one cycle. The fourth reading unit 334 of the first pixel 130a, the second transmission output signal _{N 4} from the photoelectric conversion unit 132b first modulated light 3a of the first pixel 130a at a predetermined time delayed from the read by the second reading unit 332 Read during the remaining half of one cycle. The first reading unit 331 of the second pixel 130b reads during a half cycle of one cycle of the first transmission output signal _{N 1} from the photoelectric conversion unit 132a second modulated light 3b of the second pixel 130b. The second reading unit 332 of the second pixel 130b reads the output signal N ₂ from the first photoelectric conversion unit 132a of the second pixel 130b during the remaining half cycle of one cycle of the transmission second modulated light 3b. The third reading unit 333 of the second pixel 130b, the second transmission from the photoelectric conversion unit 132b output signal _{N 3} second modulated light 3b than read by the first reading unit 331 with a predetermined time delay the second pixel 130b Read during half a cycle of one cycle. The fourth reading unit 334 of the second pixel 130b, the second transmission from the photoelectric conversion unit 132b output signal _{N 4} second modulated light 3b than read by the second reading unit 332 with a predetermined time delay the second pixel 130b Read during the remaining half of one cycle. Since this is done, it is possible to read out a signal that can be used for distance measurement based on the optical flight time measurement method and that can also be used for distance measurement by the so-called phase difference detection method.

(6) The image creating unit 13d (the image data generating unit) is added to the sum and the sum signal obtained by adding the output signal _N 1 to N ₄ of the first pixel 130a, the output signal _N 1 to N ₄ of the second pixel 130b Based on the signal, image data related to the image of the subject 2 by the optical system 11 (imaging optical system) is generated. Since this is done, the depth map data can be created while performing the focus detection of the so-called phase difference detection method, and the image data of the subject 2 can also be created.

(7) The focus detection unit 13f (phase difference type focus detection unit) is based on a pair of signals read from the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b of the first pixel 130a and the second pixel 130b. , Calculate the defocus amount. Since this is done, the focus detection of the so-called phase difference detection method can be performed in parallel with the creation of the depth map data.

(8) the focus detection unit 13f, from the first pixel 130a and the first photoelectric conversion unit output signal _N 1 read from 132a, _{N 2} addition signal obtained by adding the second photoelectric conversion portion 132b of the second pixel 130b The defocus amount is calculated based on the pair with the added signal obtained by adding the read output signals N ₃ and N ₄ . Since this is done, the focus detection of the so-called phase difference detection method can be performed in parallel with the creation of the depth map data.

(9) The first pixel 130a and the second pixel 130b are respectively arranged substantially uniformly over the entire image pickup surface 20 of the image pickup element 12, and the distance measuring unit 13c is the first pixel 130a and the first pixel 130a located in the vicinity of each other. For the second pixel 130b, the distance L is calculated based on the phase differences θ _a and θ _b measured by the first detection unit 13a and the second detection unit 13b, respectively. Since this is done, it is possible to create depth map data having an number of elements equal to the number of pixels.

(10) The distance measuring unit 13c is a depth map representing a map relating to the distance L at the positions of the first pixel 130a and the second pixel 130b based on the distance L for each first pixel 130a and the distance L for each second pixel 130b. Create data. Since this is done, it is possible to obtain not only the main part of the subject 2 but also the distance of each part of the subject 2.

(11) When the aperture value of the optical system 11 is larger than a predetermined value (dark), the focus adjusting unit 13g adjusts the focus based on the distance L calculated by the distance measuring unit 13c, and the aperture value of the optical system 11 Is less than or equal to a predetermined value (bright), the focus is adjusted based on the focus adjustment state detected by the focus detection unit 13f. Since this is done, the focus can be adjusted even when the aperture value is large (dark) and the focus detection by the phase difference method cannot be performed.

The following modifications are also within the scope of the present invention, and one or more of the modifications can be combined with the above-described embodiment.

(Modification example 1)
The number and layout of the first pixel 30a and the second pixel 30b (the first pixel 130a and the second pixel 130b) may be different from the above-described embodiment. For example, in the first embodiment, as shown in FIG. 2B, substantially the same number of first pixels 30a and second pixels 30b are arranged alternately. On the other hand, in the arrangement shown in FIG. 16A, the entire imaging surface 20 is divided into 2 × 2 4-pixel blocks, and one block is composed of three first pixels 30a and one second pixel 30b. I am trying to be done. In this way, the first pixel 30a may be present in a larger number than the second pixel 30b.

As described with reference to FIG. 10, the phase difference θ _b detected by the second pixel 30 _b is used to identify the true phase difference Θ from the phase difference θ _a . That is, the detection accuracy required for the phase difference θ _b is not higher than that for the phase difference θ _a . Therefore, the number of the second pixels 30b may be large enough to specify the true phase difference Θ from the phase difference θ _a .

That is, as shown in FIG. 16B, when the imaging surface 20 is divided into a plurality of blocks 200 of M × N, if at least one second pixel 30b exists in one block 200, the depth map Data can be created. In this case, the position of the second pixel 30b may be irregular as shown in FIG. 16B. In this case, for the position where the first pixel 30a is present, the output value _{z a} determined from the output of the first pixel 30a, determined from one of the second pixel 30b of the block 200 in which the first pixel 30a belongs The distance L corresponding to the position is calculated by using the output value z _b . Also, the position where the second pixel 30b is present, the output value z _b determined from the output of the second pixel 30b, which is determined from the output of the first pixel 30a existing in the vicinity of the second pixel 30b The distance L corresponding to the position is calculated by using the output value z _a . Here, the output values z _a and z _b may be values determined from the output of a single pixel 30, or are determined from the outputs of a plurality of pixels 30 as in the first embodiment. A value determined from a plurality of values (for example, mean value, median value, mode value, etc.) may be used. That is, the output values z _a and z _b for calculating the distance L corresponding to a certain position are the average values of the output values z _a and z _b obtained from the pixels 30 in the vicinity thereof (calculated average value, geometric mean). The average value, median value, etc.) may be used.

By increasing the first pixel 30a and decreasing the second pixel 30b in this way, the accuracy of the depth map data created by the distance measuring unit 13c is improved. As shown in FIGS. 2B and 16B, the distance measuring unit is arranged by arranging the first pixel 30a and the second pixel 30b substantially uniformly over the entire image pickup surface 20 of the image sensor 12, respectively. It can be guaranteed that the pair of the first pixel 30a and the second pixel 30b with respect to the phase difference θ _a and the phase difference θ _b used by 13c to calculate the distance L are located close to each other.

As described in the first embodiment, the first pixel 30a and the second pixel 30b have the same structure. Therefore, it is possible that a certain pixel 30 functions as a first pixel 30a at one time and a second pixel 30b at another time. For example, when the first pixel 30a and the second pixel 30b are arranged in the layout as shown in FIG. 16A, the first pixel 30a and the second pixel 30b may be interchanged according to the characteristics of the subject. Good.

(Modification 2)
As described with reference to FIG. 7, in the first embodiment divides the first period T _a into four periods, so as to output a photoelectric conversion signal in the period of each as an output signal N ₁ to N ₄ , The first pixel 30a was formed. Thus, for example, when integrating the output signal N ₁ to N _4, it becomes the photoelectric conversion signal at all period of the first period T _a. However, even if the photoelectric conversion signals obtained by different control are used as output signals N _{1 to} N ₄ , depth map data can be created in the same manner as in the above-described embodiment.

Referring to FIG. 7, for example, in the last period of the period from the time t2 1 / 4T _a, if ON signal .phi.TX1, may be the corresponding photoelectric conversion signal and the output signal N _1. Necessarily, in all of the period from the time t2 1 / 4T _a, it is not necessary to turn on the signal .phi.TX1. The same applies to the signals φTX2 to φTX4.

Another example will be described with reference to FIG. For example, a signal for turning on the second quarter period of one cycle of the transmit signals φTX1 first modulated light 3a (period T _a), and a signal for turning on the signal φTX2 the fourth quarter cycle of the signal φTX3 third The signal φTX4 can be used as a signal that turns on in the first quarter cycle. In this case, the output signals N ₁ , N ₂ , N ₃ , and N ₄ are the same signals as the output signals N ₁ , N ₃ , N ₂ , and N _{4 in} the first embodiment, respectively. The same calculation as in the embodiment may be performed.

(Modification 3)
As described with reference to FIG. 14, in the second embodiment, the signal φTX3, φTX4 (control signal _{g 3 (t), g 4} (t)) and signals φTX1, φTX2 (control signal _g 1 (t _), was a signal having a 1/4 delayed by the phase of the first period _{T a} relative to g 2 (t)). However, the signals φTX3 and φTX4 may be signals having different phases. At least, the phases may be different with respect to the signals φTX1 and φTX2.

(Modification example 4)
In each of the above-described embodiments, the transmission first modulation light 3a and the transmission second modulation light 3b are sine waves having different periods from each other. Lights modulated in a form different from this may be used as transmission first modulation light 3a and transmission second modulation light 3b. Hereinafter, a modified example using such a modulated light 3 will be described.

FIG. 17 is a circuit diagram showing the configuration of the image pickup device 12 according to the modified example 4. Note that FIG. 17 shows only the circuit diagram of the first pixel 32a and the second pixel 32b. The first pixel 32a includes a photoelectric conversion unit 32, a first reading unit 331, and a second reading unit 332. The second pixel 32b includes a photoelectric conversion unit 32, a third reading unit 333, and a fourth reading unit 334.

FIG. 18 is a timing chart showing a shooting operation. In the modified example 4, the period of the transmission first modulation light 3a and the transmission second modulation light 3b are both the second period T _b (FIG. 4). Transmitting the first modulated light 3a in its cycle, has only the amplitude _{A 1} 1/2 period from the time t21 the first period _{T a} (FIG. 4), the period of rest the amplitude is zero. Transmitting second modulated light 3b in its cycle, it has only the amplitude A ₂ from the time t21 1/2 period of the second period T _b, the period of rest the amplitude is zero. That is, the transmission first modulation light 3a and the transmission second modulation light 3b are pulse modulation lights having the same period and different duty ratios.

As shown in FIG. 18, the modulated light 3 in which the first modulated light 3a and the second modulated light 3b are superimposed has an amplitude A ₃ (= A ₁ + A ₂ ) in the period from the time t21 to the time t22. , the light having the amplitude _{a 2} to the period from time t22 to time t24. Here, time t22 is the time t21 the time after only half of the first period _{T a,} time t24 is the time t21 the time after only half of the second period _{T b.} Further, it is assumed that the amplitude A ₁ is sufficiently larger than the amplitude A ₂ .

In Modification 4, during the period from time t21 to time t22 after only half of the first period _{T a,} the signal φTX1 is on, the signal φTX2 is off. Also, during the period from time t22 to time t23 after only half of the first period _{T a,} the signal φTX1 is off, the signal φTX2 is on.

On the other hand, focusing on the second pixel 30b, at time t21, from time t21 to time t24 after only half of the second period _{T b,} the signal φTX3 is on, the signal φTX4 is off. Further, the signal φTX3 is off and the signal φTX4 is on from the time t24 to the time t25, which is 1/2 of the second period T _b .

In the floating diffusion FD1 of the first pixel 32a, the electric charge obtained by photoelectrically converting the light 210 shaded in FIG. 18 among the reflected light 4 is accumulated. In other words, the output signal N ₁ output from the first pixel 32a is based on the amount of light 210 shaded in FIG. Similarly, the output signal N ₂ output from the first pixel 32a is based on the amount of light 211 shaded in FIG. When the phase difference θ _a between the transmission first modulation light 3a and the reception first modulation light 4a is zero, all the charges obtained by photoelectrically converting the portion of the reflected light 4 having the amplitude A ₁ are transferred to the floating diffusion FD1. The output signal N ₁ is much larger than the output signal N ₂ . As the phase difference θ _a increases, the ratio of the charge obtained by photoelectrically converting the portion of the reflected light 4 having the amplitude A ₁ is transferred to the floating diffusion FD 2 increases. That is, the phase difference θ _a can be obtained by calculating the ratio of the output signal N ₁ and the output signal N ₂ . The first pixel 32a performs photoelectric conversion not only on the portion of the reflected light 4 having an amplitude of A ₁ but also on a portion of the reflected light 4 having an amplitude of A ₂ . However, if the amplitude A ₁ is made sufficiently larger than the amplitude A _2, the amount of the output signal N ₁ and the output signal N ₂ is dominated by the influence of the portion where the amplitude is A ₁ , and the amplitude is A ₂ . The effect of some parts can be ignored.

In the floating diffusion FD3 of the second pixel 32b, the electric charge obtained by photoelectrically converting the light 212 shaded in FIG. 18 among the reflected light 4 is accumulated. In other words, the output signal N ₃ which is output from the second pixel 32b is based on the amount of light 212 shown by hatching in FIG. 18. Similarly, the output signal _{N 4} outputted from the second pixel 32b is based on the amount of light 213 shown by hatching in FIG. 18. When the phase difference θ _b between the second modulated light 3b for transmission and the second modulated light 4b for reception is zero, all the charges obtained by photoelectrically converting the reflected light 4 are transferred to the floating diffusion FD3, and the output signal N ₃ is an output signal. It becomes extremely larger than N _4. As the phase difference θ _b increases, the ratio of the charge obtained by photoelectrically converting the reflected light 4 to the floating diffusion FD4 increases. That is, the phase difference θ _b can be obtained by calculating the ratio of the output signal N ₃ and the output signal N ₄ . Note that the output signal N3, the result amplitude of the reflected light 4 is converted photoelectrically moiety is A ₁ is reflected. However, if the period during which the amplitude is A ₁ (duty ratio of the first transmitted modulated light 3a) is sufficiently reduced, the influence of the portion where the amplitude is A ₁ on the output signal N ₃ becomes negligibly small.

As described above, it is possible to use the modulated light as pulsed light instead of using the modulated light as high-frequency modulated light as in the first embodiment and the second embodiment.

(Modification 5)
The modulated light 3 emitted by the light source unit 10 to the subject 2 may be infrared light (for example, near-infrared light) instead of visible light. By doing so, especially when the subject 2 is a person, the modulated light 3 is not visible and the subject 2 does not feel the modulated light 3 bothersome, and the shooting can be performed smoothly.

(Modification 6)
In the above-described embodiment, the light source unit 10 is provided outside the housing of the image pickup apparatus 1 (for example, FIG. 1). The light source unit 10 can also be provided inside the housing of the imaging device 1. For example, a half mirror is inserted diagonally between the optical system 11 and the image pickup device 12, and a light source unit 10 that emits modulated light 3 in the y direction with respect to the half mirror is provided. In this way, the modulated light 3 emitted by the light source unit 10 is reflected by the half mirror and travels in the direction of the optical system 11, that is, in the direction of the subject 2. The reflected light 4 passes through the optical system 11 and the half mirror and is incident on the image sensor 12. By doing so, the degree of freedom in designing the housing of the image pickup apparatus 1 is increased.

(Modification 7)
In the above-described embodiment, the light source unit 10 emits the modulated light 3 composed of the transmission first modulated light 3a and the transmission second modulated light 3b to the subject 2, but the transmission first modulated light 3a and the transmission second The modulated light 3b may be emitted to the subject 2 separately. For example, the first light source unit that emits the transmission first modulated light 3a and the second light source unit that emits the transmission second modulated light 3b may be provided at different positions of the image pickup apparatus 1.

(Modification 8)
In the above-described embodiment, the image pickup apparatus capable of creating image data together with the depth map data has been described, but the image data may not be created. For example, it may be a distance measuring device that creates only depth map data.

(Modification 9)
In the second embodiment, the output signal N ₁ and the output signal N ₂ are not read from the first photoelectric conversion unit 132a, and the output signal N ₃ and the output signal N ₄ are not read from the second photoelectric conversion unit 132b. The output signal N ₁ and the output signal N ₃ may be read from the 1 photoelectric conversion unit 132a, and the output signal N ₂ and the output signal N ₄ may be read from the second photoelectric conversion unit 132b. Further, the output signal N ₁ and the output signal N ₄ may be read from the first photoelectric conversion unit 132a, and the output signal N ₂ and the output signal N ₃ may be read from the second photoelectric conversion unit 132b.

(Modification example 10)
In the first embodiment, as shown in FIG. 3, for the pixels 30 of a row, has been provided to four vertical signal lines V1~V4 for outputting an output signal N ₁ to N ₄ However, this may be provided only for the two vertical signal lines V1 and V3. In this case, the output signals N ₁ and N ₂ are time-divisionally output to the vertical signal line V1, and the output signals N ₃ and N ₄ are time-divisionally output to the vertical signal line V3. Similarly, only the vertical signal line V1 may be provided, and the output signals N _{1 to} N ₄ may be output to the vertical signal line V1 in a time-division manner.

The present invention is not limited to the above-described embodiment as long as the features of the present invention are not impaired, and other embodiments considered within the scope of the technical idea of the present invention are also included within the scope of the present invention. ..

1,100 ... Imaging device, 10 ... Light source unit, 11 ... Optical system, 12,112 ... Image sensor, 13,113 ... Control unit, 13a ... First detection unit, 13b ... Second detection unit, 13c ... Distance measuring unit , 13d ... Image creation unit, 13e ... Light emission control unit, 13f ... Focus detection unit, 13g ... Focus adjustment unit

Claims (18)

A light source unit that emits first and second modulated light to an object,
An image sensor having first and second pixels that receive the first and second modulated lights reflected by the object and generate a signal.
The output signal of the first pixel causes a first phase difference between the phase of the first modulated light emitted from the light source unit and the phase of the first modulated light received by the first pixel. The first phase difference measuring unit to be measured and
The second phase difference between the phase of the second modulated light emitted from the light source unit and the phase of the second modulated light received by the second pixel is determined by the output signal of the second pixel. The second phase difference measuring unit to be measured and
A distance calculation unit that calculates the distance to the object from the first and second phase differences, and
An image data generation unit that generates image data of the object based on the output signal of the first pixel and the output signal of the second pixel.
An imaging device comprising. In the imaging device according to claim 1,
The first modulated light has a first period and has a first period.
The second modulated light is an image pickup apparatus having a second cycle different from the first cycle. In the imaging device according to claim 2,
The image pickup device reads out the output signal of the first pixel at a timing based on the first cycle, and reads out the output signal of the second pixel at a timing based on the second cycle. In the imaging device according to claim 3,
The image sensor sequentially reads out the output signal of the first pixel as a first, second, third, and fourth signal with a predetermined phase delay during the first cycle.
The first phase difference measuring unit is based on the difference between the first signal and the third signal and the difference between the second signal and the fourth signal, and the first phase difference. Measure and
The image sensor sequentially reads out the output signal of the second pixel as the fifth, sixth, seventh, and eighth signals with a predetermined phase delay during the second cycle.
The second phase difference measuring unit is based on the difference between the fifth signal and the seventh signal and the difference between the sixth signal and the eighth signal, and the second phase difference. An imaging device that measures. In the imaging device according to claim 4,
Each of the first and second pixels has a microlens and a photoelectric conversion unit.
The image sensor sequentially reads out the first, second, third, and fourth signals from the photoelectric conversion unit of the first pixel.
The image pickup device is an image pickup device that sequentially reads out the fifth, sixth, seventh, and eighth signals from the photoelectric conversion unit of the second pixel. In the imaging device according to claim 5,
The image sensor has first, second, third, and fourth reading units that read the first, second, third, and fourth signals from the photoelectric conversion unit of the first pixel, respectively. Have and
The image sensor has fifth, sixth, seventh, and eighth reading units that read the fifth, sixth, seventh, and eighth signals from the photoelectric conversion unit of the second pixel, respectively. Image sensor to have. In the imaging device according to claim 4,
Each of the first and second pixels has a microlens and first and second photoelectric conversion units.
The image sensor reads two signals out of the first, second, third, and fourth signals from the first photoelectric conversion unit of the first pixel, and reads the second photoelectric conversion unit. The remaining two signals of the first, second, third, and fourth signals are read from
The image sensor reads two signals out of the fifth, sixth, seventh, and eighth signals from the first photoelectric conversion unit of the second pixel, and reads the second photoelectric conversion unit. An image pickup device that reads out the remaining two signals from the fifth, sixth, seventh, and eighth signals. In the imaging device according to claim 7,
The image sensor has a first and second reading unit that reads the two signals from the first photoelectric conversion unit of the first pixel, respectively, and the remaining two signals from the second photoelectric conversion unit. It has a third and fourth reading unit which reads out, respectively.
The image sensor has a fifth and a sixth reading unit that reads the two signals from the first photoelectric conversion unit of the second pixel, respectively, and the remaining two signals from the second photoelectric conversion unit. An image pickup device having a seventh and eighth reading units, respectively. In the imaging device according to claim 8,
The first reading unit reads one of the two signals from the first photoelectric conversion unit of the first pixel during half a period of the first period.
The second reading unit reads the other of the two signals from the first photoelectric conversion unit of the first pixel during the remaining half of the first period.
The third reading unit transmits one of the two signals from the second photoelectric conversion unit of the first pixel with a predetermined time delay from the reading by the first reading unit in the first cycle. Read during our half cycle,
The fourth reading unit performs the other of the two signals from the second photoelectric conversion unit of the first pixel with a predetermined time delay from the reading by the second reading unit in the first cycle. Read during the other half of our cycle,
The fifth reading unit reads one of the two signals from the first photoelectric conversion unit of the second pixel during half a period of the second period.
The sixth reading unit reads the other of the two signals from the first photoelectric conversion unit of the second pixel during the remaining half of the second period.
The seventh reading unit sends one of the two signals from the second photoelectric conversion unit of the second pixel to the second period with a predetermined time delay from the reading by the fifth reading unit. Read during our half cycle,
The eighth reading unit transfers the other of the two signals from the second photoelectric conversion unit of the second pixel to the second period with a predetermined time delay from the reading by the sixth reading unit. An imaging device that reads out during the remaining half of the cycle. In the imaging device according to any one of claims 4 to 9.
The image data generation unit includes an addition signal obtained by adding the first, second, third, and fourth signals, and an addition signal obtained by adding the fifth, sixth, seventh, and eighth signals. An image pickup device that generates the image data based on the above. In the imaging apparatus according to any one of claims 7 to 9,
A pair of a signal read from the first photoelectric conversion unit of the first pixel and a signal read from the second photoelectric conversion unit, and the first photoelectric conversion of the second pixel. An imaging device further including a phase difference type focus detection unit that calculates a defocus amount based on at least one pair of a signal read from the unit and a signal read from the second photoelectric conversion unit. .. In the imaging device according to claim 11,
The phase difference type focus detection unit is an addition signal obtained by adding the two signals read from the first photoelectric conversion unit of the first pixel and the addition signal read from the second photoelectric conversion unit. A pair of an addition signal obtained by adding two signals, an addition signal obtained by adding the two signals read from the first photoelectric conversion unit of the second pixel, and a reading from the second photoelectric conversion unit. An imaging device that calculates the amount of defocus based on at least one pair with a pair with an added signal obtained by adding the two signals. In the imaging apparatus according to claim 11 or 12,
The first cycle is smaller than the second cycle,
The phase difference type focus detection unit calculates the defocus amount based on the signal read from the first photoelectric conversion unit of the first pixel and the signal read from the second photoelectric conversion unit. Imaging device. In the imaging apparatus according to any one of claims 11 to 13.
An image pickup apparatus further comprising a focus adjustment unit that adjusts the focus based on at least one of the defocus amount of the phase difference type focus detection unit and the distance of the distance calculation unit. In the imaging device according to claim 14,
When the aperture value of the imaging optical system is larger than a predetermined value, the focus adjusting unit adjusts the focus based on the distance calculated by the distance calculating unit, and the aperture value of the imaging optical system is a predetermined value. An imaging device that adjusts the focus based on the defocus amount calculated by the phase difference type focus detection unit when the following is true. In the imaging apparatus according to any one of claims 1 to 15,
A depth map creation unit that creates a depth map regarding the distance at the positions of the first and second pixels based on the distance for each of the first pixels and the distance for each second pixel by the distance calculation unit. An imaging device further equipped with. In the imaging device according to claim 1,
The first modulated light is pulse-modulated light having a first duty ratio, and is
The second modulated light is an image pickup device which is pulse-modulated light having a second duty ratio different from the first duty ratio. A light source unit that emits first and second modulated light to an object,
An image sensor having first and second pixels that receive the first and second modulated lights reflected by the object and generate a signal.
The output signal of the first pixel causes a first phase difference between the phase of the first modulated light emitted from the light source unit and the phase of the first modulated light received by the first pixel. The first phase difference measuring unit to be measured and
The second phase difference between the phase of the second modulated light emitted from the light source unit and the phase of the second modulated light received by the second pixel is determined by the output signal of the second pixel. The second phase difference measuring unit to be measured and
A distance calculation unit that calculates the distance to the object from the first and second phase differences, and
A distance measuring device equipped with.

Images