Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/483—Details of pulse systems
  • G01S7/486—Receivers
  • G01S7/4861—Circuits for detection, sampling, integration or read-out
  • G01S7/4863—Detector arrays, e.g. charge-transfer gates
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
  • G01S17/06—Systems determining position data of a target
  • G01S17/08—Systems determining position data of a target for measuring distance only
  • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
  • G01S17/18—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein range gates are used
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/483—Details of pulse systems
  • G01S7/486—Receivers
  • G01S7/4865—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/483—Details of pulse systems
  • G01S7/486—Receivers
  • G01S7/487—Extracting wanted echo signals, e.g. pulse detection
  • G01S7/4876—Extracting wanted echo signals, e.g. pulse detection by removing unwanted signals

Definitions

• the present disclosure generally pertains to the field of Time-of- Flight imaging, and in particular to devices and methods for Time-of-Flight image capturing.
• a Time-of-Flight (ToF) camera is a range imaging camera system that determines the distance of objects by measuring the time of flight of a light signal between the camera and the object for each point of the image.
• a ToF camera has an illumination unit (a LED or VCSEL, Vertical- Cavity Surface-Emitting Laser) that illuminates a scene with modulated light.
• a pixel array in the ToF camera collects the light reflected from the scene and measures phase-shift which provides information on the travelling time of the light, and hence information on distance.
• iToF indirect Time of Flight
• dToF direct Time of Flight
• 3D images of a scene are captured. These images are also commonly referred to as “depth map”, or “depth image”, wherein each pixel of the image is attributed with a respective depth measurement.
• depth image can be determined directly from a phase image, which is the collection of all phase delays determined in the pixels of the iToF camera.
• dToF systems typically offer a strongly reduced resolution at distances below Im. They provide an effective resolution of QQQVGA or lower and only accuracy in the centimetre range. Between Im and 5m, there may be no decrease in resolution or increase in accuracy.
• one dTof may have good ambient light rejection, while still preserving a good resolution which is often achieved by analysing the histogram of the received number of photons and identifying corresponding objects as peaks in these histograms. Thereby, also scattering and multi-path may be mitigated.
• the disclosure provides an electronic device comprising circuitry configured to a determine, according to a Time-of-Flight, ToF, principle, a depth measurement in the frequency domain based on three or more measurements which correspond to respective three or more taps of a pixel.
• ToF Time-of-Flight
• the disclosure provides a method comprising determining, according to a Time-of-Flight, ToF, principle, a depth measurement in the frequency domain based on three or more measurements which correspond to respective three or more taps of a pixel.
• ToF Time-of-Flight
• Fig. 1 schematically shows the basic operational principle of an indirect Time-of-Flight imaging
• Fig. 3 shows a timing diagram concerning the control of transfer gates TG1,..., TG7 and overflow gate OFG of the N-tap pixel described in Fig. 2.
• Fig. 4a schematically shows an example of gate rotary switching with an activated transfer gate TG1;
• Fig. 4b schematically shows an example of gate rotary switching with an activated transfer gate TG2;
• Fig. 6 shows in a schematic way the determination of the phase value ⁇ between the emitted and the received light from the IQ measurement F(1);
• Fig. 8 shows an embodiment of a frame structure of a 7 -tap iToF pixel;
• Fig. 9 shows a timing diagram for a sub -integration cycle of a range gated iToF image sensor with seven taps
• Fig. 10 shows a radial detection range of an iToF camera with close-range and a far-range gated areas
• Fig. 11 shows another embodiment of radial detection range of an iToF camera with close-range and a far-range gated areas
• Fig. 12 shows two timing diagrams for a sub-integration cycle of a range gated iToF image sensor with seven taps using a tap skipping and a high precision method
• Fig. 13 shows four timing diagrams for four sub-integration cycles of an iToF image sensor with seven taps using dynamic range extension
• Fig. 14 shows a graph depicting the acquired current in the different taps of a 7-tap system (using dynamic range extension) depending on the distance of a detected object;
• Fig. 15 shows a timing diagram for a sub-integration cycle of an iToF image sensor with seven taps using range specific binning.
• Fig. 16 shows radial detection range of an iToF camera with range specific binning
• Fig. 17 schematically describes an embodiment of an iToF device that can implement the processes of determining the depth value of the object with an N-tap pixel.
• the embodiments disclose an electronic device comprising circuitry configured to a determine, according to a Time-of-Flight, ToF, principle, a depth measurement in the frequency domain based on three or more measurements which correspond to respective three or more taps of a pixel.
• ToF Time-of-Flight
• the electronic device may for example be an imaging sensor of an imaging camera, in particular a sensor of a ToF camera.
• the electronic device may be an imaging sensor with N-tap pixels, where N denotes the number of taps provided in a pixel of the imaging sensor.
• the electronic device may also comprise additional circuitry.
• circuitry of the electronic device may include electronic components such as switching elements (gates, transistors, etc.), resistors, memory elements (capacitors, RAM, ROM or the like), pixel circuitry, a storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g.
• liquid crystal liquid crystal, (organic) light emitting diode, etc.), loudspeakers, etc., a (wireless) interface, etc., as it is generally known for electronic devices (computers, smartphones, etc.).
• electronic devices computers, smartphones, etc.
• sensors for sensing still image or video image data image sensor, camera sensor, video sensor, etc.
• sensing a fingerprint for sensing environmental parameters (e.g. radar, humidity, light, temperature), etc.
• a tap may be any structure or configuration a camera sensor, respectively a pixel of a camera sensor uses to output data.
• a pixel may be any single cell of an image sensor.
• a photo sensor e.g. a photodiode
• the electrical charges accumulated in the pixels of an image sensor are transferred by read-out electronics from the pixels to a processer for post-processing of an image.
• the circuitry of the electronic device described below in more detail may be configured to determine a discrete Fourier transform based on the three or more measurements related to the taps of the pixel, and to determine the depth measurement based on the discrete Fourier transform.
• the term Fourier transform refers to both the frequency domain representation and the mathematical operation that associates the frequency domain representation to a function of space or time.
• the Fourier transform of a function of time is typically a complex-valued function of frequency.
• the circuitry may for example be configured to determine the discrete Fourier transform based on the total number of taps of the pixel and based on measurements provided by the respective taps of the pixel.
• the circuitry may be configured to determine the depth measurement based on a real part and an imaginary part of a Fourier transform.
• the discrete Fourier transform may for example be a first harmonic estimate which is also referred to as an IQ measurement (with I and Q being the real, respectively, imaginary part of the first harmonic estimate) which yields the phase delay between the emitted light and the reflected light.
• the phase delay yields the distance of the object. That means the phase delay information and depth information (as well as confidence and amplitude) may be obtained in the Fourier domain and, once the (discrete) Fourier transform is computed on the samples of the correlation waveform (i.e. the measurements provided by the taps), the first harmonic will contain I and Q information as its real and imaginary part, respectively.
• the circuitry further may comprise three or more transfer gates each corresponding to one of the three or more taps.
• a transfer gate may for example be implemented as a transistor. However, other switching elements may be used as an alternative to transistors.
• the circuitry may be configured to open the transfer gates of the respective taps of the pixel one after the other for a predetermined time.
• the circuitry may be configured to open the transfer gates according to a predetermined pulse width which is equal to the pulse width of pulses of an illumination pulse train.
• illumination pulse width may be wider or narrower than predetermined pulse width opening the transfer gates.
• the circuitry may be configured to generate an illumination pulse train which has a duty cycle equal or lower than 1%.
• the typical duty cycle may be 1/N, wherein N is the number of taps, which may yield a duty cycle of 12.5% for an 8-tap system.
• the circuitry may further comprise an overflow gate which is activated in a sub-integration cycle after the transfer gates have been activated.
• the circuitry may be configured to determine ambient light corrected measurements based on the measurements obtained by respective taps of the pixel.
• the circuitry may be configured to determine empty taps to perform ambient light suppression.
• An empty tap may be a tap without illumination reflected light determined after peak detection.
• the circuitry may be configured to determine ambient light corrected measurements by subtracting an average signal over empty taps from the measurements obtained by the respective taps of the pixel.
• the circuitry may be configured to set measurements that correspond to empty taps to zero.
• the circuitry may be configured to perform close-range gating.
• the circuitry may be configured to perform far-range gating.
• the circuitry may be configured to perform dynamic range extension.
• the circuitry may be configured to perform range-specific binning.
• the embodiments also disclose a method comprising determining, according to a Time-of-Flight, ToF, principle, a depth measurement in the frequency domain based on three or more measurements which correspond to respective three or more taps of a pixel.
• ToF Time-of-Flight
• Fig. 1 schematically shows the basic operational principle of an indirect Time-of-Flight imaging system which can be used for depth sensing.
• the iToF imaging system 101 includes an iToF camera with an imaging sensor 102 having a matrix of pixels and a processor (CPU) 105.
• a scene 107 is actively illuminated with amplitude-modulated infrared light LMS at a predetermined wavelength using an illumination device 110, for instance with some light pulses of at least one predetermined modulation frequency DML generated by a timing generator 106.
• the amplitude-modulated infrared light LMS is reflected from objects within the scene 107.
• a lens 103 collects the reflected light 109 and forms an image of the objects within the scene 107 onto the imaging sensor 102.
• the CPU 105 determines for each pixel a phase delay between the modulated signal DML and the reflected light RL. Based on these correlations a so called in-phase component value (“I value”) and a so called quadrature component value (“Q value”) can be determined (see below for a detailed description) for each pixel.
• I value in-phase component value
• Q value quadrature component value
• the ToF pixel also called here a variable-tap photo-electric converter
• the ToF pixel comprises a photodiode 201 which collects incident photons and generates a photo current.
• the photodiode 201 is connected to first transfer gates TG1 to TG7, each first transfer gate TG1 to TG7 being related to a respective tap of the pixel.
• By opening one of the transfer gates TG1 to TG7 the charge produced by photodiode 201 is transferred onto a respective memory MEM1 to MEM7 where it is stored.
• the charge collected onto a memory MEM1 to MEM7 is transferred to amplifier AMP and select transistor SEL to read out the amplified signal on a read out line VSL.
• a reset transistor RST is provided to reset the pixel.
• the photodiode 201 is also connected to an overflow gate OFG.
• the first transfer gates TG1 to TG7 and the overflow gate OFG are controlled according to the timing diagram as shown in Fig. 3.
• the activation of a new tap corresponds to an acquisition with a different phase shift applied on the reference/ emitted signal (a train of pulses, a sinusoid, etc).
• the received signal will include both this phase shift and an additional phase shift induced by the time of flight.
• the first transfer gates TG1,..., TG7, the second transfer gates TX1,..., TX7 and the overflow gate OFG may be implemented e.g. as transistors, and the memories MEM1 to MEM7 may for example be implemented as capacitors.
• N 7 -tap
• other numbers of taps may be foreseen in an N-tap pixel.
• N 3, 4, 5, 6, 8, 9, 10, 11, 12 or more taps.
• Fig. 3 shows a timing diagram concerning the control of transfer gates TG1,..., TG7 and overflow gate OFG of the N-tap pixel described in Fig. 2.
• a laser pulse for illuminating a scene is activated for a predefined illumination period.
• each of the transfer gates TG1, ..., TG7 of the pixel is activated one after the other for a predetermined activation time window of a window length that is identical to the illumination period.
• the overflow gate OFG is opened after the last transfer gate TG7 has been closed.
• the described process (called “sub-integration cycle") is repeated multiple times (see also Fig. 8) to achieve measurements over a predefined integration cycle (the sensor may be also configured without use of OFG, for example the transfer gate TG1 may be opened right after the transfer gate TG7).
• the length of the time window is equal for each transfer gate TG1, ..., TG7 and identical in length to the illumination period. It should however be noted that in alternative embodiments, there may be other time periods chosen.
• the predetermined activation time window may be different for all or some of the transfer gates TG1,...,TG7.
• the first activation time window may be longest for the first transfer gate TG1 and decreasing down to the last transfer gate TG7, or it may be shortest for the shortest for the first transfer gate TG1 and increasing up to the last transfer gate TG7.
• the activation time window may be equal for all transfer gates, while the width of the illumination signal may be lower or higher (though do not changed during integration period)
• the respective iToF measurements provides N different measurements collected at the N taps corresponding to N electronic transmit delays ( ⁇ E, n in Eq. 9 below). These N measurements are used to determine an approximation (estimate) of a first harmonic of the correlation waveform with a Fourier Transform as described below in more detail.
• the tap information is considered as component information, i.e. samples of the correlation waveform between the reference/ emitted signal and the received signal. That is, obtaining depth information (as well as confidence and amplitude) is done in the Fourier domain.
• the read-out period (not depicted in Fig. 3) starts.
• the signals q 0 ,..., q 6 are then used for determining the phase value between the emitted and the received light.
• the control signals according to Fig. 3 may be generated by a controller or any other means known to the skilled person.
• Gate rotary switching as depicted in a schematic way in Figs. 4a and 4b may symbolize this generation of the control signals.
• the gate rotary switching opens (i.e. activates) the transfer gates TG1, ... TG7 and the overflow gate OFG one after each other for a predefined period of time.
• Fig. 4a shows that the transfer gate TG1 is activated and the photo-electrons are directed to the transfer gate TG1.
• Fig. 4b shows that the transfer gate TG2 is activated and the photo-electrons are directed to the transfer gate TG2.
• Gate rotary switching as shown in Fig. 4 may also be used to control the transfer gates TX1, ... TX7 at the read-out side of the N-tap pixel.
• a (differential) iToF pixel measurement ⁇ ( ⁇ E , T D ) as obtained in the iToF pixel is a variable whose expected value ⁇ ( ⁇ E , T D ) is given by where, t is the time variable, T I is the exposure time (integration time), m(t) is the in-pixel reference signal which corresponds to the modulation signal (i.e. the emitted light signal) or a phase shifted version of the modulation signal (for example one of the control signals applied to transfer transistors TG1,...,TG7 as shown in Figs. 2 and 3).
• ⁇ R (t, ⁇ E , T D ) is the pixel irradiance signal which represents the reflected light (RL in Fig. 1) captured by the pixel.
• ⁇ E represents a time variable indicative of the time delay between the in-pixel reference signal (modulation signal) and the emitted light ( in Fig. 1)
• T D is a time variable representing the time that it is required for the light to travel from the iToF camera (101 in Fig. 1) to the object (107 in Fig. 1) and back.
• the time variable T D is given by: (Eq. 2) where D is the distance between the ToF camera and the object, and c is the speed of light.
• the reflected light signal ⁇ R (t, ⁇ E , T D ) is a scaled and delayed version of the emitted light ⁇ E (t — ⁇ E ).
• the pixel irradiance signal ⁇ R (t, ⁇ E , T D ) is given by: where ⁇ (T D ) is a real value scaling factor that depends on the distance D between the ToF camera and the object, and ⁇ E (t — ⁇ E — ⁇ D ) is the emitted light ⁇ E (t — ⁇ E ) (16 in Fig. 1) additionally delayed with the time variable T D .
• both m(t) and ⁇ E (t) are typically periodical signals with period (f M being the fundamental frequency or modulation frequency generated by the timing generator (106 in Fig. 1).
• f M being the fundamental frequency or modulation frequency generated by the timing generator (106 in Fig. 1).
• the expected signal ⁇ (T E , T D ) is also a periodical function with respect to the electronic delay T E between in-pixel reference signal m(t) and optical emission ⁇ E (t — ⁇ E ) with the same fundamental frequency f M .
• the expected signal ⁇ (T E , T D ) is not periodical with respect to the time-of-flight T D .
• H 1, ⁇ (T D ) is formulated in terms of the expected value ⁇ (T E , T D ) of signal measurements ⁇ (T E , T D ). Estimating this expected value from measurements may be performed by multiple repeated acquisitions (of static scene) to average out noise.
• H 1, ⁇ (T D ) is given as an integral over all possible transmit delays T E . Approximating this integral may require a high number of transmit delays.
• iToF systems measure an approximation of this first harmonic H 1, ⁇ (T D ).
• the approximation of the first harmonic H 1, ⁇ (T D ) is obtained by an N-point EDFT (Extended Discrete Fourier Transform), according to (Eq.10) with n being the N-point EDFT bin considered.
• n 1 in the remainder of this disclosure: (Eq.11)
• H 1, ⁇ ( ⁇ D ; t E ) is also referred to as IQ measurement (with I and Q the real resp. imaginary part of the first harmonic estimate).
• IQ measurement In order to stay close to iToF nomenclature, in the following H 1, ⁇ ( ⁇ D ; t E ) is denoted as “IQ measurement”. However, it is important to remember that an IQ measurement is an estimate of the first harmonic H 1, ⁇ (T D ) of the expected measurement (as function of transmit delay).
• the phase value ⁇ between the emitted and the received light is obtained as (Eq.12) with Im() and Re() being respectively the imaginary part and the real part operator, andarctan2 being the 4-quadrant inverse tangent function. Due to the statistical nature of the signal measurements ⁇ ( ⁇ E,n , ⁇ D ) , the IQ measurement H 1, ⁇ ( ⁇ D ; t E ) is a random variable with the following expected value (Eq.13) This expected value is here referred to as expected IQ measurement.
• the first harmonic will contain I and Q information as its real and imaginary part, respectively.
• the correlation waveform 502 (solid line) between reference (mix) signal and received signal is approximated by the acquired samples q 0 , ... , q 7 via the measurements obtained at the eight taps of the pixel.
• the samples q 0 , ... , q 7 are distributed equidistant in time and phase.
• the correlation waveform 501 (dashed line) between reference (mix) signal and emitted signal has no time of flight contribution.
• the time difference ⁇ D between the correlation waveforms 501 and 502 corresponds to the distance between the ToF camera and the object (see Eq. 2).
• the imaginary part Im(F(1)) of the IQ measurement F(1) is denoted as Q component and the real part Re(F(1)) of the IQ measurement F(1) is denoted as the I component of the IQ measurement F(1).
• the phase value ⁇ is obtained from the Q component and I component of the IQ measurement F(1) according to trigonometric principles.
• the corresponding depth value d for the pixel is determined as follows: with f M being the modulation frequency of the emitted signal and c being the speed of light.
• the amplitude value amp for the N-tap pixel which is an indicator of the signal strength and, implicitly, of the noise level, is determined as follows: (Eq. 19)
• scene descriptors may be determined from the measurements.
• the embodiments described below in more detail refer to a background light suppression algorithm.
• One basic means of achieving the background light suppression is to take the average value across so-called “empty” taps (i.e. taps not containing active light) and subtracting this from each tap value.
• the “empty” taps may still contain various types of noise (system noise, shot noise, etc.) which may influence the results of the Fourier transform and, implicitly, the first harmonic.
• system noise system noise, shot noise, etc.
• the remaining value corresponding to each tap may be set to zero (and thus eliminating measurement noise produce by ambient light).
• the timing diagram shows nine rows, where the first row shows the illumination signal ILLUM over time t, the second to eights rows show the control signals applied to transfer gates TG1,...,TG7 over time t, and the nineth row shows the control signal applied to the overflow gate OFG over time t.
• the illumination signal ILLUM (also called emitted signal) as emitted by an illumination unit of a iToF camera (110 in Fig. 1) is a pulse train which comprises pulses 301 with a predetermined pulse width t p .
• the pulse width may for example be 5ns and the duty cycle of the pulse train may for example be 1%.
• the emitted light pulse 301 is reflected by the scene and received as incident light INCIDENT by the N-tap pixel as a light pulse 303 with a certain time delay as compared to the emitted pulse 301.
• Ambient light 304 may for example be background light from the sun or a room illumination, or scattered light from an illumination unit (110 in Fig. 1).
• the ambient light 304 is considered here as constant over time with an amplitude smaller than the emitted light pulse.
• the first to seventh transfer gate control signals control the activation and deactivation of first to seventh transfer gates TG1,...,TG7, respectively, as described above with regard to Fig. 3.
• a transfer gate control signal has a zero amplitude the corresponding transfer gate is closed and when a transfer gate control signal has a non-zero amplitude the corresponding transfer gate is open.
• the control signals for the first to seventh transfer gates TG1,...,TG7 are respective pulse train signals.
• the respective pulses 302-1 to 302-7 have a predetermined pulse width which is equal to the pulse width t p of the illumination signal and the pulses 302-1 to 302-7 are all phase-shifted with respect to each other.
• the time shift between the control signals for transfer gates TG1,...,TG7 is equal to the activation pulse width t p and translates to a respective phase shift.
• the control signal for transfer gate TG1 is in synchronization with the illumination signal ILLU. Accordingly, its phase shift is zero.
• the pulse trains applied to transfer gates TG1,...,TG7 also have the same duty cycle.
• the transfer gates are opened for a predetermined time period one after the other. This can be seen as adding a concept a time-domain histogram to the classical 2-tap iToF.
• the acquisition of the signals is done in the time domain.
• the processing of the signals is however (see description below) done in the frequency domain (Fourier domain).
• the overflow gate OFG is activated.
• the overflow gate control signal OFG controls the activation and deactivation of the overflow gate OFG.
• the overflow gate control signal OFG is also a pulse train signal and it comprises a pulse 302- 8.
• the overflow gate OFG accumulates and collects all remaining and unnecessary charges generated due to received light by the pixel outside the time period t max (i.e. the time period where all transfer gates TG1, ...,TG7 are closed).
• the pulse width of pulse 302-8 may be much longer than the pulse width t p of the emitted light, for example 465ns, and the duty cycle of the overflow flow control signal may accordingly also be higher than the duty cycle of the transfer gate control signals, for example 90%.
• One sub-integration cycle is defined as lasting from the beginning of the activation of the first transfer gate TG1 until the deactivation of the overflow gate OFG.
• the sub- integration cycle may be repeated several times, for example 500, or 800 times, depending on the signal-to-noise (SNR) target. All sub-integration cycles together form one integration cycle. After one integration cycle is finished the read-out of the stored charges in the capacitors is read out (see also Fig. 12) as corresponding measurements q 0 ,..., q 6 (samples) for further processing and depth calculation (see Eq. 16 to 20 above).
• the ambient light 304 is received by the N-tap pixel is constant. Therefore, each tap of the N-tap pixel receives a respective part 304-1, ..., 304-7 of the ambient light 304 when transfer gate TG1,...,TG 7 is open.
• a first part 303-1 of the light pulse 303 related to the reflected light is received by the N-tap pixel while transfer gate TG3 is open.
• a second part 303-2 of the light pulse 303 is received by the N-tap pixel while transfer gate TG4 is open.
• the transfer gates TG1, TG2, TG5, TG6 and TG 7 are closed while the light pulse 303 is received by the N-tap pixel. Therefore, the corresponding taps 203-1, 203-2, 203-5, 203-6 and 203-7 do not collect any charges caused by light pulse 303. They receive only ambient light 304.
• the pulse width of the emitted light t p is smaller or equal to the transfer gate activation pulse width t p , there are at least N-2 taps which do not collect any charges generated by the emitted light pulse (so called active light) but only ambient light. These at least N-2 taps which do not collect any active light are called “empty” taps. The remaining taps (i.e. the taps containing charges generated by the emitted light pulse) are called “non-empty" taps. As long as the number of taps N is greater than or equal to three, there is always at least one empty tap.
• the ambient light contributions 304-1, ..., 304-7 received at the taps of the N-tap pixel may degenerate the quality of the depth measurement that is obtained based on the signals generated by the taps of the N-tap pixel.
• Taps may be classified as empty taps by determining which of the taps do provide a corresponding measurement q i which is below a predetermined threshold qthreshold- That is each tap whose corresponding measurement q i ⁇ q 0 , ... , q N-1 fulfils the condition q i ⁇ q threshold is classified as empty tap. Otherwise, a tap is classified as non-empty tap. Alternatively, empty taps may be identified if no peak is observed in the number of detected photons as compared to all tap information.
• ambient light corrected measurement values can be used to determine the Fourier transform of Eq. 16 as: (Eq. 24) and in accordance with Eq. 17 to obtain an ambient light corrected phase value according to: (Eq. 25).
• the measurements for the empty taps may be set to zero (Eq.26) and the resulting set of measurement values may be used in Eqs. 24 and 25 to compute the ambient light corrected phase value .
• the N-tap pixel that applies the ambient light suppression algorithm described above which is based on empty taps has an improved SNR as compared to a 2-tap pixel without ambient light suppression based on empty taps.
• the ambient light suppression may be based on techniques other than subtracting the average of the empty taps, for example techniques which involve passing the “empty” tap information through a Fourier transform.
• the N-tap solution combines the strengths of dTof and 2-tap iToF.
• the N-tap pixel preserves a good resolution (such as VGA). It may provide a mm-accuracy below Im. Between Im and 5m, the N-tap pixel solution described above has extended de-aliasing capabilities, for example via peak detection in the histogram of the signal for peak or range gating (see below) etc., while still working in single frequency. Moreover, the N-tap pixel has a better SNR than for example the 2-tap iToF due to ambient light rejection based on information from the empty taps (i.e. taps where active light is not detected). Above 5m, close-object scattering can be mitigated using the histogram operating principle in dToF.
• Fig. 8 shows an embodiment of a frame structure of a 7 -tap iToF pixel.
• a depth frame (here for example depth frame n+1) comprises a reset period, followed by an integration period, again followed by a read-out period.
• a sequence 1204 of light pulses are emitted by the illumination unit of the iToF imaging system.
• the illumination period may for example last 400 ⁇ s and may comprise 800 pulses of 5ns, which yields a duty cycle of 1%.
• the read-out period may last 5.3ms and the read-out may be performed MIPI standard compliant.
• Each pulse 1205 of the sequence 1204 of light pulses defines a sub-integration cycle.
• the first to the seventh transfer gates (TSG1, ..., TSG7 in Fig. 2) which correspond to taps 0 to 6 are active one after the other followed by an activation of the overflow gate OFG.
• One sub-integration cycle is defined as lasting from the beginning of the activation of the first transfer gate TG1, corresponding to tap 0, until the deactivation of the overflow gate OFG.
• the pulse width of the emitted light may be equal or smaller than the activation pulse width t p of the transfer gates.
• the overflow gate may be activated the remaining time of the sub-integration cycle which may be 465ns.
• the combined activation time t max of all transfer gates and corresponding taps defines the (radial) range in which the iToF camera can record objects.
• the emitted light pulse 1205 may have a time delay t b (phase shift) with respect to the activation of the first transfer gate, corresponding to tap 0.
• the number of sub -integration cycles per integration period may be determined according to the SNR target.
• iToF pixels for example a 2-tap pixel
• a single integration period may be enough for full depth acquisition. Therefore, less or even no motion blurring (due to a movement of a recorded object during read-out periods) may occur. And the N-tap pixel is therefore natively motion robust.
• the number of taps and the pulse width t p of the transfer gates determines the range of the iToF camera. Therefore, as compared to the 2-tap pixel with the same range, in the N-tap pixel the pulse width and/ or the emitted light pulse width t p may be shorter and therefore the duty cycle may be smaller. Therefore, with the N-tap pixel it is possible to get a higher peak optical power without increasing the size of the illumination unit (for example a VCSEL array), while avoiding saturation at close distances and while keeping eye safety. Therefore, a better SNR can be achieved.
• the illumination unit for example a VCSEL array
• N-tap pixels There are a series of system features unique to N-tap pixels. These additional system features unique to N-tap pixels include close-range gating (which may yield reduced multipath interference and scattering for close objects) and far-range gating (also covering suppression of echo from objects located further than a certain distance), dynamic range extension for short distances. Still further, depth dependent selective/adaptative filtering and binning are also enabled and self-calibration of a mismatch between taps (in terms of gain, phase, and correlation waveform) may be improved by using the scattering in the cover glass as a calibration setup.
• close-range gating which may yield reduced multipath interference and scattering for close objects
• far-range gating also covering suppression of echo from objects located further than a certain distance
• dynamic range extension for short distances.
• depth dependent selective/adaptative filtering and binning are also enabled and self-calibration of a mismatch between taps (in terms of gain, phase, and correlation waveform) may be improved by using the scattering in the cover glass
• Range gating is possible due to the capturing of depth information in time domain. During a gating period, all unwanted photoelectrons are drained by the overflow gate OFG. Range gating is performed on a radial detection range of the iToF camera. Both, close and far-ranges can be gated.
• Fig. 9 shows a timing diagram for a sub -integration cycle of a range gated iToF pixel with seven taps.
• a light pulse 1301 is emitted onto a scene.
• the emitted light pulse 1301 is received by the image sensor as close object rejection light pulse 1302, as far object rejection light pulse 1304 and as a usable range object reflection 1303.
• Close-range gating is achieved by advancing (delaying) the illumination pulse 1301 by the time t b from the sub -integration start moment, that is before the transfer gate TG1 corresponding to the tap 0 is activated.
• the overflow gate OFG is enabled such that it can drain out the photoelectrons which stem from close object rejection light pulse 1302 and ambient light reflected by objects in the close-range.
• the usable range d ur is defined by the pulse width t p of the transfer gate opening and the number of taps n, i.e. .
• Fig. 10 shows a radial detection range of an iToF camera with close-range and a far-range gated areas.
• An illumination unit 1401 is illuminating a scene along a radial direction 1403.
• An image sensor 1402 is capturing reflected light from the scene.
• the total operating time for the overflow gate OFG is (t e + t b ) and the corresponding close + far-range distance which is gated-out is determined as: .
• Some applications such as AR gaming may benefit from a selective range depth acquisition, for example an even further reduced usable range d ur .
• Occlusions appearing in the depth map will not be solved through selective depth ranging. That is, any objects in the usable range that are occluded by the eliminated foreground objects will not appear in the depth map even after depth ranging, as they are still behind the foreground objects in the real scene.
• a desired reduced usable range may be obtained by close-range gating either with less active taps (tap skipping) or by shortening the activation pulse width of the transfer gate and thereby the overall sub-integration time (“high precision”).
• Fig. 11 shows another embodiment of radial detection range of an iToF camera with close-range and far-range gated areas.
• An illumination unit 1501 is illuminating a scene along a radial direction 1503.
• An image sensor 1502 is capturing reflected light from the scene.
• the usable range d ur is set to be in a range between 3.0m and 5.25m.
• the far-range gated radial area 1505 begins at 5.25m.
• Fig. 12 shows two timing diagrams for a sub-integration cycle of a range gated iToF image sensor with seven taps using a tap skipping and a high precision method.
• a light pulse 1601 is emitted a predetermined time t b before the sub-integration time starts.
• the emitted light pulse 1601 is received by the image sensor as close object rejection light pulse 1602 and as a usable range object reflection 1603.
• a time span t e the overflow gate OFG is activated.
• tap skipping method Thereby, from 7 existing taps only 4 taps, i.e. taps 3 to 6, are active taps (“tap skipping method”).
• the reduced number of active taps allows for less power consumption for the MIX drivers and the depth processing.
• a light pulse 1607 is emitted at the same predetermined time t b before the sub-integration time starts, just as in the upper timing diagram 1600 with the reduced number of taps.
• Dynamic range extension is achieved by opening taps which are measuring close distances only for a part of the integration time, thus significantly extending the equivalent dynamic range by ensuring that theses taps do not get saturated. The process results in more data to describe objects in the usable range. Thus, more details can be seen for objects in the far-range while fine details remain clear for objects in the close-range while at the same time avoiding saturation.
• Fig. 13 shows four timing diagrams for four respective sub -integration cycles of an iToF pixel with seven taps using dynamic range extension.
• the first timing diagram 1701 for the first sub-integration cycle shows that initially all seven taps 0,...,7 are activated (one after the other) during the sub- integration cycle.
• a second timing diagram 1702 shows a new acquisition during a second sub- integration cycle, where tap 0 stays closed all the time and the overflow gate OFG is instead open.
• a third timing diagram 1703 shows a new acquisition during a third sub-integration cycle where taps 0 and 1 stay closed all the time and the overflow OFG is open instead.
• a fourth timing diagram 1704 shows a fourth sub -integration cycle where taps 0, 1 and 2 stay closed all the time and the overflow OFG is open instead.
• tap 0 being active only 1/16 of its normal active time of the total integration time T int
• tap 1 being active only 1/4 of its normal active time of the total integration time T int
• tap 2 and 3 being active only 1/2 of their normal active time of the total integration time T int
• only taps 4, 5 and 6 being active the full time of their normal active time of the total integration time T int .
• the tap signal values need to be normalized during depth processing. This above described case would imply x16 for tap 0, x4 for tap 1, x2 for taps 2 and 3.
• Fig. 14 shows a graph depicting the acquired current in the different taps of a 7-tap system depending on the distance of a detected object.
• the x-axis of the graph shows the distance in m and the y-axis shows the acquired charge in the taps in C (Coulomb).
• the seven bins 0 to 6 correspond to the seven taps 0 to 6.
• the dashed line 1801 shows the acquired charge in taps 0 to 6 during normal (non-range extended) operations.
• the dotted line 1802 shows the acquired charge in the taps 0 to 6 during range extended operations. In both cases the first tap 0 acquires the most charge and the acquired charge decreases from the first tap 0 to the last tap 6.
• the line 1803 shows the maximal tap saturation Q sat that still can be resolved by the pixel. Any acquired charge that is above the maximal tap saturation Q sat leads to objects not being detectable anymore.
• the acquired charge in normal (non-range extended) operations shown by the dashed line 1801 is above the maximal tap saturation Q sat in tap 0.
• the range extension mode shown as the dotted line 1802
• the acquired charge in tap 0 can be reduced below the maximal tap Q sat , so that tap saturation can be prevented.
• N-tap pixel Another system feature achievable with the N-tap pixel is the support for an implementation of range-specific binning. More specifically, since individual taps collect information from different ranges, certain taps can be binned independently from others. Depending on the range, different binning levels can be set for different (groups of) taps.
• Fig. 15 shows a timing diagram for a sub-integration cycle of an iToF image sensor with seven taps using range specific binning.
• An emitted light pulse 1901 is received as usable range object reflection 1902 by the image sensor.
• the taps 0 to 6 are activated one after the other.
• the first three taps 0 to 2 which acquire close-range objects are grouped as a first group 1906.
• the fourth and fifth tap 3 and 4 which acquire mid-range objects, are grouped as a second group 1907.
• the sixth and seventh tap 5 and 6, which acquire far-range objects, are grouped as a third group.
• the first group 1906 is set to acquire a full resolution depth frame by not applying any binning.
• the second group 1907 applies a 2x2 binning (meaning that 2x2 pixels are considered as one pixel) to obtain a higher SNR with a small impact on resolution.
• the third group 1908 applies 4x4 binning (to opt for background removal) .
• Fig. 16 shows a radial detection range of an iToF camera with range specific binning.
• An illumination unit 2001 is illuminating a scene along a radial direction 2003.
• An image sensor 2002 is capturing reflected light from the scene.
• the usable range d ur may be set to be in a range between 0.15m and 4.65m. Due to the applied range-specific binning, there are different areas where objects are detected with different resolutions within usable range d ur .
• a first area 2005 objects may be detected with a high resolution, for example VGA.
• In a second area 2006 objects may be detected with a reduced resolution, for example QVGA.
• In a third area 2007 objects may be detected with a low resolution, for example QQVGA.
• Fig. 17 schematically describes an embodiment of an iToF device that can implement the processes performing depth measurement (and other scene descriptors) with an N-tap pixel and perform ambient light suppression, close-range gating, dynamic range extension, and range-specific binning.
• the electronic device 2100 may further implement all other processes of a standard iToF/ spot ToF system.
• the electronic device 2100 comprises a CPU 1201 as processor.
• the electronic device 2100 further comprises an iToF sensor 2106 connected to the processor 2101.
• the processor 2101 may for example implement performing a depth measurement of a N-tap pixel, as described with regard to Figs. 7 to 11.
• the electronic device 2100 further comprises a user interface 2107 that is connected to the processor 2101.
• This user interface 2107 acts as a man-machine interface and enables a dialogue between an administrator and the electronic system. For example, an administrator may make configurations to the system using this user interface 2107.
• the electronic device 2100 further comprises a Bluetooth interface 2104, a WLAN interface 2105, and an Ethernet interface 2108. These units 2104, 2105 act as 1/ O interfaces for data communication with external devices. For example, video cameras with Ethernet, WLAN or Bluetooth connection may be coupled to the processor 2101 via these interfaces 2104, 2105, and 2108.
• the electronic device 2100 further comprises a data storage 2102, and a data memory 2103 (here a RAM).
• the data storage 2102 is arranged as a long-term storage, e.g. for storing parameters for one or more use-cases, for recording iToF sensor data obtained from the iToF sensor 2106 the like.
• the data memory 2103 is arranged to temporarily store or cache data or computer instructions for processing by the processor 2101.
• the pulse widths of the transfer gate control signals are described as being equal to the pulse width t p , of the emitted light pulse, in alternative embodiments the pulse widths of the transfer gate control signals may also be shorter or longer than the pulse width t p of the emitted light pulse. However, to achieve a high number of "empty" taps, it is preferred that the pulse widths of the transfer gate control signals are equal to or shorter than the pulse width t p of the emitted light pulse. In yet another embodiment all or some pulse widths of the transfer gate control signals may be different from each other.
• the phase shift between the first transfer gate control signal and the illumination signal ILLU may be non-zero.
• the emitted light pulse 301 and the pulse 302-1 of the first transfer gate transistor may have a time/ phase shift with respect to each other. This time span may be used for close-range gating (see Fig. 12).
• the duty cycle of the illumination signal ILLU may be shorter or longer than 5ns for example Ins or 10ns. In yet another example the duty cycle may be below or above 1% for example 0.3% or 0.5% or 5% or 10%. In anomer embodiment the illumination signal may be a sinusoid.
• Figs. 2, 3 and 7 relate to an N-tap pixel with seven taps with respective transfer gate plus one overflow gate. It should however be noted that the aspect of providing an overflow gate in the embodiments described above is an optional aspect. In alternative embodiments, for example an eighth tap with respective transfer gate may be foreseen instead of the overflow gate.
• An electronic device comprising circuitry configured to a determine, according to a Time-of- Flight, ToF, principle, a depth measurement (d) in the frequency domain based on three or more measurements (q 0 ,..., q N -1 ) which correspond to respective three or more taps of a pixel.
• ToF Time-of- Flight
• circuitry is configured to determine a discrete Fourier transform (F(1)) based on the three or more measurements (q 0 ,..., q N -1 ) related to the taps of the pixel, and to determine the depth measurement (d) based on the discrete Fourier transform (F(1)).
• circuitry is configured to determine a discrete Fourier transform (F(1)) according to where N is the total number of taps of the pixel and q i are the corresponding measurements provided by the respective taps of the pixel.
• circuitry is configured to determine the depth measurement (d) based on a real part (Re(F(1))) and an imaginary part (Im(F(1))) of a Fourier transform (F(1)).
• circuitry is configured to determine a depth measurement (d) as where Im(F(1)) and Re(F(1)) are respectively the imaginary part and the real part of a Fourier transform (F(1)) and arctan2 is the is the 4-quadrant inverse tangent function.
• circuitry further comprises three or more transfer gates (TG1,...,TG7) each corresponding to one of the three or more taps (208-

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Computer Networks & Wireless Communication (AREA)
• General Physics & Mathematics (AREA)
• Radar, Positioning & Navigation (AREA)
• Remote Sensing (AREA)
• Electromagnetism (AREA)
• Optical Radar Systems And Details Thereof (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=78829633

Family Applications (1)

Country Status (2)

Families Citing this family (1)

Family Cites Families (3)

• 2022
  • 2022-11-14 WO PCT/EP2022/081811 patent/WO2023104438A1/en not_active Ceased
  • 2022-11-14 EP EP22817252.4A patent/EP4445170A1/de not_active Withdrawn

Also Published As

Similar Documents

Legal Events