WO2022200132A1

WO2022200132A1 - Time-of-flight imaging system and method

Info

Publication number: WO2022200132A1
Application number: PCT/EP2022/056749
Authority: WO
Inventors: Morin DEHAN
Original assignee: Sony Semiconductor Solutions Corporation; Sony Depthsensing Solutions Sa/Nv
Priority date: 2021-03-26
Filing date: 2022-03-15
Publication date: 2022-09-29
Also published as: EP4314880A1

Abstract

A time-of-flight imaging system comprising an illumination unit configured to emit light on a scene, a sensor configured to capture a depth image of the illuminated scene, an optical feedback system configured to provide optical feedback concerning the light emitted by the illumination unit, a first tunable filter located before the sensor and a filter control unit configured to control the first tunable filter based on the optical feedback provided by the optical feedback.

Description

TIME-OF-FLIGHT IMAGING SYSTEM AND METHOD

TECHNICAL FIELD

The present disclosure generally pertains to the field of Time-of-Flight imaging, and in particular to systems and methods for Time-of-Flight image processing.

TECHNICAL BACKGROUND

A Time-of-Flight (ToF) camera is a range imaging camera system that determines the distance of objects by measuring the time of flight of a light signal between the camera and the object for each point of the image. Generally, a ToF camera has an illumination unit (a LED or VCSEL, Vertical- Cavity Surface-Emitting Laser) that illuminates a scene with modulated light. A pixel array (sensor) in the ToF camera collects the light reflected from the scene and measures phase-shift which provides information on the travelling time of the light, and hence information on distance.

The active illumination system of a ToF camera typically emits a predefined specific wavelength (a center operating wavelength) to illuminate the scene, e.g. 850 nm or 940 nm. However, this center operating wavelength of the active illumination system can vary due to manufacturing process variation, temperature, peak power, self-heating, and the like.

An optical band-pass filter is typically located in front of the sensor. This band-pass filter is configured to pass substantially only light having the same wavelength as the light emitted by the illumination unit. In this way, ambient light and noise from external light sources can be suppressed.

Although there exist techniques for filtering light in time-of-flight systems, it is generally desirable to provide improved filtering in time-of-flight systems and improved methods for controlling filtering of such time-of-flight systems.

SUMMARY

According to a first aspect the disclosure provides a time-of-flight imaging system comprising: an illumination unit configured to emit light on a scene; a sensor configured to capture a depth image of the illuminated scene; an optical feedback system configured to provide optical feedback concerning the light emitted by the illumination unit; a first tunable filter located before the sensor; and a filter control unit configured to control the first tunable filter based on the optical feedback provided by the optical feedback system.

According to a second aspect the disclosure provides a computer-implemented method comprising: driving an illumination unit to emit light on a scene; driving a sensor to capture a depth image of the illuminated scene; obtaining from an optical feedback system optical feedback concerning the light emitted by the illumination unit; and controlling a first tunable filter based on the optical feedback provided by the optical feedback system.

Further aspects are set forth in the dependent claims, the following description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments are explained by way of example with respect to the accompanying drawings, in which:

Fig. 1 schematically shows the basic operational principle of an indirect Time-of-Flight imaging system, which can be used for depth sensing or providing a distance measurement;

Fig. 2 schematically illustrates an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the sensor sub-system;

Fig. 3 schematically illustrates a cross-sectional side view of an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the sensor sub-system;

Fig. 4a shows in a first diagram a spectrum of light intensity emitted by an illumination unit as observed by a photodetector at one wavelength defined by a tunable filter arranged before the photodetector, and in a second diagram the tunable band of the tunable filter and its bandwidth;

Fig. 4b shows in a diagram the tunable band of the tunable filter and its bandwidth;

Fig. 4c shows in a diagram a sweep of the tunable filter, such as predefined voltage steps realized after applying a voltage to the tunable filter;

Fig. 4d shows in a diagram a spectrum of the light intensity (photo-current) according to a tunable voltage;

Fig. 4e shows in a diagram a spectrum of the light intensity detected by the photodetector over time, wherein the ToF imaging system shows no instability;

Fig. 4f shows in a diagram a spectrum of the light intensity detected by the photodetector over time, wherein the ToF imaging system shows instability;

Fig. 4g shows in a diagram a spectrum of light intensity detected by a photodetector having a tunable filter tuned at a specific center wavelength;

Fig. 4h shows in a diagram a spectrum of light intensity detected by a photodetector having a tunable filter tuned at a specific center wavelength, wherein the a drift of the laser to the right is detected; Fig. 4i shows in a diagram a spectrum of light intensity detected by a photodetector having a tunable filter tuned at a specific center wavelength, wherein a filter sweep, i.e. voltage step is applied to the tunable filter due to the drift of the laser;

Fig. 5 shows a flow diagram visualizing a method for performing initialization of an iToF imaging system;

Fig. 6 schematically illustrates an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located outside the sensor sub-system and outside the illumination sub-system;

Fig. 7 schematically illustrates an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the illumination sub-system;

Fig. 8 an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the illumination sub system and the tunable filter is located before the photodetector;

Figs. 9a shows in diagram a first spectrum of light intensity emitted by an illumination unit as observed at three wavelengths defined by tunable filters, the first spectrum being centered at a center operating wavelength;

Figs. 9b shows in diagram a second spectrum of light intensity emitted by an illumination unit as observed at three wavelengths defined by tunable filters, the second spectrum being shifted away from the center operating wavelength towards higher wavelengths;

Figs. 9c shows in diagram a second spectrum of light intensity emitted by an illumination unit as observed at three wavelengths defined by tunable filters, the second spectrum being shifted away from the center operating wavelength towards lower wavelengths;

Figs. 10a shows in diagram a first spectrum of light intensity emitted by an illumination unit as observed at three wavelengths defined by tunable filters, the first spectrum being centered at a center operating wavelength;

Figs. 10b shows in diagram a second spectrum of light intensity emitted by an illumination unit as observed at three wavelengths defined by tunable filters, the second spectrum being shifted away from the center operating wavelength towards higher wavelengths;

Figs. 10c shows in diagram a third spectrum of light intensity emitted by an illumination unit as observed at three wavelengths defined by tunable filters, the third spectrum being shifted away from the center operating wavelength towards lower wavelengths; Fig. 11 shows a flow diagram visualizing a method for adapting a bandpass characteristic of an optical tunable filter of an iToF imaging system; and

Fig. 12 schematically describes an embodiment of an iToF imaging device that can implement a depth sensing system having an illumination sub-system and a sensor sub-system.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 1 to Fig. 12, general explanations are made.

As mentioned in the outset, time-of-flight (ToF) devices (e.g. time-of-flight camera system) typically have an illumination unit, such as a light source, optical parts, such as a lens system and an optical band-pass filter, and an image sensor, etc.

Generally, a Time-of-Flight (ToF) camera is a range imaging camera system that determines the distance of objects in a scene by measuring the time of flight of a light signal between the camera and the object for each point of the image. Typically, a depth imaging system, such as ToF camera, has an illumination unit (a LED or VCSEL, Vertical-Cavity Surface-Emitting Laser) that illuminates the scene with modulated light. A pixel array in the ToF camera collects the light reflected from the scene and measures phase-shift which provides information on the travelling time of the light, and hence information on distance.

In indirect Time-of-Flight (iToF), three-dimensional (3D) images of a scene are captured. These images are also commonly referred to as “depth map”, or “depth image”, wherein each pixel of the image is attributed with a respective depth measurement. The depth image can be determined directly from a phase image, which is the collection of all phase delays determined in the pixels of the iToF camera.

The optical band-pass filter, typically, passes the light having the same wavelength as the light emitted by the illumination unit. Moreover, it is known that a ToF camera may use specific operating wavelength, however, center operating wavelength of an active illumination unit of the ToF camera, like lasers and LED may vary due to manufacturing process variation, temperature, peak power, self-heating, and the like. To accommodate to the variation of the ToF camera, the ToF sensor is combined with an optical filter which has a bandpass large enough to compensate for the variation of the active illumination system. Consequently, the ToF sensor is exposed to all ambient light illumination which can pass through this optical filter. From the discussion above, it can be taken that controlling the bandpass characteristics of an optical filter arranged before the ToF sensor may provide a suitable reduction of the amount of ambient light captured during the sensor operation, while keeping optimal sensitivity to the active signal.

Moreover, generally, it is known that illumination units have a temperature dependency, such that the wavelength of the emitted light may change (drift) with temperature changes. In such cases, it has been recognized that, when the wavelength drifts to other values, for example, the image sensor may have a different (reduced) quantum efficiency, the operating wavelength is in a different part of the solar spectrum, such that the intensity of the sun light (e.g. ambient light) at the operating wavelength may be increased, and, thus, the efficiency of the ToF system may be reduced, since the operating wavelength may change, while the filtering characteristic of the bandpass filter of the ToF sensor remains the same, such that the light at operating wavelength may be (partially) filtered out by the bandpass filter.

It has been recognized that at least one or more of the issues mentioned above, may be addressed by using a tunable filter arranged before the ToF sensor and before the illumination unit or before a photodetector comprised in the ToF system for adjusting a bandpass characteristic of the tunable filter, such that the tunable filter allows to pass the operating wavelength of a ToF device (system or the like).

Consequently, the embodiments described below in more detail describe a time-of-flight imaging system comprising an illumination unit configured to emit light on a scene, a sensor configured to capture a depth image of the illuminated scene, an optical feedback system configured to provide optical feedback concerning the light emitted by the illumination unit, a first tunable filter located before the sensor and a filter control unit configured to control the first tunable filter based on the optical feedback provided by the optical feedback system.

The time-of-flight imaging system may be a camera, may be included in another device, may be an apparatus or system or the like and the illumination unit is adapted to function as an illumination unit for a time-of-flight imaging system.

Generally, the illumination unit may be configured as a pulsed light source, a continuous light source or the like and it may be based on a vertical-cavity surface-emitting laser (VCSEL). The term “laser” is understood functionally in some embodiments and the VCSEL may include multiple vertical- cavity surface-emitting lasing elements which together form the laser, i.e. the VCSEL.

The sensor may be specifically designed for time-of-flight measurements and may be adapted to capture a depth image of the illuminated scene as described herein. The sensor may be configured for direct ToF, where the time delay of the photons emitted by the illumination unit and reflected by the scene are detected, it may be configured for indirect ToF, where basically a phase shift of the light emitted by the illumination unit and reflected by the scene is detected, etc. The sensor may be based on at least one of the following: CMOS (complementary metal-oxide semiconductor), CCD (charge coupled device), SPAD (single photon avalanche diode), CAPD (current assisted photonic demodulator) technology or the like.

The optical feedback system that provides optical feedback concerning the light emitted by the illumination unit may comprise a light coupler, a photodetector and at least one tunable filter, such as the first tunable filter.

The time-of-flight imaging system comprises a first tunable filter as described above, without limiting the present disclosure in that regard. Alternatively, the time-of-flight imaging system may comprise a second tunable filter, or more than two tunable filters or only one tunable filter, or the like.

The first tunable filter is located before the sensor, without limiting the present disclosure in that regard. The first tunable filter may be also located before the photodetector.

The tunable filter may be implemented using liquid crystal tunable filter technologies, for example comprising polymer based liquid crystal, or using mems based filter integrating tunable Fabry-Perot cavity technologies, or the like.

The filter control unit controls the first tunable filter based on the optical feedback provided by the optical feedback system and may further control the photodetector by controlling a monitoring of the photodetector. The filter control unit may be implemented in lower chip of ToF device (system or the like) or externally.

The filter control unit may determine a filter absorption coefficient from the light intensity detected by the photodetector(s) and may further determine a direction of the wavelength drift, that is a wavelength mismatch, of the emitted light, resulting to an immediate feedback and to a reliable filtering on the reflected light by the tunable filter arranged on the sensor.

The sensor may be combined with a temperature sensor configured to send a temperature information to the filter control unit. The sensor temperature may advantageously be combined with the light intensity detected by the photodetector(s).

Furthermore, the ToF imaging system may comprise a light intensity detection means configured to determine of an onset of the active light by the photodetector, e.g. a wavelength drift of the illumination unit during the illumination sequence due to temperature increase, and thus, to tune the bandpass wavelength, such as the center wavelength of the filter during a train of pulse of the active light. In some embodiments, the optical feedback system may comprise a photodetector configured to detect light emitted by the illumination unit. The photodetector may detect light emitted from the illumination unit and may also detect intensity of the reflected light as described herein. The photodetector may be light intensity detection means and may comprise a first portion with a filter and a second portion with no filter, such that a filter absorption coefficient may be derived from the light intensity received from both first and second filter portion. Based on the filter absorption coefficient, the filter control unit may adjust the wavelength absorbed by a tunable filter.

The time-of-flight imaging system may comprise one photodetector as described above, without limiting the present disclosure in that regard. Alternatively, the time-of-flight imaging system may comprise a plurality of photodetectors. For example, time-of-flight imaging system may comprise two photodetectors, three photodetectors, four photodetectors, or the like. Each one of these photodetectors may comprise a tunable filter or not.

In some embodiments, the optical feedback system may further comprise a light coupler configured to redirect a part of the light emitted by the illumination unit to the photodetector. In this way, the light coupler and the photodetector may form an optical feedback loop, such that an electrically tunable filter is used in the optical feedback loop. Using the optical feedback received from the feedback loop, the filter control unit may control the tunable filter(s).

In some embodiments, the filter control unit may perform adaptive filtering of the light reflected from the scene. The filter control unit by controlling the tunable filter may perform adaptive filtering of the light reflected from the scene. The adaptive filtering may be performed using an optical feedback acquired from an optical feedback loop in the ToF system. The filter control unit may perform adaptive filtering using a filtering technique which could be used for different illumination, which could be adapted to different operating wavelengths, which may be used for various types of applications, and for example, without requiring the manufacturing of new tunable filters, or the like. By performing adaptive filtering of the ambient light around the light of the illumination, eye safety may be improved by lowering the illumination required for a same level of Signal to Noise Ratio (SNR).

For example, the filter control unit may increase the intensity of light detected by the photodetector. The filter control unit may maximize the intensity of light detected by the photodetector. The filter control unit by controlling the tunable filter by performing adaptive filtering may maximize transmission for the ToF laser and may reduce the impact of ambient light in the ToF system (device or the like). The filter control unit may electrically control a narrow optical tunable filter by applying for example, a voltage, such that to dynamically tune the filter in order to maximize its transmission to the precise operating wavelength emitted by the illumination unit. In some embodiments, the filter control unit may adapt a filter characteristic of the first tunable filter. The filter characteristic may be the bandpass characteristic, such as a bandpass wavelength, such as a center wavelength, of the tunable filter. By adapting the filter characteristic of the first tunable filter, the first tunable filter may be adapted to the exact wavelength of the ToF illumination unit. The reference characteristic of the laser wavelength may be obtained through an optical feedback using the photodetector.

In some embodiments, the filter control unit may adapt a bandpass characteristic of the first tunable filter. The bandpass characteristic may be a bandpass wavelength, such as a center wavelength. By adapting the bandpass characteristic of the first tunable filter, the bandpass filter width may be reduced, and thus the impact of ambient light may be reduced. Additionally, by adapting the bandpass characteristic of the first tunable filter, the first tunable filter may be improved, e.g. optimized, and a drift of the (laser) operating wavelength of the ToF imaging system may be compensated. Still further, the bandpass characteristic of the first tunable filter may be continuously adapted to the (laser) operating wavelength of the ToF imaging system.

The tunable filter may have a bandwidth supporting a certain fluctuation of the active light, e.g. nominal wavelength +- 30nm, or the like. The bandwidth may be designed to take into account that fluctuation and may be wider than strictly needed. The effective width of the bandpass of the tunable filter may be reduced, and any drift of the illumination unit due to thermal, aging, or manufacturing may be compensated.

The bandpass characteristics may for example be controlled by the filter control unit by applying a voltage to the first tunable filter.

In some embodiments, the time-of-flight imaging system may further comprise a second tunable filter, and the filter control unit may further adapt a filter characteristic of the second tunable filter. Alternatively, the time-of-flight imaging system may comprise more than two tunable filters or only one tunable filter, or the like.

The first tunable filter may be located before the sensor, without limiting the present disclosure in that regard. The first tunable filter may be also located before the photodetector.

In some embodiments, the second tunable filter may be located before the photodetector, without limiting the present disclosure in that regard. In some embodiments, the second tunable filter may be located before the illumination unit.

In some embodiments, the filter control unit may further adapt the filter characteristic of the first tunable filter and the filter characteristic of the second tunable filter in the same way. The filter characteristic of the first tunable filter and the filter characteristic of the second tunable filter adapted by the filter control unit may be the same filter characteristic, for example, a bandpass characteristic.

In some embodiments, the time-of-flight system may comprise a sensor sub-system and an illumination sub-system. For example, the sensor sub-system and the illumination sub-system may be located on different circuit boards.

In some embodiments, the sensor sub-system may include the filter control unit, the first tunable filter and the photodetector, and the illumination sub-system may include the illumination unit, the light coupler and an illumination optics. The sensor sub-system may further include an isolation configured to optically isolate the photodetector from the sensor.

In some embodiments, the sensor sub-system may include the sensor and the first tunable filter, and the illumination sub-system may include the illumination unit, the light coupler and an illumination optics. The illumination optics may be optic lens, or the like. The time-of-flight system may further comprise the filter control unit, the photodetector, an isolation, and a second tunable filter located before the photodetector to perform adaptive filtering of the light reflected from the scene.

In some embodiments, the sensor sub-system may include the sensor and the first tunable filter, and the illumination sub-system may include the filter control unit, a second tunable filter, the photodetector, the illumination unit, the light coupler and an illumination optics. The second tunable filter may be located between the illumination unit and the light coupler.

In some embodiments, the sensor and the photodetector are part of a same chip, such a chip comprising a sensing portion and a photodetector portion. In some embodiments, the time-of-flight imaging system may comprise two, three, or four photodectors, the photodectors are also part of the same chip, defining a second, third and fourth photodetector portions.

In some embodiments, the sensor and the photodetector are distinct components.

In some embodiments, the optical feedback system of the time-of-flight imaging system may comprise a first and a second photodetector configured to detect light emitted by the illumination unit. For example, a tunable filter may be located before the first photodetector, the tunable filter being the same as the first tunable filter located before the sensor and no filter may be located before the second photodetector. Alternatively, a tunable filter may be located before the first photodetector, the tunable filter having a center wavelength higher than a center wavelength of the first tunable filter located before the sensor and a tunable filter may be located before the second photodetector, the tunable filter having a center wavelength lower than the center wavelength of the first tunable filter located before the sensor. The tunable filter may be a bandpass filter. In some embodiments, the optical feedback system of the time-of-flight imaging system may further comprise a third photodetector configured to detect light emitted by the illumination unit for example, a tunable filter may be located before the third photodetector, the tunable filter having a center wavelength lower or higher than the center wavelength of the first tunable filter located before the sensor. The tunable filter may be a bandpass filter.

In some embodiments, the optical feedback system of the time-of-flight imaging system may further comprise a third and a fourth photodetector configured to detect light emitted by the illumination unit. For example, a tunable filter may be located before the third photodetector, the tunable filter having a center wavelength higher than a center wavelength of the first tunable filter located before the sensor and a tunable filter may be located before the fourth photodetector, the tunable filter having a center wavelength lower than the center wavelength of the first tunable filter located before the sensor. The tunable filter may be a bandpass filter.

Some embodiments pertain to a computer-implemented method comprising driving an illumination unit to emit light on a scene, driving a sensor to capture a depth image of the illuminated scene, obtaining from an optical feedback system optical feedback concerning the light emitted by the illumination unit and controlling a first tunable filter based on the optical feedback provided by the optical feedback system.

Operational principle of an indirect Time-of-Flight imaging system (iToF)

Fig. 1 schematically shows the basic operational principle of an indirect Time-of-Flight imaging system which can be used for depth sensing. The iToF imaging system 1 includes an iToF camera, with an imaging sensor 2 having a matrix of pixels and a processor (CPU) 5. A scene 7 is actively illuminated with amplitude-modulated infrared light 8 at a predetermined wavelength using an illumination device 10, for instance with some light pulses of at least one predetermined modulation frequency generated by a timing generator 6. The amplitude-modulated infrared light 8 is reflected from objects within the scene 7. A lens 3 collects the reflected light 9 and forms an image of the objects within the scene 7 onto the imaging sensor 2. In indirect Time-of-Flight (iToF) the CPU 5 determines for each pixel a phase delay between the modulated light 8 and the reflected light 9.

This may be achieved by sampling a correlation wave between a demodulation signal 4 generated by a timing generator 6 and reflected light 9 that is captured by each respective pixel of the imaging sensor 2 and by sampling for each pixel a correlation wave between one or more shifted demodulation signals generated by the timing generator 6 (for example shifted about 0°, 90°, 180° and 210°) and the reflected light 9 that is captured by each respective pixel of the imaging sensor 2. This yields an in-phase component value (“I value”) for and quadrature component value (“Q- value”) for each pixel, so called I and Q values. Based on the I and Q values for each pixel a phase delay value f for each pixel may be determined as f = arctan which yields a phase image. The phase delay <pis proportional to the object’s distance modulo the wavelength of the modulation frequency. The depth image can thus be determined directly from the phase image. Still further, based on the I and Q values an amplitude value and a confidence conf value may be determined for each pixel as conf = |/| + |Q| which yields the amplitude image and the confidence image.

The illumination device 10 and the imaging sensor 2 described in the embodiment of Fig. 1 above, basically correspond to an illumination unit (see 27 in Figs. 2 to 5) and a sensor (see 23 in Figs. 2 to 5), respectively, described in the embodiments of Figs. 2 to 5 below.

Electrically tunable filter within an iToF system

Fig. 2 schematically illustrates an embodiment of a depth sensing system, such as the iToF imaging system of Fig. 1, having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the sensor sub-system.

A depth sensing system, such as the iToF imaging system of Fig. 1, has a sensor sub-system 20 and an illumination sub-system 21. An illumination unit 27 comprised in the illumination sub-system 21 illuminates a scene (see 7 in Fig. 1) and a sensor 23, such as a sensor array, comprised in the sensor sub-system 20 detects the light reflected from the scene.

A filter control unit 22, included in the sensor sub-system 20, controls a tunable filter 24 and a monitoring of a photodetector 25. Both the tunable filter 24 and the photodetector 25 are included in the sensor sub-system 20. The photodetector 25, which is for example, a light sensor, detects the intensity of the light emitted by the illumination unit 27. The photodetector 25 is located close to the illumination unit 27 and is not exposed to outdoor light. The photodetector 25 is also located close to the sensor 23, such that to realize an optical feedback. An isolation 26, which is included in the sensor sub-system 20, and which is located between the sensor 23 and the photodetector 25, optically isolates the sensor 23 from the photodetector 25. The tunable filter 24 is placed such as to cover the photodetector 25, the sensor 23 and the isolation 26. The tunable filter 24 is a tunable optical filter which avoids lateral light diffusion within the tunable filter 24 and is tuned to increase, e.g. to maximize, the light intensity detected by the photodetector 25 and by the sensor 23 based on a predefined filter characteristic. The predefined filter characteristic is for example a bandpass characteristic of the tunable filter 24. A light coupler 28, included in the illumination sub-system 21 and which forms an optical feedback system, redirects a part of the light, emitted from the illumination unit 27, to the photodetector 25, which is located behind the tunable filter 24. Illumination optics 29 are optical parts of the illumination sub-system 21, such as a lens system. In the embodiment of Fig. 2, the control of the tunable filter 24 is integrated in an optical feedback loop performed by the optical feedback system (here the light coupler 28), where the band pass of the tunable filter 24 is continuously adapted to the operating wavelength, e.g. laser wavelength, compensating a possible wavelength drift due to for example, the self-heating of the illumination system. The reference characteristic of the operating wavelength is obtained through the optical feedback using the photodetector 25. Such an optical feedback is realized by capturing parts of the light emitted by the illumination unit 27 with a light coupler 28 and feeding the captured light to the photodetector 25 next to the sensor 23 within the sensor sub-system 20, as described above. In this way, the filter control unit 22 adapts (e.g. updates) the filter characteristic (e.g. a bandpass characteristic) of the tunable filter 24 based on an optical feedback received from the optical feedback system (light coupler 28) to perform adaptive filtering of the light reflected from the scene. In other words, the filter control unit 22 adapts the filter characteristic (e.g. a bandpass characteristic) of the tunable filter 24 based on the light intensity detected by the photodetector, and therefore the tunable filter 24 is dynamically tuned to maximize the detected light intensity. In this way, the filter can adapt to wavelength shifts, i.e. sweep, (see Figs. 10a, b, and c described below in more detail) of the center operating wavelength due to manufacturing process variation, temperature, peak power, self-heating, and the like.

The tunable filter 24 is electrically controlled for example, by applying a predefined voltage to the tunable filter 24. The predefined voltage applied in order to electrically control the tunable filter 24 may have voltage range of a few millivolts to a few Volts. For example, the operating wavelength may be controlled by applying a low voltage e.g. 0.5-1 mV, without limiting the present embodiment in that regard. Alternatively, the applied voltage may be in a different range, e.g. 1-2 V. By applying these different voltages, the tunable filter 24 in the embodiment of Fig. 2 may for example have a bandwidth supporting a certain fluctuation of the active light, e.g. nominal wavelength +- 30 nm around a center operating wavelength of e.g. 850 nm or 940 nm.

In the embodiment of Fig. 2, by applying corrective voltages to the tunable filter on the sensor and on the photodetectors, the tunable filter is adapted such that its band pass allows to pass only the light at a wavelength being the same as the operating wavelength of the illumination unit, and thus, the detected light intensity is increased, e.g. maximized.

In the embodiment of Fig. 2, the tunable filter 24 may for example be implemented using a liquid crystal polymer-based technology, without limiting the present embodiment in that regard. Alternatively, the tunable filter 24 may be implemented using a MEMS based filter integrating tunable Fabry-Perot cavity, or the like.

The filter control unit 22 may be implemented in lower chip of the ToF device/ system or externally. Fig. 3 schematically shows a cross-sectional side view of an embodiment of the depth sensing system of Fig. 2, wherein the illumination unit 27, comprised in the illumination sub-system (see 20 in Fig. 2), illuminates a scene and the sensor 23 detects the light reflected from the scene. The light coupler 28, realized for example as a laminated prism, is inserted in the illumination sub-system (see 20 in Fig. 2) on top of the illumination unit 27, and forms an optical feedback system. A part of the emitted light is redirected to the photodetector 25 through the light coupler 28, i.e. the laminated prism. The redirected light is going through the tunable filter 24 before being detected by the photodetector 25. As described with regard to the embodiment of Fig. 2 above, the photodetector 25 is isolated from the sensor 23 by an isolation 26, such as a black wall. The illumination optics 29 are optical parts of the illumination sub-system 21, comprising an optical cavity, a lens system, or the like.

Fig. 4a shows in a diagram a spectrum of light intensity emitted by an illumination unit as observed by a photodetector at one wavelength defined by a tunable filter arranged before the photodetector. In the embodiment of Fig. 4a, the spectrum of the light intensity emitted by the illumination unit (see 27 in Figs. 2 and 3), is illustrated, as observed by the photodetector (see 25 in Figs. 2 and 3) at one wavelength defined by the tunable filter (see 24 in Figs. 2 and 3) being arranged before the photodetector (see 25 in Figs. 2 and 3). The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by photodetector (see 25 in Figs. 2 and 3), and the ordinate represents the intensity of the detected light.

Fig. 4b shows in a diagram the tunable band of the tunable filter and its bandwidth. The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by the photodetector (see 25 in Figs. 2 and 3), and the ordinate represents the filter transmission. The tunable filter and its wavelength range is represented by a rectangle, i.e. the tunable filter response at a given bias. The bandwidth of the tunable filter is represented by a small double arrow located inside the rectangle. The highest and lowest values of the bandpass wavelength, e.g. center wavelength, of the tunable filter are represented by two vertical dashed lines. The tunable band of the tunable filter is represented by a big double arrow located between the two vertical dashed lines. Here the center wavelength of the tunable filter is set to its start value, which is its lowest center wavelength. The nominal center wavelength is defined as a parameter of the system calibration.

Fig. 4c shows in a diagram a sweep of the tunable filter, such as predefined voltage steps realized after applying a voltage to the tunable filter. The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by the photodetector (see 25 in Figs. 2 and 3), and the ordinate represents the filter transmission. As described in Fig. 4b above, the tunable filter, and thus, its wavelength range is represented by a rectangle, the highest and lowest values of the bandpass wavelength of the tunable filter are represented by two vertical dashed lines, and the tunable band of the tunable filter is represented by a big double arrow located between the two vertical dashed lines. Here the filter center wavelength is tuned from its lowest values to its highest values and the filter center wavelength is modified by a predetermined amount. A sweep of the filter is realized, i.e. the filter is tuned by applying a voltage step to the tunable filter. The sweep steps of the center wavelength of the tunable filter are represented by the plurality of the rectangles.

In that way, auto-calibration (see 62 in Fig. 5) of the ToF imaging system is performed.

Fig. 4d shows in a diagram a spectrum of the light intensity (photo-current) according to a tunable voltage. This spectrum is the result of the auto-calibration as described in Fig. 4c above, wherein the diamonds represent the different intensity values detected at each wavelength step (sweep) of the center wavelength during the auto-calibration process. The highest value and lowest values of the center wavelength of the tunable filter are represented by vertical dashed lines, and the reference photo-current of the photodetector, i.e. the reference intensity I_ref, which is the reference value of the system, is represented by a horizontal dashed line. The value of the reference intensity I_ref may for example be the value of the lowest wavelength, the value of the highest wavelength, or the like. In the embodiment of Fig. 4d, the intensity of the highest value and lowest values of the center wavelength tend to the reference intensity I_ref. The reference value I_ref is used to calibrate the system, gives the noise of the system and is measured with the illumination switched off.

For example, from the measured photo-current, the filter control unit (see 22 in Fig. 2) identifies the center wavelength of the tunable filter where the maximum peak photo-current is observed. That center wavelength is then defined as the reference value for the sensing operation. The maximum peak photo-current, i.e. peak intensity is stored for example, in a storage unit in the ToF imaging system, together with the associated wavelength.

Fig. 4e shows in a diagram a spectrum of the light intensity detected by the photodetector over time, wherein the ToF imaging system shows no instability. The abscissa of the diagram represents the time, and the ordinate represents the light intensity detected by the photodetector over time. The photodetector (see 25 in Figs. 2 and 3) is sampled, for example, either continuously or at fixed interval depending on the configuration of the system. The horizontal dotted line represents the average intensity detected by the photodetector and the horizontal dashed line represents a predefined threshold. The horizontal solid line represents the reference intensity I_ref, (see Fig. 4d), here the reference intensity I_ref is an intensity close to 0. The predefined threshold is positioned between the average and the l_ref, when defining the threshold. The reference intensity I_ref allows to give a limit value for the predefined threshold. The threshold is higher than the reference intensity I_ref. For example, the threshold may be defined as threshold = 90% * average, without limiting the present embodiment in that regard.

In the embodiment of Fig. 4e, if the ToF imaging system shows no instability, the average current measured at the photodetector remains constant on average. This is shown by the continuous black line, which represents the detected intensity in a case where the ToF imaging system shows no instability.

Fig. 4f shows in a diagram a spectrum of the light intensity detected by the photodetector over time, wherein the ToF imaging system shows instability, i.e. drift. The abscissa of the diagram represents the time, and the ordinate represents the light intensity detected by the photodetector over time. The photodetector (see 25 in Figs. 2 and 3) is sampled, for example, either continuously or at fixed interval depending on the configuration of the system, as described in Fig. 4e above. The horizontal dotted line represents the average intensity detected by the photodetector and the horizontal dashed line represents a predefined threshold, as described in Fig. 4e above. The continuous black line represents the detected intensity. The diamond (a) represents a first intensity detected by the photodetector (see 25 in Figs. 2 and 3) at a specific point in time, as Fig. 4g shows. The diamond (b) represents a second intensity detected by the photodetector (see 25 in Figs. 2 and 3) at another specific point in time, as Fig. 4h shows. The value indicated by the diamond (b) is under the predefined threshold, and thus, the laser starts to drift (see Fig. 4h).

In the embodiment of Fig. 4f, the ToF imaging system shows instability, i.e. drift and the detected intensity crosses the predefined threshold. Once the predefined threshold is crossed, the auto- calibration process restarts between close range values, close to the original one (+ 1 bandwidth / bandpass). For example, in this case, where the ToF imaging system detects a drift of the laser, i.e. a drop in the detected intensity shifts by applying a sweep, i.e. a predefined voltage step to the tunable filter located before the photodetector (see 25 in Figs. 2 and 3) and the sensor (see 23 in Figs. 2 and 3). In this way, it is also possible to reduce the auto-calibration process, since the starting point of the following dynamic loop (see 63 in Fig. 5) may be the last intensity detected after applying the predefined voltage step to the tunable filter.

Fig. 4g shows in a diagram a spectrum of light intensity detected by a photodetector having a tunable filter tuned at a specific center wavelength, as indicated by diamond (a) in Fig. 4f. The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by photodetector (see 25 in Figs. 2 and 3), and the ordinate represents the intensity of the detected light. Fig. 4h shows in a diagram a spectrum of light intensity detected by a photodetector having a tunable filter tuned at a specific center wavelength, wherein a drift of the laser to the right is detected, as indicated by diamond (b) in Fig. 4f. The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by photodetector (see 25 in Figs. 2 and 3), and the ordinate represents the intensity of the detected light. As it can be taken from Figs. 4g and 4h, the laser starts to drift to the right, but the tunable filter remains stable.

Fig. 4i shows in a diagram a spectrum of light intensity detected by a photodetector having a tunable filter tuned at a specific center wavelength, wherein a filter sweep, i.e. voltage step is applied to the tunable filter due to the drift of the laser. The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by photodetector (see 25 in Figs. 2 and 3), and the ordinate represents the intensity of the detected light. The embodiment of Fig. 4i is the result of the filter sweep performed during dynamic loop process (see 63 in Fig. 5), wherein a corrective voltage is applied to the tunable filter (see 24 in Figs. 2 and 3).

Method for performing initialization of an iToF imaging system

Fig. 5 shows a flow diagram visualizing a method for performing initialization of an iToF imaging system, such as the depth sensing system of Fig. 2. At 60, setting a center wavelength of the tunable filter to a start value is performed (see Fig. 4a). The center wavelength of the tunable filter is first set to its start value, which could be, for example, its lowest or its highest center wavelength. The nominal center wavelength is defined as a parameter of the system calibration. At 61, measurement of a reference photo-current of the photodetector is performed, such as a reference intensity I_ref. The photo-current of the photodetector is measured as a reference value of the system ( I_ref ). The reference value I_ref is used to calibrate the system. A measured signal is only considered valid if the intensity is larger than the reference current, i.e. reference intensity l_ref. At 62, auto-calibration is performed. The auto-calibration is performed by starting the illumination process. During the illumination process the filter center wavelength is tuned from its lowest values to its highest values (see Fig. 4c). The center wavelength could be modified by a small or by a large amount. For each step, the photocurrent is measured. From the measured photo-current, the filter control unit 22 identifies the center wavelength of the tunable filter where the maximum peak photo-current is observed (see Fig. 4d). That center wavelength is then defined as the reference value for the sensing operation (see Fig. 4d). At 63, dynamic loop is performed, e.g. sampling of the photodetector over time. The starting point of the dynamic loop is the reference intensity I_ref. When the depth sensing system, here the ToF imaging system, is working, the filter control unit 22 samples the photodetector over time. The detection is performed either continuously or at fixed interval depending on the configuration of the system. If the system shows no instability (see Fig. 4e), the average current measured at the photodetector remains constant on average. Alternatively, if the laser starts to drift, the measured current crosses a predefined threshold (see Fig. 4f). Once the threshold is crossed, the auto-calibration process restarts between close range values, close to the original one (+ 1 bandwidth / bandpass). Still alternatively, if the laser temperature stability coefficient is known, the direction of the drift may be estimated, and the number of steps required to detect the peak configuration may be reduced.

Fig. 6 schematically illustrates an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located outside the sensor sub-system and outside the illumination sub-system.

A depth sensing system, such as the iToF imaging system of Fig. 1, has a sensor sub-system 30 and an illumination sub-system 31. The illumination unit 27, described under the embodiment of Fig. 2, comprised in the illumination sub-system 31 illuminates a scene (see 7 in Fig. 1) and the sensor 23, described under the embodiment of Fig. 2, is comprised in the sensor sub-system 30 and detects the light reflected from the scene.

The filter control unit 22, described under the embodiment of Fig. 2, is located outside the sensor sub-system 30 and outside the illumination sub-system 31. The filter control unit 22 controls a first tunable filter 32 and a second tunable filter 33, as well as a monitoring of the photodetector 25 (see Fig. 2). The first tunable filter 32 and the sensor 23 are located within the sensor sub-system 30, wherein the first tunable filter 32 is arranged before the sensor 23, such as to cover the sensor 23. The second tunable filter 33 and the photodetector 25 are located outside the sensor sub-system 30 and outside the illumination sub-system 31, wherein the second tunable filter 33 is arranged before the photodetector 25, such as to cover the photodetector 25. The isolation 26 (see Fig. 2), which is located outside the sensor sub-system 30 and also outside the illumination sub-system 31, is located between the sensor 23 and the photodetector 25, such that to optically isolate the sensor 23 from the photodetector 25. The photodetector 25 detects the intensity of the light emitted by the illumination unit 27. The photodetector 25 is located close to the illumination unit 27 and is not exposed to outdoor light. The photodetector 25 is also located close to the sensor 23, such that to realize an optical feedback.

The first tunable filter 32 and the second tunable filter 33 are tuned to increase, e.g. to maximize, the light intensity detected from the photodetector 25 and from the sensor 23 based on a predefined filter characteristic. The predefined filter characteristic is for example a bandpass characteristic of the tunable filters 32 and 33. A light coupler 28, included in the illumination sub-system 31 and which forms an optical feedback system, redirects a part of the light, emitted from the illumination unit 27, to the photodetector 25. Illumination optics 29 includes optical parts of the illumination sub-system 31, such as a lens system.

The filter control unit 22 adapts e.g. updates, the filter characteristic of the tunable filters 32 and 33 based on an optical feedback received from the optical feedback system to perform adaptive filtering of the light reflected from the scene. The tunable filters 32 and 33 are dynamically tuned to increase, e.g. to maximize, the intensity detected by the photodetector 25 and by the sensor 23. The tunable filters 32 and 33 are electrically controlled for example, by applying a predefined voltage to the tunable filters 32 and 33.

In the embodiment of Fig. 6, the control of the tunable filters 32 and 33 is integrated in an optical feedback loop performed by the optical feedback system (here the light coupler 28), where the band pass characteristic of the tunable filters 32 and 33 is continuously adapted to the operating wavelength, e.g. laser wavelength, compensating a possible wavelength drift due to for example, the self-heating of the illumination system.

Fig. 7 schematically illustrates an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the illumination sub-system.

A depth sensing system, such as the iToF imaging system of Fig. 1, has a sensor sub-system 40 and an illumination sub-system 41. The illumination unit 27 (see Fig. 2) comprised in the illumination sub-system 41 illuminates a scene (see 7 in Fig. 1) and the sensor 23 (see Fig. 2) comprised in the sensor sub-system 40 detects the light reflected from the scene.

The filter control unit 22 (see Fig. 2), which is included in the illumination sub-system 41, controls a first tunable filter 42 and a second tunable filter 43, as well as a monitoring of the photodetector 25 (see Fig. 2). The first tunable filter 42 is located within the sensor sub-system 40 and is arranged before the sensor 23, such that to cover the sensor 23. The second tunable filter 43 is located within the illumination sub-system 41 together with the illumination unit 27, the photodetector 25, the light coupler 28 (see Fig. 2) and the illumination optics 29 (see Fig. 2). The second tunable filter 43 is arranged between the illumination unit 27 and the light coupler 28, such that to cover the illumination unit 27. The photodetector 25, which is for example, a light sensor, detects the intensity of the light emitted by the illumination unit 27. The photodetector 25 is located close to the illumination unit 27 and is not exposed to outdoor light. The tunable filters 42 and 43 are tunable optical filters which avoid lateral light diffusion within the tunable filters 42 and 43 and are tuned to maximize the light intensity detected by the photodetector 25 and by the sensor 23 based on a predefined filter characteristic. The predefined filter characteristic is for example a bandpass characteristic of the tunable filters 42 and 43. The light coupler 28 forms an optical feedback system and redirects a part of the light, emitted by the illumination unit 27, to the photodetector 25, the illumination unit 27 being located behind the second tunable filter 43. The illumination optics 29 includes optical parts of the illumination sub-system 41, such as a lens system.

In the embodiment of Fig. 7, the filter control unit 22 adapts e.g. updates, the filter characteristic of the tunable filters 42 and 43 based on an optical feedback received from the optical feedback system to perform adaptive filtering of the light reflected from the scene. In other words, the filter control unit 22 adapts the bandpass characteristic of the tunable filters 42 and 43 based on the detected intensity, and therefore the tunable filters 42 and 43 are dynamically tuned to increase, e.g. to maximize, the detected light intensity. The tunable filters 42 and 43 are electrically controlled for example, by applying a predefined voltage to the tunable filters 42 and 43.

In the embodiment of Fig. 7, the control of the tunable filters 42 and 43 are integrated in an optical feedback loop performed by the optical feedback system (here the light coupler 28), where the band pass of the tunable filters 42 and 43 is continuously adapted to the operating wavelength, e.g. laser wavelength, compensating a possible wavelength drift due to for example, the self-heating of the illumination system.

Fig. 8 schematically illustrates an embodiment of a depth sensing system having a sensor sub-system and an illumination sub-system, wherein a photodetector and a tunable filter are located within the illumination sub-system and the tunable filter is located before the photodetector.

A depth sensing system, such as the iToF imaging system of Fig. 1, has a sensor sub-system 50 and an illumination sub-system 51. The illumination unit 27 (see Fig. 2) comprised in the illumination sub-system 51 illuminates a scene (see 7 in Fig. 1) and the sensor 23 (see Fig. 2) comprised in the sensor sub-system 50 detects the light reflected from the scene.

The filter control unit 22 (see Fig. 2), which is included in the illumination sub-system 51, controls a first tunable filter 52 and a second tunable filter 53, as well as a monitoring of the photodetector 25 (see Fig. 2). The first tunable filter 52 is located within the sensor sub-system 50 and is arranged before the sensor 23, such that to cover the sensor 23. The second tunable filter 53 is located within the illumination sub-system 51 together with the illumination unit 27, the photodetector 25, the light coupler 28 (see Fig. 2) and the illumination optics 29 (see Fig. 2). The second tunable filter 53 is arranged before the photodetector 25, such that to cover the photodetector 25. The photodetector 25, which is for example, a light sensor, detects the intensity of the light emitted by the illumination unit 27. The photodetector 25 is located close to the illumination unit 27 and is not exposed to outdoor light. The tunable filters 52 and 53 are tunable optical filters which avoid lateral light diffusion within the tunable filters 52 and 53 and are tuned to maximize the light intensity detected by the photodetector 25 and by the sensor based on a predefined filter characteristic. The predefined filter characteristic is for example a bandpass characteristic of the tunable filters 52 and 53. The light coupler 28 forms an optical feedback system and redirects a part of the light, emitted by the illumination unit 27, to the photodetector 25, the photodetector 25 being located behind the second tunable filter 53. The illumination optics 29 includes optical parts of the illumination sub system 51, such as a lens system.

In the embodiment of Fig. 8, the filter control unit 22 adapts e.g. updates, the filter characteristic of the tunable filters 52 and 53, such as the bandpass characteristic, based on the detected intensity to perform adaptive filtering of the light reflected from the scene. The tunable filters 52 and 53 are electrically controlled for example, by applying a predefined voltage to the tunable filters 52 and 53.

In the embodiment of Fig. 8, the control of the tunable filters 52 and 53 is integrated in an optical feedback loop performed by the optical feedback system (here the light coupler 28), where the band pass of the tunable filters 52 and 53 is continuously adapted to the operating wavelength, e.g. laser wavelength, compensating a possible wavelength drift due to for example, the self-heating of the illumination system.

In the embodiments of Figs. 2, 6 to 8 described above, the depth sensing system comprises one photodetector (see 25 in Figs. 2, 6 to 8), without limiting the present embodiments in that regard.

Alternatively, the depth sensing system may comprise two photodetectors. According to such embodiments, there are two light intensity detection portions.

A first exemplary embodiment of a depth sensing system with two photodetectors, namely photodetectors A, B is described in more detail. Photodetector A is a photodetector comprising no filter, photodetector B comprises a tunable filter with the same band pass wavelength (voltage parameter) as the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). The filter control unit (see 22 in Figs. 2, 6 to 8) may determine a filter absorption coefficient from the light intensity received from photodetectors A and B.

A second exemplary embodiment of a depth sensing system with two photodetectors, namely photodetector C comprises a tunable filter with a higher band pass wavelength than tunable filter on the sensor (see 23 in Figs. 2, 6 to 8) and photodetector D comprises a tunable filter with lower band pass wavelength than tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). The filter control unit (see 22 in Figs. 2, 6 to 8) may also determine a direction of the wavelength drift (wavelength mismatch) of the emitted light compared to the band pass wavelength of the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). When the direction of the wavelength drift is known, a feedback can immediately and with high reliability be provided to the tunable filter on the sensor. In the second exemplary embodiment of a depth sensing system with two photodetectors, the wavelength drift may for example be obtained from the intensities I_c, I_D of the two photodetectors C and D as follows.

Defining an asymmetry:

Looking at the asymmetry A_s allows to determine a voltage shift applied to the tunable filter of the sensor filter and on the filters C and D as follows:

where V is the voltage applied to a filter and AV is predefined voltage step, e.g. AV = 0.1F, which realizes a step of a filter sweep to larger, or, respectively lower wavelengths.

In this way, a feedback loop may be implanted.

The light intensity detected by photodetectors C and D, which comprise a tunable filter, is shown in Figs. 9a, 9b, 9c, wherein a spectrum of the light intensity obtained by the optical feedback system is represented as a function of the wavelength.

The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by photodetectors C and D, and the ordinate represents the intensity of the detected light. Each one of the photodetectors C and D comprises a tunable filter, as described above. The photodetectors C and D and the wavelength of the light detected by each one of them is represented by a rectangle, namely the wavelength of the light detected by photodetector C is represented by a rectangle having a dotted pattern, and the wavelength of the light detected by photodetector D is represented by a rectangle having a diagonal lined pattern.

In the embodiment of Fig. 9a, the maximum wavelength intensity (peak) is well aligned with the bandpass wavelength, such as the center wavelength of the filter on the sensor. Accordingly, no drift is observed by photodetectors C, and D and no corrective voltage needs to be applied to the tunable filter on the sensor.

In the embodiment of Fig. 9b, the maximum wavelength intensity is not aligned with pass band wavelength of the filter on the sensor, the light intensity detected by photodetector C is higher than the light intensity detected by photodetector D. Therefore, drift of the illumination unit (see 27 in Figs. 2, 6 to 8) toward the higher wavelength is observed by the photodetectors C and D, and a corrective parameter, such as a corrective voltage is applied to the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). Also, the corrective parameters, i.e. the corrective voltages to be applied to the tunable filter on photodetectors C and D, are adapted accordingly. By applying corrective voltages to the tunable filter on the sensor and on the photodetectors, the tunable filter is adapted such that its band pass allows to pass only the light at a wavelength being the same as the operating wavelength of the illumination unit, and thus, the detected intensity is increased, e.g. maximized.

In the embodiment of Fig. 9c, the maximum wavelength intensity is not aligned with pass band wavelength of the filter on the sensor, and the light intensity detected by photodetector C is lower than the light intensity detected by photodetector D, and thus, drift of the illumination unit (see 27 in Figs. 2, 6 to 8) toward the lower wavelength is observed by the photodetectors C and D, and a corrective voltage is applied to the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). Also, the corrective settings, i.e. the corrective voltages to be applied to the tunable filter on photodetectors B, C, and D, are adapted accordingly. By applying corrective voltages to the tunable filter on the sensor and on the photodetectors, the tunable filter is adapted such that its band pass allows to pass only the light at a wavelength being the same as the operating wavelength of the illumination unit, and thus, the detected intensity is increased, e.g. maximized.

Alternatively, the depth sensing system may comprise three photodetectors. According to such embodiments, there are three light intensity detection portions. Such depth sensing system may comprise a photodetector A comprising no filter, a photodetector B comprises a tunable filter with the same band pass wavelength (voltage parameter) as the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8), and one of a photodetector C with a higher band pass wavelength than tunable filter on the sensor (see 23 in Figs. 2, 6 to 8) or of a photodetector D comprises a tunable filter with lower band pass wavelength than tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). The filter control unit (see 22 in Figs. 2, 6 to 8) may determine a filter absorption coefficient from the light intensity received from photodetectors A and B. The filter control unit (see 22 in Figs. 2, 6 to 8) may also determine a direction of the wavelength drift (wavelength mismatch) from light intensity received from photodetectors B and C or B and D, of the emitted light compared to the band pass wavelength of the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). Other embodiments with three photodetectors are also possible.

Alternatively, the depth sensing system may comprise four photodetectors. According to such embodiments, there are four light intensity detection portions.

In the following, an embodiment of a depth sensing system with four photodetectors, namely photodetectors A, B, C, D, is described in more detail. Photodetector A is a photodetector comprising no filter, photodetector B comprises a tunable filter with the same band pass wavelength (voltage parameter) as the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8), photodetector C comprises a tunable filter with a higher band pass wavelength than photodetector B, and photodetector D comprises a tunable filter with lower band pass wavelength than photodetector B.

In this case, the filter control unit (see 22 in Figs. 2, 6 to 8) may determine a filter absorption coefficient from the light intensity received from photodetectors A and B, and may also determine a direction of the wavelength drift (wavelength mismatch) of the emitted light compared to the band pass wavelength of the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8) from the light intensity received from photodetectors C and D, as it is described below in more detail with regard to Figs. 10a, b, and c. When the direction of the wavelength drift is known, a feedback can immediately and with high reliability be provided to the tunable filter on the sensor.

The light intensity detected by photodetectors B, C and D, which comprise a tunable filter, is shown in Figs. 10a, 10b, 10c, wherein a spectrum of the light intensity obtained by the optical feedback system is represented as a function of the wavelength.

The abscissa of the diagram represents the wavelength of the light captured by the optical feedback system and detected by photodetectors B, C and D, and the ordinate represents the intensity of the detected light. Each one of the photodetectors B, C and D comprises a tunable filter, and the photodetector A comprises no filter. The photodetectors A, B, C and D and the wavelength of the light detected by each one of them is represented by a rectangle, namely the wavelength of the light detected by photodetector B is represented by a rectangle having a vertical lined pattern, the wavelength of the light detected by photodetector C is represented by a rectangle having a dotted pattern, wavelength of the light detected by photodetector D is represented by a rectangle having a diagonal lined pattern. Photodetector A without filter is represented by the rectangle having no pattern and located outside the diagram.

In the embodiment of Fig. 10a, the maximum wavelength intensity is well aligned with the bandpass wavelength, such as the center wavelength of photodetector B with the same band pass characteristic as the filter on the sensor. Accordingly, no drift is observed by photodetectors A, B, C and D, and no corrective voltage needs to be applied to the tunable filter on the sensor.

In the embodiment of Fig. 10b, the maximum wavelength intensity is not aligned with pass band wavelength of photodetector B, the light intensity detected by photodetector C is higher than the light intensity detected by photodetector D. Therefore, drift of the illumination unit (see 27 in Figs. 2, 6 to 8) toward the higher wavelength is observed by the photodetectors A, B, C and D, and a corrective parameter, such as a corrective voltage is applied to the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). Also, the corrective parameters, i.e. the corrective voltages to be applied to the tunable filter on photodetectors B, C, and D, are adapted accordingly. By applying corrective voltages to the tunable filter on the sensor and on the photodetectors, the tunable filter is adapted such that its band pass allows to pass only the light at a wavelength being the same as the operating wavelength of the illumination unit, and thus, the detected intensity is increased, e.g. maximized.

In the embodiment of Fig. 10c, the maximum wavelength intensity is not aligned with pass band wavelength of photodetector B, and the light intensity detected by photodetector C is lower than the light intensity detected by photodetector D, and thus, drift of the illumination unit (see 27 in Figs. 2, 6 to 8) toward the lower wavelength is observed by the photodetectors A, B, C and D, and a corrective voltage is applied to the tunable filter on the sensor (see 23 in Figs. 2, 6 to 8). Also, the corrective settings, i.e. the corrective voltages to be applied to the tunable filter on photodetectors B, C, and D, are adapted accordingly. By applying corrective voltages to the tunable filter on the sensor and on the photodetectors, the tunable filter is adapted such that its band pass allows to pass only the light at a wavelength being the same as the operating wavelength of the illumination unit, and thus, the detected intensity is increased, e.g. maximized.

The intensity detected by the photodetector B is used for example, in order to check if the intensity of B, i.e. I_B, is higher or lower than the intensity detected by the photodetector C, i.e. I_c, or by the photodetector D, i.e. I_D. This allows to check if a sweep, such as a predefined voltage step, is needed or not in order to find a band wavelength where I_B is significandy higher than I_c or I_D. The intensity I_B detected by the photodetector B improves robustness of the system in order to initialize the system.

The wavelength drift may for example be obtained from the three intensities I_B, I_c, I_D as follows:

Defining the asymmetry as in the embodiments of Figs. 6a, 6b, 6c:

Looking at the asymmetry A_s allows to determine a voltage shift applied to the tunable filter of the sensor filter and on the filters A, B and C as follows:

If |A_S| < e and I_B > average(/ , / ) = no sweep

If |A_S| < e and I_B £ average (7_C, /_D) = (arbitrary) sweep

If |A_S| > e => sweep wherein e is a small number (e.g. 0.01, 0.05. 0.1, or the like).

If the A_s is smaller or larger than e, and the intensity I_B detected by the photodetector B is larger than the average of the intensities I_c and I_D, then no sweep is necessary to be realized. No drift is observed by photodetectors C, and D and thus, no corrective voltage needs to be applied to the tunable filter on the sensor.

If the A_s is smaller or larger than e, and the intensity I_B detected by the photodetector B is lower than or equal to the average of the intensities I_c and I_D, then an arbitrary sweep is realized. An arbitrary sweep may be realized by applying to the filter either a voltage V = V — Dn or a voltage

V = V + V, where A is predefined voltage step, e.g. A = 0.1F, which realizes a step of an arbitrary filter sweep to lower, or, respectively larger wavelengths.

If the A_s is smaller than or equal to e, then a sweep is realized by applying to the filter a voltage F =

V — AV, where AF is predefined voltage step, e.g. AF = 0.1V, which realizes a step of a filter sweep to lower wavelengths. Alternatively, if the A_s is larger than or equal to e, then a sweep is realized by applying to the filter a voltage V = V + AF, where AF is predefined voltage step, e.g. AF = 0.1V, which realizes a step of a filter sweep to higher wavelengths.

A way to estimate which value I_B should have is to determine I_A X m , where m is the absorption coefficient of the tunable filter. According to yet another embodiment, this information can be taken into account when deciding whether or not a sweep is necessary or not.

In this way, a feedback loop may be implanted.

In the embodiments of Figs. 2 to 8 described above, the filter control unit 22 updates the predefined parameter of the tunable filters based on an optical feedback received from the optical feedback system and therefore the tunable filters are dynamically tuned to increase, e.g. to maximize the intensity detected by the photodetector 25 and by the sensor 23. Additionally, the sensor sub-system may comprise a temperature sensor configured to send a temperature information to the filter control unit 22. The sensor temperature may be combined with the light intensity from the photodetector 25 to maximize the intensity detected by the photodetector 25 and by the sensor 23.

Moreover, in the embodiments of Figs. 2 to 8 described above, the photodetector 25 that detects the light intensity may comprise a first portion with a filter and a second portion with no filter, such that a filter absorption coefficient can be derived from the light intensity received from both first and second filter portion. Based on the filter absorption coefficient, the filter control unit 22 may adjust the wavelength absorbed by the tunable filters.

Furthermore, in the embodiments of Figs. 2 to 8 described above, the photodetector 25 that detects the light intensity may additionally determine an onset of the active light detected by the photodetector 25, causing a wavelength drift of the illumination unit 27 during the illumination sequence due to temperature increase. Based on the information of the onset of active light, the filter control unit 22 may be configured to start modulating the band pass characteristic, such as a band pass wavelength, of the tunable filter arranged on the sensor 23, and thus, the filter control unit 22 may tune tunable filter by adapting the band pass wavelength of the tunable filter during a train of pulse of the active light.

Method for adapting the bandpass optical filter

Fig. 11 shows a flow diagram visualizing a method for adapting a bandpass characteristic of an optical tunable filter of an iToF imaging system, such as the depth sensing system of Figs. 2, 6 to 8. At 70, illumination of a scene is performed by the illumination unit (see 27 in Figs. 2, 6 to 8) of the illumination sub-system (see 21, 31, 41 and 51 in Figs. 2, 3, 6 to 8). At 71, light intensity, of the light reflected from the illuminated scene, is detected by the photodetector (see 25 in Figs. 2, 6 to 8), wherein the photodetector is covered by a tunable filter (see 24, 33, and 53 in Figs. 2, 3, and 8 respectively). At 72, the tunable filter being a band pass tunable filter is controlled based on the detected light intensity, such that to increase the light intensity detected by the photodetector.

The tunable filter is electrically controlled by an applied voltage such that the filter characteristics, e.g. bandpass characteristics, of the tunable filter are adapted based on the detected light intensity in order to maximize the light intensity detected by the photodetector. A tunable filter is also arranged before the sensor and its filter characteristics are adapted in the same way as the filter characteristics of the tunable filter arranged before the photodetector. The filter characteristics of the tunable filter arranged before the sensor and before the photodetector may be the same filter characteristics, such as bandpass characteristics, or the like.

Implementation

Fig. 12 schematically describes an embodiment of an iToF imaging device that can implement a depth sensing system having an illumination sub-system and a sensor sub-system, as described in Fig. 2, 3, 6 to 8 above. The electronic device 80 comprises a CPU 81 as processor. The electronic device 80 further comprises an iToF sensor 86 (e.g. sensor 23 of Figs. 2, 3, 6 to 8; or the illuminator unit 27 of Figs. 2, 3, 6 to 8) connected to the processor 81. The processor 81 may for example implement a process of dynamically tune a tunable filter based on continuously adapted filter parameters that are optimized for the operating wavelength of the electronic device as described in Fig. 11. The electronic device 80 further comprises a user interface 87 that is connected to the processor 81. This user interface 87 acts as a man-machine interface and enables a dialogue between an administrator and the electronic system. For example, an administrator may make configurations to the system using this user interface 87. The electronic device 80 further comprises a Bluetooth interface 84, a WLAN interface 85, and an Ethernet interface 88. These units 84, 85 act as 1/ O interfaces for data communication with external devices. For example, video cameras with Ethernet, WLAN or Bluetooth connection may be coupled to the processor 81 via these interfaces 84, 85, and 88. The electronic device 80 further comprises a data storage 82, and a data memory 83 (here a RAM). The data storage 82 is arranged as a long-term storage, e.g. for storing algorithm parameters for one or more use-cases, for recording iToF sensor data obtained from the iToF sensor 86, and the like. The data memory 83 is arranged to temporarily store or cache data or computer instructions for processing by the processor 81.

It should be noted that the description above is only an example configuration. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces, or the like.

It should also be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is, however, given for illustrative purposes only and should not be construed as binding.

It should also be noted that the division of the electronic device of Fig. 12 into units is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, at least parts of the circuitry could be implemented by a respectively programmed processor, field programmable gate array (FPGA), dedicated circuits, and the like.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example, on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

Note that the present technology can also be configured as described below:

(1) A time-of-flight imaging system comprising: an illumination unit (27) configured to emit light on a scene; a sensor (23) configured to capture a depth image of the illuminated scene; an optical feedback system (25, 28, 24, 33, 43, 53) configured to provide optical feedback concerning the light emitted by the illumination unit (27), a first tunable filter (24; 32; 42; 52) located before the sensor (23); and a filter control unit (22) configured to control the first tunable filter (24; 32; 42; 52) based on the optical feedback provided by the optical feedback system (25, 28, 24, 33, 43, 53) .

(2) The time-of-flight imaging system of (1), wherein the optical feedback system (25, 28, 24, 33, 43, 53) comprises a photodetector (25; A, B, C, D) configured to detect light emitted by the illumination unit (27).

(3) The time-of-flight imaging system of (2), wherein the optical feedback system (25, 28, 24, 33, 43, 53) further comprises a light coupler (28) configured to redirect a part of the light emitted by the illumination unit (27) to the photodetector (25; A, B, C, D).

(4) The time-of-flight imaging system of anyone of (1) to (3), wherein the filter control unit (22) is configured to perform adaptive filtering of the light reflected from the scene.

(5) The time-of-flight imaging system of anyone of (1) to (4), wherein the filter control unit (22) is configured to adapt a filter characteristic of the first tunable filter (24; 32; 42; 52).

(6) The time-of-flight imaging system of anyone of (1) to (5), wherein the filter control unit (22) is configured to adapt a bandpass characteristic of the first tunable filter (24; 32; 42; 52).

(7) The time-of-flight imaging system of (2), wherein the time-of-flight imaging system further comprises a second tunable filter (24; 33; 43; 53), and wherein the filter control unit (22) is further configured to adapt a filter characteristic of the second tunable filter (24; 33; 43; 53).

(8) The time-of-flight imaging system of (7), wherein the second tunable filter (24; 33; 43; 53) is located before the photodetector (25).

(9) The time-of-flight imaging system of (7), wherein the second tunable filter (24; 33; 43; 53) is located before the illumination unit (27).

(10) The time-of-flight imaging system of (7), wherein the filter control unit (22) is further configured to adapt the filter characteristic of the first tunable filter (24; 33; 43; 53) and the filter characteristic of the second tunable filter (24; 33; 43; 53) in the same way.

(11) The time-of-flight imaging system of (2), wherein the time-of-flight system comprises a sensor sub-system (20; 30; 40; 50) and an illumination sub-system (21; 31; 41; 51).

(12) The time-of-flight imaging system of (11), wherein the sensor sub-system (20) includes the filter control unit (22), the first tunable filter (24) and the photodetector (25), and the illumination sub-system (21) includes the illumination unit (27), the light coupler (28) and an illumination optics (29).

(13) The time-of-flight imaging system of (12), wherein the sensor sub-system (20) further includes an isolation (26) configured to optically isolate the photodetector (25) from the sensor (23).

(14) The time-of-flight imaging system of (11), wherein the sensor sub-system (30) includes the sensor (23) and the first tunable filter (32), and the illumination sub-system (31) includes the illumination unit (27), the light coupler (28) and an illumination optics (29). (15) The time-of- flight imaging system of (14), wherein the time-of- flight system further comprises the filter control unit (22), the photodetector (25), an isolation (26), and a second tunable filter (33) located before the photodetector (25) to perform adaptive filtering of the light reflected from the scene.

(16) The time-of- flight imaging system of (11), wherein the sensor sub-system (40; 50) includes the sensor (23) and the first tunable filter (42; 52), and the illumination sub-system (41; 51) includes the filter control unit (22), a second tunable filter (43; 53), the photodetector (25), the illumination unit (27), the light coupler (28) and an illumination optics (29).

(17) The time-of- flight imaging system of (16), wherein the second tunable filter (43) is located between the illumination unit (27) and the light coupler (28).

(18) The time-of- flight imaging system of anyone of (1) to (17), wherein the optical feedback system (25, 28, 24, 33, 43, 53) comprises a first and a second photodetector (A, B, C, D) configured to detect light emitted by the illumination unit (27).

(19) The time-of- flight imaging system of (18), wherein a tunable filter is located before the first photodetector (B), the tunable filter being the same as the first tunable filter (24; 32; 42; 52) located before the sensor (23) and no filter is located before the second photodetector (A).

(20) The time-of- flight imaging system of (18), wherein a tunable filter is located before the first photodetector (C), the tunable filter having a center wavelength higher than a center wavelength of the first tunable filter (24; 32; 42; 52) located before the sensor (23) and a tunable filter is located before the second photodetector (D), the tunable filter having a center wavelength lower than the center wavelength of the first tunable filter (24; 32; 42; 52) located before the sensor (23).

(21) The time-of-flight imaging system of (19), wherein the optical feedback system (25, 28, 24, 33, 43, 53) further comprises a third photodetector (C; D) configured to detect light emitted by the illumination unit (27).

(22) The time-of-flight imaging system of (21), wherein a tunable filter is located before the third photodetector (C; D), the tunable filter having a center wavelength lower or higher than the center wavelength of the first tunable filter (24; 32; 42; 52) located before the sensor (23).

(23) The time-of-flight imaging system of (19), wherein the optical feedback system (25, 28, 24, 33, 43, 53) further comprises a third and a fourth photodetector (C, D) configured to detect light emitted by the illumination unit (27).

(24) The time-of-flight imaging system of (23), wherein a tunable filter is located before the third photodetector (C), the tunable filter having a center wavelength higher than a center wavelength of the first tunable filter (24; 32; 42; 52) located before the sensor (23) and a tunable filter is located before the fourth photodetector (D), the tunable filter having a center wavelength lower than the center wavelength of the first tunable filter (24; 32; 42; 52) located before the sensor (23). (25) A computer-implemented method comprising: driving an illumination unit (27) to emit light on a scene; driving a sensor (23) to capture a depth image of the illuminated scene; obtaining from an optical feedback system (25, 28, 24, 33, 43, 53) optical feedback concerning the light emitted by the illumination unit (27); and controlling a first tunable filter (24; 32; 42; 52) based on the optical feedback provided by the optical feedback system (25, 28, 24, 33, 43, 53).

Claims

1. A time-of-flight imaging system comprising: an illumination unit configured to emit light on a scene; a sensor configured to capture a depth image of the illuminated scene; an optical feedback system configured to provide optical feedback concerning the light emitted by the illumination unit, a first tunable filter located before the sensor; and a filter control unit configured to control the first tunable filter based on the optical feedback provided by the optical feedback system.

2. The time-of-flight imaging system according to claim 1, wherein the optical feedback system comprises a photodetector configured to detect light emitted by the illumination unit.

3. The time-of-flight imaging system according to claim 2, wherein the optical feedback system further comprises a light coupler configured to redirect a part of the light emitted by the illumination unit to the photodetector.

4. The time-of-flight imaging system according to claim 1, wherein the filter control unit is configured to perform adaptive filtering of the light reflected from the scene.

5. The time-of-flight imaging system according to claim 1, wherein the filter control unit is configured to adapt a filter characteristic of the first tunable filter.

6. The time-of-flight imaging system according to claim 1, wherein the filter control unit is configured to adapt a bandpass characteristic of the first tunable filter.

7. The time-of-flight imaging system according to claim 2, wherein the time-of-flight imaging system further comprises a second tunable filter, and wherein the filter control unit is further configured to adapt a filter characteristic of the second tunable filter.

8. The time-of-flight imaging system according to claim 7, wherein the second tunable filter is located before the photodetector.

9. The time-of-flight imaging system according to claim 7, wherein the second tunable filter is located before the illumination unit.

10. The time-of-flight imaging system according to claim 7, wherein the filter control unit is further configured to adapt the filter characteristic of the first tunable filter and the filter characteristic of the second tunable filter in the same way.

11. The time-of-flight imaging system according to claim 2, wherein the time-of-flight system comprises a sensor sub-system and an illumination sub-system.

12. The time-of-flight system according to claim 11, wherein the sensor sub-system includes the filter control unit, the first tunable filter and the photo detector, and the illumination sub-system includes the illumination unit, the light coupler and an illumination optics.

13. The time-of-flight imaging system according to claim 12, wherein the sensor sub-system further includes an isolation configured to optically isolate the photodetector from the sensor.

14. The time-of-flight imaging system according to claim 11, wherein the sensor sub-system includes the sensor and the first tunable filter, and the illumination sub-system includes the illumination unit, the light coupler and an illumination optics.

15. The time-of-flight imaging system according to claim 14, wherein the time-of-flight system further comprises the filter control unit, the photodetector, an isolation, and a second tunable filter located before the photodetector to perform adaptive filtering of the light reflected from the scene.

16. The time-of-flight imaging system according to claim 11, wherein the sensor sub-system includes the sensor and the first tunable filter, and the illumination sub-system includes the filter control unit, a second tunable filter, the photodetector, the illumination unit, the light coupler and an illumination optics.

17. The time-of-flight imaging system according to claim 16, wherein the second tunable filter is located between the illumination unit and the light coupler.

18. The time-of-flight imaging system according to claim 1, wherein the optical feedback system comprises a first and a second photodetector configured to detect light emitted by the illumination unit.

19. The time-of-flight imaging system according to claim 18, wherein a tunable filter is located before the first photodetector, the tunable filter being the same as the first tunable filter located before the sensor and no filter is located before the second photodetector.

20. The time-of-flight imaging system according to claim 18, wherein a tunable filter is located before the first photodetector, the tunable filter having a center wavelength higher than a center wavelength of the first tunable filter located before the sensor and a tunable filter is located before the second photodetector, the tunable filter having a center wavelength lower than the center wavelength of the first tunable filter located before the sensor.

21. The time-of-flight imaging system according to claim 19, wherein the optical feedback system further comprises a third photodetector configured to detect light emitted by the illumination unit.

22. The time-of-flight imaging system according to claim 21, wherein a tunable filter is located before the third photodetector, the tunable filter having a center wavelength lower or higher than the center wavelength of the first tunable filter located before the sensor.

23. The time-of-flight imaging system according to claim 19, wherein the optical feedback system further comprises a third and a fourth photodetector configured to detect light emitted by the illumination unit.

24. The time-of-flight imaging system according to claim 23, wherein a tunable filter is located before the third photodetector, the tunable filter having a center wavelength higher than a center wavelength of the first tunable filter located before the sensor and a tunable filter is located before the fourth photodetector, the tunable filter having a center wavelength lower than the center wavelength of the first tunable filter located before the sensor.

25. A computer-implemented method comprising: driving an illumination unit to emit light on a scene; driving a sensor to capture a depth image of the illuminated scene; obtaining from an optical feedback system optical feedback concerning the light emitted by the illumination unit; and controlling a first tunable filter based on the optical feedback provided by the optical feedback system.