EP4690818A1

EP4690818A1 - Image sensor and sensing device for extended reality (xr) applications

Info

Publication number: EP4690818A1
Application number: EP24714888.5A
Authority: EP
Inventors: Matteo Perenzoni; David Stoppa; Daniel Van Nieuwenhove; Christian BRAENDLI; Ariel Ben Shem; Lavi SEMEL; Keiji Yamaguchi; Tetsuo Nomoto; Daniele PERENZONI; Daniele Giorgetti; Johannes Solhusvik
Original assignee: Sony Advanced Visual Sensing AG; Sony Semiconductor Solutions Corp
Current assignee: Sony Advanced Visual Sensing AG; Sony Semiconductor Solutions Corp
Priority date: 2023-03-31
Filing date: 2024-03-21
Publication date: 2026-02-11
Also published as: JP2026511740A; WO2024200217A1; KR20250165431A; CN120982108A

Abstract

A sensor module comprising circuitry configured to provide a dual sensing layer (400), the dual sensing layer (400) providing self-registered RGB imaging capabilities and SPAD sensing capabilities, wherein the self-registration is realized by a stacked implementation of sensing layers.

Description

IMAGE SENSOR AND SENSING DEVICE FOR EXTENDED REALITY (XR) APPLICATIONS

The present disclosure generally pertains to the technical field of extended reality (XR) devices, methods, and systems.

TECHNICAL BACKGROUND

Extended reality (XR) describes the perception of enhanced sensory, predominantly visual impressions within a multidimensional environment consisting of a combination of real and computer-generated virtual elements. It is typically mediated to a use by mobile devices such as smartglasses, smartwatches, and smartphones, and tablet computers. The term extended reality is a common denominator for virtual (VR), augmented (AR), or mixed reality (MR).

Mobile devices, such as smartglasses, smartwatches, and smartphones, need to gain a high level of awareness of the surrounding environment in order to let the user interact in extended reality applications. This implies the use of a relevant number of imaging and sensing devices able to gather different aspects of the scene, like images suitable for machine vision (i.e. deep neural network processing, automated tasks), images suitable for video streaming and photoshooting, depth maps for scene reconstruction, plus other inputs such as acceleration from an inertial unit, temperature and ambient light sensing, etc. State of the art commercial devices use multiple sensors (e.g. world facing cameras in smartphones) and complex processing to realize registration and image fusion.

Although there exist techniques for imaging sensors for XR devices, it is generally desirable to enhance these techniques.

SUMMARY

According to an aspect, the disclosure provides a sensor module comprising circuitry configured to provide a dual sensing layer, the dual sensing layer providing self-registered RGB imaging capabilities and Single-Photon Avalanche Diode (SPAD) sensing capabilities, wherein the self-reg- istration is realized by a stacked implementation of sensing layers.

According to a further aspect, the disclosure provides an electronic device comprising the above sensor module and an application processor that is communicatively coupled to the sensor module.

Further aspects are set forth in the dependent claims, the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to the accompanying drawings, in which:

Fig. 1 shows a smartglasses device with an all-in-one sensor module according to an embodiment of the disclosure;

Fig. 2 shows in more detail an example architecture of an XR system of a smartglasses device, the XR system comprising an all-in-one sensor module;

Fig. 3 shows, in a cross-sectional schematic view, an example of a self-registered and single-aperture RGB+SPAD sensor (110 in Fig. 2) of an all-in-one sensor module;

Fig. 4 schematically shows an exploded view of a self-registered and single-aperture RGB+SPAD sensor;

Fig. 5 shows, in a more detailed cross-sectional view, an example of a self-registered and singleaperture RGB+SPAD sensor as an all-in-one sensor module;

Fig. 6 schematically shows an exemplifying timing diagram of the operation of the all-in-one sensor module for a direct time-of-flight operating mode;

Fig. 7a shows an ultralow power always on (internal loop) operating mode of the all-in-one sensor module;

Fig. 7b shows a low power single sensor XR (external loop) operation mode of the all-in-one sensor module;

Fig. 8 shows an example of a system typical operation of the all-in-one sensor module over time. The diagram shows the power consumption of the all-in-one sensor module over time;

Fig. 9a schematically describes a “full mapping” imaging/sensing state of the all-in-one sensor module;

Fig. 9b schematically describes an “anchoring” imaging/sensing state of the all-in-one sensor module; and

Fig. 9c schematically describes a “smart update” imaging/sensing state of the all-in-one sensor module. DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 1, general explanations are made.

The embodiments provide a sensor module comprising circuitry configured to provide a dual sensing layer, the dual sensing layer providing self-registered RGB imaging capabilities and SPAD sensing capabilities, wherein the self-registration is realized by a stacked implementation of sensing layers. Here, RGB sensing refers to imaging of (color) intensity by using photodiodes, thus getting in each pixel a measure of the light intensity over a certain period of time. SPAD sensing, to the contrary, refers to single-photon imaging, thus getting in each pixel the information related to the single photon arrival time and rate.

The dual sensing layer may be configured to provide a self-registered and single-aperture RGB+SPAD sensing.

Self-registration may for example mean that the stacked implementation is coaxial and spatially coherent. In this way, the stacked implementation of sensing layers may provide a single-aperture sensor. The self-registration may for example provide for coaxial image planes. The term self-registered may refer to both spatial and temporal alignment between RGB and SPAD.

The circuitry may for example be configured according to a triple wafer stacking technique. The third wafer may for example host the readout and processing electronics supporting RGB and SPAD wafers that are located above the third waver. The embodiments are, however, not restricted to a triple wafer stacking. They may also use front-end electronics that are hosted on the RGB and SPAD wafers.

The SPAD sensing may for example provide several modes of operation, from photon counting (PC) to time-of-flight (ToF) range imaging. The sensor module may for example allow for multiple RGB & EVS operating modes. It may further provide fully programmable macropixels of a SPAD array. Macropixels may for example be configured as a small SPAD array (e.g. 2x2, 3x3, 4x4 pixels) connected to dedicated processing electronics.

The dual sensing layer may for example comprise a photodiode layer. In this way, the sensor module may for example provide RGB imaging/sensing.

The dual sensing layer may further comprise a SPAD layer. The SPAD layer may for example be Silicon (VIS-NIR) or Germanium (SWIR) -based. Other implementations are possible for SWIR detection e.g. InGaAs/InP, or the like. Also, in some embodiments, visible light could be sensed by the SPAD layer. The SPAD layer may provide SPAD range imaging capabilities. The SPAD layer may for example enable applications such as iToF/dToF and/or photon counting (PC).

An example implementation may for example foresee an RGB wafer being on top and targeting visible light (spectrum) and a SPAD implemented below and devoted to NIR or SWIR spectrum.

The dual sensing layer may further comprise a visible-cut filter for residual visible light removal. The visible cut may for example let pass either NIR or SWIR, or both.

The circuitry may further comprise a dual band pass filter arranged above the dual sensing layer.

The circuitry may further comprise a readout layer arranged below the dual sensing layer. The readout layer 407 may for example be realized as a CMOS readout and processing layer. The connectivity for the SPADs may for example be realized as a high density wafer-to-wafer connectivity (e.g. by hybrid bonding).

The circuitry of the sensor module may further be configured to provide EVS sensing capabilities. For example, in a self-registered and single-aperture RGB+SPAD sensor, Event-based Vision Sensor (EVS) functionality may be added at RGB wafer level or into readout wafer level. The EVS sensing may for example provide an event-based mode for low-latency and low-power operation.

The circuitry of the sensor module may further provide an all-in-one imaging and sensing mode for mobile extended reality applications.

The circuitry may further provide a programmable laser transmitter. With the help of a flexibly programmable illuminator, the sensor module may satisfy all typical needs for extended reality application (XR), e.g. capturing of variable resolution images and reconfigurable depth maps. The programmable laser transmitter may for example comprise a laser driver and a VCSEL array. The programmable laser transmitter may for example be used for dynamic arbitrary patterns generation.

The circuitry of the sensor module may further provide a programmable on-chip control for local autonomous closed loop control. The on-chip control may for example be provided by a programmable on-chip control unit, e.g. a CPU or NPU which is able to selectively optimize the module operation. The on-chip control unit may for example comprise one or more of a logic, a memory, and a processor, for on-chip closed loop operation. The sensor module may for exam- pie provide an all-in-one image sensor architecture which is able to acquire self-consi stent, synchronized, and self-registered images with on-chip processing capabilities for internal closed- loop operation.

The on-chip control may for example provide an ultralow power mode with an internal closed loop.

The on-chip control may in particular be configured to control a programmable laser transmitter.

According to an embodiment, the circuitry may provide a single-aperture receiver comprising an on-chip control, a self-registered and single-aperture RGB+SPAD sensor.

The sensor module may further comprise a receiving optics associated with the single-aperture receiver.

The embodiments also disclose an electronic device comprising the sensor module of claim 1 and an application processor that is communicatively coupled to the all-in-one sensor module. The application processor may for example be a main processing unit of a smart device such as a smartglasses device or a smartphone.

An on-chip control of the sensor module may provide a first operating mode in which the application processor is in a sleep state. This first operating mode may be an ultralow power always on operating mode. In this ultralow power always on operating mode the on-chip control unit may provide a local autonomous closed loop control in which low power sensing on RGB and/or low latency sensing on EVS and/or low power sensing on SPAD with optional use of the fully programmable pulsed laser transmitter (TX) may be provided.

The on-chip control of the sensor module may further provide a second operation mode in which the application processor is active. The second operating mode may for example be a low power single sensor (external loop) operation mode. In this second operating mode an external loop via the application processor may be configured with CPU/NPU control. In this low power single sensor XR (external loop) operation mode fully customizable imaging/ sensing on RGB/EVS is possible and/or fully customizable sensing on SPAD with optional use of the fully programmable pulsed laser transmitter (TX) may be possible.

Embodiments are now described in more detail with regard to the figures.

Fig. 1 shows a smartglasses device with an all-in-one sensor module. The smartglasses 200 is configured to be an eye or head-worn wearable computer that offers extended reality (XR) experiences to the user. The smartglasses device 200 includes displays and/or projectors that overlay information on top of what the wearer sees. The smartglasses device 200 comprises an all-in-one XR sensor module 101 which is configured to gather different aspects of the scene in front of the user, like images suitable for machine vision (i.e. deep neural network processing, automated tasks), images suitable for video streaming and photoshooting, depth maps for scene reconstruction. In addition, the smartglasses device 200 may comprise additional sensors (not shown in Fig. 1) which provide other information such as acceleration from an inertial measurement unit (IMU), temperature and ambient light sensing, etc. Based on the information obtained by the sensors the smartglasses 200 can act as an augmented reality device that performs pose tracking.

Fig. 2 shows in more detail an example architecture of an XR system of a smartglasses device, the XR system comprising an all-in-one sensor module. The XR system comprises a main processing unit (application processor AP) 100 that is communicatively coupled to the all-in-one sensor module 101. The all-in-one sensor module 101 is configured to capture a scene 102. A self-registered and single-aperture RGB+SPAD sensor 110 of the all-in-one sensor module 101 provides selfregistered (coaxial and spatially coherent) single-aperture RGB imaging capabilities, EVS sensing capabilities and SPAD sensing capabilities. The RGB imaging may provide high quality imaging down to low power low resolution sensing. The EVS sensing provides an event-based mode for low-latency and low-power operation. The SPAD sensing provides several modes, from photon counting (PC) to time-of-flight (ToF) range imaging. The all-in-one sensor module 101 further comprises a fully programmable pulsed laser transmitter (TX) 103 with a laser driver 105 and a VCSEL array 106 for dynamic arbitrary patterns generation. A transmitting optics I l l is arranged before the transmitter (TX) 103. The closely controlled and programmable VCSEL array 106 is driven by a laser diode driver (LDD) 105. The programmable pulsed laser transmitter (TX) 103 is controlled an on-chip control unit 107, 108, 109 for local autonomous closed loop control. The on- chip control unit comprises a logic 107, a memory (MEM) 108, and a processor (CPU/NPU) 109 for on-chip closed loop operation. The self-registered and single-aperture RGB+SPAD sensor 110 together with the CPU/NPU 109, a memory (MEM) 108 and the logic (107) form a single-aperture receiver (RX) of the all-in-one sensor module 101 which is configured with a receiving optics 113. The all-in-one image sensor module 101 allows for multiple RGB & EVS operating modes and provides fully programmable macropixels arranged in a SPAD array.

The all-in-one image sensor architecture and module of Fig. 2 is able to acquire self-consi stent, synchronized, and self-registered images with on-chip processing capabilities for internal closed- loop operation (see Fig. 7a for more details). The sensing unit satisfies all typical needs for XR, namely variable resolution images and reconfigurable depth maps with the help of a flexibly programmable illuminator. The self-registration is realized by stacked implementation of the sensing layers (see Fig. 3) for perfectly coaxial image planes, and the ultralow power internal closed loop by a programmable control unit and either a CPU or NPU able to selectively optimize the module operation.

Fig. 3 shows, in a cross-sectional schematic view, an example of a self-registered and single-aperture RGB+SPAD sensor (110 in Fig. 2) of an all-in-one sensor module. The sensor is configured according to a triple wafer stacking technique with dual sensing layer 400. A dual band pass filter (DBPF) 401 for wavelengths of interest selection is arranged above the dual sensing layer 400. The dual sensing layer 400 comprises a thin RGB sensitive layer 404 (photodiode layer, PD) with no or little NIR/SWIR absorption. This RGB sensitive layer 404 may for example comprise Silicon photodiodes or organic photodiodes. The dual sensing layer 400 further comprises a visible-cut filter 405 for residual visible light removal. The dual sensing layer 400 further comprises a Silicon (NIR) or Germanium (SWIR) -based SPAD layer 406 for iToF/dToF/PC. The SPAD layer 406 may for example comprise Germanium on Silicon or Germanium-based Singlephoton avalanche diodes. A readout layer 407 in CMOS technology is arranged below the dual sensing layer 400. The readout layer 407 may for example be realized as a CMOS readout and processing layer. The connectivity for the SPADs may for example be realized as a high density wafer-to-wafer connectivity (e.g. by hybrid bonding).

In the self-registered and single-aperture RGB+SPAD sensor of Fig. 3, Event-based Vision Sensor (EVS) functionality may be added at RGB wafer level or into readout wafer level.

Fig. 4 schematically shows an exploded view of a self-registered and single-aperture RGB+SPAD sensor. An RGB sensitive layer 404 comprising red, green and blue pixels is stacked above a SPAD layer 406. A readout layer 407 is arranged below the RGB sensitive layer 404 and the SPAD layer 406.

The RGB+SPAD sensor may provide the advantage that RGB, EVS and SPAD data/images are self-registered and consistent. That is, a 1-to-l spatial and temporal correspondence in the image of intensity, color, and distance data may be possible. Still further, the image data may also be temporarly registered. For example, the sensor module may provide full parallel simultaneous acquisition of both RGBZEVS and SPAD imaging. The all-in-one imaging sensor of the embodiments is particularly suitable for application in mobile devices with an extremely tight power budget and slim form factor and may avoid the need for a multitude of separate imagers and sensors.

With the configuration of the all-in-one imaging sensor, the extremely tight requirements in terms of area (size) and power of sensing devices that XR applications have can be met. In particular, the challenge in size is solved as a relevant number of imaging and sensing devices are provided that are able to gather different aspects of the scene, like images suitable for machine vision (i.e. deep neural network processing, automated tasks), images suitable for video streaming and photoshooting, depth maps for scene reconstruction, plus other inputs such as acceleration from an inertial unit, temperature and ambient light sensing, etc.

The single-module approach of the all-in-one sensor module may provide a huge area and volume gain as it requires only slightly more area than a single sensor. Still further, the single-module approach of the all-in-one sensor module requires just one electrical interface for the different sensor types that are integrated in the module.

Additionally, the processing power needed for combining these diverse sources of information is increased by their etherogeneity, different field of views, different control and different data format. All decisions and data must be complete and are often redundant, because any decision can be taken only after they are conveyed through a power-hungry application/host processing unit, eventually jeopardizing the possibility to obtain an effective low-power always-on operation. By the all-in-one imaging sensor tight power requirements are met without constraining both the operation of the most power-hungry sensors (e.g. ToF sensors) thus prohibiting their continuous acquisition, and the streaming of all sensors due to the interfaces and data transfer power consumption.

Additionally, with the all-in-one imaging sensor of the embodiments, extra complexity (mechanical assembly, data processing, more interfaces) can be avoided that is needed for the calibration of separate sensors (often from different vendors), their spatial and temporal registration (i.e. data alignment in the image space and in time) and the preservation of this calibration during operation (e.g. baseline of cameras in smartglasses with non-rigid frame).

The all-in-one sensor module may further provide the advantage that much more information is available to the sensor. This allows for local combination and processing of dToF, RGB, and EVS data which may enable additional functions such as mapping, awareness, SLAM, and virtual surfaces interaction. This may in particular allow for a machine vision optimized approach. The sensor can be designed not for image quality but for information quality, with a push for power optimization. The all-in-one sensor module may for example allow for devices performing a dynamic and adaptive laser pattern allocation and acquisition without the aid of other external sensors.

The all-in-one sensor module may for example be used in virtual reality (VR) applications, extended reality (XR) applications, mixed reality (MR) applications, e.g. on battery-powered lightweight mobile devices like smartglasses but also in mobile phones, smartwatches, etc. Applications in which the all-in-one sensor module can be used may comprise gaming, navigation, virtual interfaces, or the like.

Fig. 5 shows, in a more detailed cross-sectional view, an example of a self-registered and singleaperture RGB+SPAD sensor as an all-in-one sensor module. The RGB layer 404 comprises a plurality of photodiodes (PD) 307 that are arranged in a pixel array (313a and/or 313b). The first pixel array 313a is an example of how a sensor 304 having RGB peripheral connectivity can be arranged. The second pixel array 313b is an example of how a sensor 305 having dense per-pixel or per-pixel group connectivity can be arranged. The SPAD layer 406 comprises a plurality of single-photon avalanche diodes (SPADs) 308a, 308b, 308c, and 308d. In the first sensor example 304, the pixel array 313a is connected to the CMOS readout layer 407 via a peripheral RGB connectivity 312 (Btm-Top conn) and there is provided a per pixel SPAD connectivity for SPADs 308a and 308b to the CMOS readout layer 407. In the second sensor example 305, the pixel array 313b is connected to the CMOS readout layer 407 via a per pixel RGB connectivity and there is provided a per pixel SPAD connectivity for SPADs 308c and 308d to the CMOS readout layer 407. The CMOS layer 407 comprises readout circuits 309 that is arranged to read out the data obtained from the photo diodes 307 of the pixel array 313a and/or 313b and from the SPADs 308a, 308b, 308c, and 308d.

It should be noted that the embodiment of Fig. 5 comprises two alternative configurations 304 and 305 of the sensor module that are typically applied as alternatives and are not necessarily implemented on the same sensor. According to a further embodiment, a sensor module comprises only the configuration 304. According to a another embodiment, a sensor module comprises only the configuration 305.

I/O pads 306 are arranged in the RGB layer 404 that connect the internal signals of the sensor to e.g. external pins of a chip package. Conductive through-chip connections 310 and are arranged in the SPAD layer 404 and metal connectivity is arranged in RGB 404, SPAD 406 and CMOS 407 layers to transport signals from the CMOS readout layer circuitry 309 the I/O pads 306. The top-to-bottom connectivity (for the photodiodes of the RBG layer, the I/O pads, and so on) may for example be realized by Through Silicon Via technology (TSV) or by thermo-compression bonding (TCB) technology.

Fig. 6 schematically shows an exemplifying timing diagram of the direct time-of-flight operation for depth sensing of the all-in-one sensor module for a direct time-of-flight operating mode. The VCSEL array of the all-in-one sensor module is selectively driven by the control unit so as to activate only selected VSELs. In a time interval A, only a single VCSEL of the array, namely the upper-left VCSEL of the array is activated and periodically driven with high light intensity. In time interval A there are three strong peaks in energy consumption related to this driving of the VCSEL. In a subsequent time interval B, another single VCSEL of the array are activated and periodically driven with low light intensity. In time interval B there are three peaks in energy consumption related to this driving of this VCSEL. As the pixel is driven with medium light intensity, the energy consumption is less than in time interval A and the peaks are less strong than in time interval A. In a subsequent time interval C, no VCSELs of the array are activated. In time interval C there are consequently no peaks in energy consumption. In a subsequent time interval D, two other VCSELs of the array are activated and periodically driven with medium light intensity. In time interval D there are six peaks in energy consumption related to this periodic driving of the two VCSELs. SPADs of the self-registered and single-aperture RGB+SPAD sensor 110 detect the light emitted by the VCSELs in time interval A, B, C, and D.

It should be noted that the sequence of Fig. 6 is only an illustrative example. The sequence and energy may be optimized in different ways, e.g. to achieve the required information with minimum power consumption, or to satisfy certain precision requirements with just enough power.

Figs. 7a and 7b show two alternative sensor-enabled closed loop operating modes of the all-in- one sensor module.

Fig. 7a shows an ultralow power always on (internal loop) operating mode of the all-in-one sensor module. In this ultralow power always on (internal loop) operating mode the application processor 100 (the main processor of the wearable device) is in a sleep state. The on-chip control unit provides a local autonomous closed loop control. This internal control loop of the self-regis- tered and single-aperture RGB+SPAD sensor 110 is configured via local CPU/NPU 109 of the all-in-one sensor module. In this ultralow power always on (internal loop) operating mode, low power sensing on RGB and/or low latency sensing on EVS and/or low power sensing on SPAD with optional use of the fully programmable pulsed laser transmitter (TX) may be provided. This operating mode for example allows for a fast and cheap proximity processing (e.g. presence detection, gesture recognition, etc). The sensor all-in-one sensor module may act as a smart trigger: decision and reaction (e.g. wake up AP, acquire better image, etc). No data transfer to the application processor 100 takes place.

The internal control loop of the self-registered and single-aperture RGB+SPAD sensor 110 may, in addition to CPU/NPU 109 may also use memory 108 and logic 107 (which makes it a system- on-chip, not requiring external memory or logic). This allows operation without accessing to external power-hungry memory or logic. The memory 108 can contain either CPU/NPU program or algorithms, and data.

The ultralow power always on (internal loop) operating mode has the advantage that no data streamout to the application processor is needed while the loop is closed internally. This results in ultra low power operation which is not possible to achieve with separate sensors.

Due to local processing and feedback in the all-in-one sensor module intelligent decisions can be taken aiming at reducing the power by dynamically allocating laser power in space and time without data streamout.

Fig. 7b shows a low power single sensor XR (external loop) operation mode of the all-in-one sensor module. In this low power single sensor XR (external loop) operation mode the application processor 100 (the main processor of the wearable device) is active. An external loop via the application processor 100 with CPU/NPU control is configured. In this low power single sensor XR (external loop) operation mode fully customizable imaging/sensing on RGB/EVS is possible and/or fully customizable sensing on SPAD with optional use of the fully programmable pulsed laser transmitter (TX) is possible. This operating mode allows for complex external processing (e.g. semantic extraction, scene segmentation, etc) and sensor programming (e.g. high resolution imaging, region of interest ToF sampling, etc).

Providing different operation modes such as described with regard to Figs. 7a and b above may provide programmability and flexibility. For example, internal and external loops can be combined to minimize power and maximize performance.

Fig. 8 shows an example of a system typical operation of the all-in-one sensor module over time. The diagram shows the power consumption of the all-in-one sensor module over time. Operation starts at time to with an ultralow power always-on operating mode 800. This operating mode corresponds to the operating mode described above with regard to Fig. 7a above. In this ultralow power always-on mode an internal control loop of the self-registered and single-aperture RGB+SPAD sensor 110 is configured via local CPU/NPU 109 of the all-in-one sensor module. Between time to and time ti the all-in-one sensor module operates at low resolution (LowRes) and acts as a smart trigger. At time ti the internal control loop registers an event (e.g. by hand recognition) which triggers acquisition and streaming of better images (e.g. mid-resolution grayscale or RGB). After time ti tasks such as mapping, awareness, SLAM, and virtual surfaces interaction are performed by the application processor (AP) 100 (the main processor of the wearable device) in different imaging/sensing states (RGB sensing/imaging and/or SPAD: range imaging). Between tl and t7 the all-in-one sensor can still operate in a mixed situation between Fig. 7a and Fig. 7b, where the internal loop continues to optimize power and performance while the external loop enables frames acquisition by the AP and more complex application-based reconfigurability. After time ti and until time t2the sensor is in a ’’full mapping” imaging/sensing state (see also Fig. 9a and corresponding description) in which better images are very rarely acquired at a mid-resolution with all dots (generated by the VCSEL array 106) active. This results in a high power consumption. At time t2 the sensor changes to an “anchoring” imaging/sensing state (see also Fig. 9b and corresponding description) in which very frequently only a few dots are active. During anchoring the power consumption of the all-in-one sensor module is significantly reduced. At time ti the all-in-one sensor module changes to a “smart update” imaging/sensing state (see also Fig. 9c and corresponding description) in which sometimes some dots are active. At time t4 the all-in-one sensor module changes back to the “anchoring” state in which very frequently only a few dots are active. The power consumption of the all-in-one sensor module is again significantly reduced. At time ts the all-in-one sensor module changes to the “full mapping” state in which better images are very rarely acquired at a mid-resolution with all dots active. This results in a significantly increased power consumption. At time te the all-in-one sensor module changes back to the “anchoring” state in which very frequently only a few dots are active. The power consumption of the all-in-one sensor module is again significantly reduced. At time t? the application processor (AP) 100, depending on the results or the applications requirements, may decide to capture images at high resolution (High res). From time t? to time ts the all- in-one sensor module is in the low power single sensor XR (external loop) operation mode described above with regard to Fig. 7b above. An external loop via the application processor 100 with CPU/NPU control is configured. External processing by the application processor is in- voked for complex function such as streaming, semantic analysis and photo processing (e.g. person identification). With the application processor being active for heavy duty tasks, power consumption is strongly increased (not shown in Fig. 8).

It is stated in the example above that a ’’full mapping” is performed only very rarely. “Full mapping” may for example happen only at transition between the intemal/external loops, or when the user points to a completely new part of the environment that has not been mapped at all by the application processor AP (like at time t5 in Fig. 8). This could for example happen in a few percent of the cases, likely 1-2% of time. But it should be noted that the embodiments are not limited to the ’’full mapping” happening very rarely. This is only given as an example.

Fig. 9a schematically describes a “full mapping” imaging/sensing state of the all-in-one sensor module. In this “full mapping” state images are very rarely acquired at a mid-resolution with all dots active. That is all VCSELs of the VCSEL array 106 are active. This results in a high power consumption.

Fig. 9b schematically describes an “anchoring” imaging/sensing state of the all-in-one sensor module. In this “anchoring” state very frequently only a few dots are active. During anchoring the power consumption of the all-in-one sensor module is significantly reduced. During anchoring, only relevant VCSELs (e.g. to illuminate and track relevant regions for SLAM, like edges, corners, texture in surface) are active. The dots follow the relevant features. For example, with the help of an image of the RGB-sensor an image analysis may be carried out, in order to recognize which regions are particularly relevant for the task of face recognition. The programmable pulsed laser transmitter (TX) 103 triggers only those VCSELs which have been considered relevant by the image analysis. Also, only those pixels of the SPADs are activated for which incident light from the triggered VCELs is expected. This saves power.

Fig. 9c schematically describes a “smart update” imaging/sensing state of the all-in-one sensor module. In the “smart update” state sometimes some dots are active. This is particularly helpful if the scene moves (e.g. because the camera moves). It is assumed that the new areas entering the image are of particular interest and thus covered with dots.

***

It should be noted that the description above is only an example configuration. Alternative configurations may be implemented with additional or other units, sensors, or the like. It should also be noted that the division of the systems into units is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example, on a chip, in FPGA, or the like, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

Note that the present technology can also be configured as described below.

(1) A sensor module comprising circuitry configured to provide a dual sensing layer (400), the dual sensing layer (400) providing self-registered RGB imaging capabilities and SPAD sensing capabilities, wherein the self-registration is realized by a stacked implementation of sensing layers.

(2) The sensor module of (1), wherein the dual sensing layer (400) comprises a photodiode layer (404).

(3) The sensor module of (2), wherein the photodiode layer (404) is configured for sensing visible light (RGB).

(4) The sensor module of any one of (1) to (3), wherein the dual sensing layer (400) comprises a SPAD layer (406) for sensing NIR or SWIR spectra.

(5) The sensor module of (4), wherein the SPAD layer (406) is configured for sensing NIR or SWIR spectra.

(6) The sensor module of any one of (1) to (5), wherein the dual sensing layer (400) further comprises a visible-cut filter (405) for residual visible light removal.

(7) The sensor module of any one of (1) to (6), wherein the circuitry comprises a dual band pass filter (401) arranged above the dual sensing layer (400).

(8) The sensor module of any one of (1) to (7), wherein the circuitry comprises a readout layer (407) arranged below the dual sensing layer (400). (9) The sensor module of any one of (1) to (8) wherein the circuitry is configured to further provide EVS sensing capabilities.

(10) The sensor module of (9), wherein the EVS sensing provides an event-based mode for low- latency and low-power operation.

(11) The sensor module of any one of (1) to (10), wherein the circuitry provides an all-in-one imaging and sensing for mobile extended reality applications.

(12) The sensor module of any one of (1) to (11), wherein the circuitry further provides a programmable laser transmitter (103).

(13) The sensor module of any one of (1) to (12), wherein the circuitry further provides a programmable on-chip control (107, 108, 109) for local autonomous closed loop control.

(14) The sensor module of (13), wherein the on-chip control (107, 108, 109) provides an ultralow power mode with an internal closed loop.

(15) The sensor module of (13) or (14), wherein the on-chip control (107, 108, 109) is configured to control a programmable laser transmitter (103).

(16) The sensor module of any one of (1) to (15), wherein the circuitry provides a single-aperture receiver (RX) comprising an on-chip control (107, 108, 109), and a self-registered single-aperture RGB+SPAD sensor (110).

(17) The sensor module of (16), wherein self-registered refers to both spatial and temporal alignment between RGB and SPAD.

(18) An electronic device comprising the sensor module of any one of (1) to (17) and an application processor (100) that is communicatively coupled to the sensor module (101).

(19) The electronic device of (18), wherein an on-chip control (107, 108, 109) of the sensor module provides a first operating mode in which the application processor (100) is in a sleep state.

(20) The electronic device of (189 or (19), wherein an on-chip control (107, 108, 109) of the sensor module provides a second operation mode in which the application processor (100) is active.

Claims

1. A sensor module comprising circuitry configured to provide a dual sensing layer, the dual sensing layer providing self-registered RGB imaging capabilities and SPAD sensing capabilities, wherein the self-registration is realized by a stacked implementation of sensing layers.

2. The sensor module of claim 1, wherein the dual sensing layer comprises a photodiode layer.

3. The sensor module of claim 2, wherein the photodiode layer is configured for sensing visible light.

4. The sensor module of claim 1, wherein the dual sensing layer comprises a SPAD layer for sensing NIR or SWIR spectra.

5. The sensor module of claim 4, wherein the SPAD layer is configured for sensing NIR or SWIR spectra.

6. The sensor module of claim 1, wherein the dual sensing layer further comprises a visible-cut filter for residual visible light removal.

7. The sensor module of claim 1, wherein the circuitry comprises a dual band pass filter arranged above the dual sensing layer.

8. The sensor module of claim 1, wherein the circuitry comprises a readout layer arranged below the dual sensing layer.

9. The sensor module of claim 1 wherein the circuitry is configured to further provide EVS sensing capabilities.

10. The sensor module of claim 9, wherein the EVS sensing provides an event-based mode for low-latency and low-power operation.

11. The sensor module of claim 1, wherein the circuitry provides an all-in-one imaging and sensing for mobile extended reality applications.

12. The sensor module of claim 1, wherein the circuitry further provides a programmable laser transmitter.

13. The sensor module of claim 1, wherein the circuitry further provides a programmable on- chip control for local autonomous closed loop control.

14. The sensor module of claim 13, wherein the on-chip control provides an ultralow power mode with an internal closed loop.

15. The sensor module of claim 13, wherein the on-chip control is configured to control a programmable laser transmitter.

16. The sensor module of claim 1, wherein the circuitry provides a single-aperture receiver comprising an on-chip control, and a self-registered single-aperture RGB+SPAD sensor.

17. The sensor module of claim 16, wherein self-registered refers to both spatial and temporal alignment between RGB and SPAD.

18. An electronic device comprising the sensor module of claim 1 and an application processor that is communicatively coupled to the sensor module.

19. The electronic device of claim 18, wherein an on-chip control of the sensor module provides a first operating mode in which the application processor is in a sleep state.

20. The electronic device of claim 18, wherein an on-chip control of the sensor module provides a second operation mode in which the application processor is active.