WO2025202082A1

WO2025202082A1 - Image sensor circuitry and electronic device

Info

Publication number: WO2025202082A1
Application number: PCT/EP2025/057912
Authority: WO
Inventors: Prabhat Avasare; Cedric Caron; Yusuke Ikeda; Marc Pauwels
Original assignee: Sony Depthsensing Solutions NV SA; Sony Semiconductor Solutions Corp
Current assignee: Sony Depthsensing Solutions NV SA; Sony Semiconductor Solutions Corp
Priority date: 2024-03-26
Filing date: 2025-03-24
Publication date: 2025-10-02
Anticipated expiration: 2026-09-26
Also published as: TW202603666A

Abstract

The disclosure pertains to image sensor circuitry that is configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, wherein the compute-in memory circuitry is configured to: detect a feature in an image that is represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; and wherein the image sensor circuitry is further configured to control image data generation based on the image generation parameter that is indicated by the imaging control data.

Description

IMAGE SENSOR CIRCUITRY AND ELECTRONIC DEVICE

TECHNICAL FIELD

The present disclosure generally pertains to image sensor circuitry and an electronic device.

TECHNICAL BACKGROUND

It is generally known to detect a feature based on an image. For example, an image sensor may capture an image and output the image to an application processor, and the application processor may detect the feature in the image.

Although there exist techniques for detecting a feature in an image, it is generally desirable to provide an improved image sensor circuitry and an improved electronic device.

SUMMARY

According to a first aspect, the disclosure provides image sensor circuitry that is configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, wherein the compute-in memory circuitry is configured to: detect a feature in an image that is represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; and wherein the image sensor circuitry is further configured to control image data generation based on the image generation parameter that is indicated by the imaging control data.

According to a second aspect, the disclosure provides an electronic device that includes: image sensor circuitry that is configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, wherein the compute-in memory circuitry is configured to: detect a feature in an image that is represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; wherein the image sensor circuitry is further configured to control image data generation based on the image generation parameter that is indicated by the imaging control data; and wherein the image sensor circuitry further includes a communication section that is configured to output feature data that indicate the detected feature; and an application processor that is configured to receive the feature data from the communication section and to perform processing according to the detected feature that is indicated by the detected feature.

Further aspects are set forth in the dependent claims, the drawings and the following description. BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to the accompanying drawings, in which:

Fig. 1 illustrates image sensor circuitry and an electronic device according to an embodiment;

Fig. 2 illustrates a hand tracking pipeline according to an embodiment;

Fig. 3 illustrates a system diagram according to an embodiment;

Fig. 4 illustrates image sensor circuitry with two semiconductor substrates according to an embodiment;

Fig. 5 illustrates image sensor circuitry with three semiconductor substrates according to an embodiment;

Fig. 6 illustrates a processing flow performed by image sensor circuitry according to an embodiment; and

Fig. 7 illustrates an example of a hardware platform.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

As mentioned in the outset, feature (e.g., a body part of a user, such as a hand or a face, which may perform a gesture such as a hand gesture, facial expression, or the like) may be detected based on an image. For example, an image sensor may capture an image and output the image to an application processor, and the application processor may detect a feature in the image. For example, the feature may include a body part of a user, e.g., a hand, a face, or the like. For example, the body part may perform a gesture (e.g., a hand gesture, a facial expression, etc.), and the feature may include the gesture. For example, the feature may include a human, an animal, an object (e.g., a car, a traffic sign, a building, etc.) or the like.

It has been recognized that low-latency and low-power body-part detection (e.g., hand tracking and/or face detection) may be a desirable feature for power constrained and/or latency critical devices like head mounted devices (HMD)s for augmented reality (AR) and/or virtual reality (VR).

In some instances, hand tracking and/or face recognition with computer vision requires computing capabilities and power that may make it difficult to use it on battery powered devices. The proposed technology may allow to offload a major part or at least a proportion of a load from an application processor. The proposed technology may also allow to reduce a latency of the solution by not requiring the data to be streamed out of an image sensor that captures an image to which the body part detection is applied.

Some embodiments of the disclosure pertain to image sensor circuitry that is configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, wherein the compute-in memory circuitry is configured to: detect a feature in an image that is represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; and wherein the image sensor circuitry is further configured to control image data generation based on the image generation parameter that is indicated by the imaging control data.

The image sensor circuitry may include an application-specific circuitry (ASIC), a field- programmable gate array (FPGA), a programmed microcontroller, or the like. The image sensor circuitry may be configured such that the feature can be detected in the image by the image sensor circuitry.

The image sensor circuitry may include an imaging section. The imaging section may be configured according to a complementary-metal-oxide semiconductor (CMOS) and/or a charge- coupled device (CCD) technology. Light from a scene (which may include the body part of the user) may be focused by an optical element (e.g., lens, mirror or the like) on the imaging section. The imaging section may photoelectrically convert the incident light, generate the image data based on an amount of photoelectrically converted light, and output the generated image data. The image data may include a plurality of pixels, which may be two-dimensionally arranged per image frame. Each of the plurality of pixels may correspond to a portion of the scene and may have a value that may indicate an amount of incident light received from the corresponding portion of the scene.

The compute-in-memory (CIM) circuitry may be configured as a memory that stores the image data generated by the imaging section. The CIM circuitry may receive the image data from the imaging section via Camera Serial Interface (CSI) or any other interface provided by the Mobile Industry Processor Interface (MIPI) Alliance, I²C, Serial Peripheral Interface (SPI), a proprietary protocol, or the like. The CIM circuitry may be based on static random-access memory (SRAM), dynamic random-access memory (DRAM), resistive random-access memory (RRAM), magnetoresistive random-access memory (MRAM), an Indium-Gallium-Zinc-Oxide (IGZO) based field-effect transistor (FET), Phase Change Memory (PCM), or the like. The CIM circuitry may further include computing elements for performing computational operations (e.g., arithmetic operations, logical operations). For example, the CIM circuitry may store parameters (e.g., weights) of a machine learning model (e.g., artificial neural network), and may be configured to execute the machine learning model with the image data stored in the CIM circuitry as input. Executing the machine learning model may include, e.g., tensor multiplication, tensor transpose and/or tensor reshape. The machine learning model may be configured to detect the feature. Thus, the feature may be detected in the image without transferring the image data from the CIM circuitry to an external device (such as an application processor).

The feature may include a body part (e.g., hand, finger, arm, face, eye, head, leg, foot, etc.) of a user (e.g., of a human being) that may be performing a gesture or pose (e.g., that may be performing a predefined gesture or pose, or that may be moving), a human (e.g., a specific predetermined person, a person of a predetermined category such as age, size, sex, clothing, activity, movement direction, etc.), an animal (e.g., cat, dog, bird, fish, etc.), an object (e.g., car, bicycle, traffic sign, building, etc.) or the like.

The CIM circuitry may determine, based on the detected feature, an image generation parameter for controlling an image generation by the imaging section such that the feature may be detected in the image data and/or such that a power consumption of the imaging section may be reduced. The CIM circuitry may provide the imaging control data that indicate the image generation parameter to an imaging control section of the image sensor circuitry.

The imaging control section may control the image data generation by the imaging section according to the image generation parameter indicated by the imaging control data. For example, the imaging control data may directly indicate the image generation parameter, e.g., the imaging control data may include the image generation parameter, and the image control parameter may be stored in sensor control registers (e.g., in a set of registers or inside a register file). For example, the imaging control data may indirectly indicate the image generation parameter, e.g., the imaging control data may include an identifier, a key, a pointer or the like, and the imaging section may retrieve the image generation parameter based on the identifier, key, pointer or the like from a look-up table. The look-up table may be stored in memory, and an on-sensor microcontroller of the imaging section may copy the image control parameter from the look-up table to a sensor control register. It may depend on a sensor control mechanism whether the image generation parameter is directly or indirectly indicated by the imaging control data. For example, in a security sensor, the imaging control data may indirectly indicate the image generation parameter. For example, the imaging control section may control the imaging section such that the imaging section may generate, as the image data, only pixels in a region in which the feature is present (e.g., detected) according to the imaging control data. Thus, energy may be saved due to a reduced number of photoelectrical conversions, and/or due to a reduced image data size that needs to be transmitted, stored and/or processed. For example, the imaging control section may control, according to the image generation parameter indicated by the imaging control data, an exposure time, an exposure index, a pixel binning, a dynamic range, focus, or the like.

The imaging control section may, based on the imaging control data, control a scene illuminator. This may be an addressable illuminator, for example, with distinct lighting elements, such as a one or two dimensional array of lighting elements. The scene illuminator may receive or process signals to control the lighting or lighting elements such that only a part of the illuminator illuminates a scene at a certain time. The illuminator may include an interface for controlling individual or groups of lighting elements. The illuminator may therefore at a given time illuminate one or more regions-of-interest (ROIs), or may illuminate one or more ROIs differently to the rest of the scene, for example more brightly in the one or more ROIs whilst optionally maintaining a level of illumination over the rest of the scene, or, in a case of a dot illuminator that emits light dots, the illuminator may adapt a density of light dots that are emitted into the scene or into one or more ROIs according to the imaging control data. Therefore, a system that includes an illuminator which illuminates part of a scene differently and an imaging section using CIM according to the disclosure may offer lower power consumption. A reduced latency of the imaging section using CIM according to the disclosure may ensure that the illumination for example for one or more ROIs can be adapted rapidly, which may improve an image quality or feature detection in subsequent image temporal frames.

The imaging section may generate frames of image data at subsequent points in time, such that, e.g., the feature may be tracked through time. The imaging control section may control the imaging section such that the imaging section may generate image data at future points in time according to the image generation parameter.

Since the image generation parameter for controlling the imaging section may be generated based on image data generated by the imaging section (at an earlier point in time), the controlling of the imaging section according to the image generation parameter may correspond to a feedback loop.

In some embodiments, the detecting of the feature includes executing an artificial neural network (NN) that is configured to detect a feature in an image inputted into the artificial NN. The artificial NN may include a Feed-Forward Network, a Residual Network (ResNet), a Recurrent Neural Network (RNN), a Convolutional Neural Network (CNN), a Generative Adversarial Network (GAN), a Transformer Neural Network and/or any other suitable NN architecture. The skilled person may find a suitable architecture for the artificial NN based on his expert knowledge.

The artificial NN may be optimized for being executed by the CIM circuitry. For example, the artificial NN may be based on an existing artificial NN, and portions of the existing artificial NN that are not necessary for the detecting of the feature may be omitted. Further optimizations, which may be apparent to the skilled person, may be performed.

As mentioned, the artificial NN may be executed by the CIM circuitry. The artificial NN may receive, as an input, the image data stored in the CIM circuitry, and may provide, as an output, an indication of the feature detected in the inputted image. For example, output may indicate a region-of-interest (ROI) that may correspond to a region in the image (e.g., a set of pixels) in which the feature is detected.

The CIM circuitry may include an NN hardware accelerator that may be configured to execute the artificial NN. For example, the NN hardware accelerator may have a size of 2 mm² or 3 mm² and may include 100 KB or few MB of CIM memory (without limiting the disclosure to these values or to this range of values).

For example, the artificial NN may be optimized to have a size in a range from 100 KB to few MB (without limiting the disclosure to these values or to this range), which may be sufficient for detecting the feature. Accordingly, a specification of the NN hardware accelerator may be sufficient for storing and executing the artificial NN.

In some embodiments, the controlling of the image data generation based on the image generation parameter includes controlling the image data generation such that an energy consumption is reduced.

For example, as mentioned, the controlling of the image data generation may cause the imaging section to generate subsequent image data (e.g., subsequent image data frames) based only on a fraction of the imaging section (e.g., of an array of photosensitive elements of the imaging section), e.g., on a fraction of the imaging section that corresponds to (a ROI associated with) the feature. Thus, an image size may be reduced, and electrical energy may be saved because fewer data may be transmitted and/or processed. For example, as mentioned, the controlling of the image data generation may cause a scene illuminator to emit less light into a scene, e.g., to reduce a brightness of the emitted light and/or to illuminate one or more ROIs (which may be indicated by the image generation parameter) while not emitting light into a rest of the scene, as described above.

In some embodiments, the controlling of the image data generation based on the image generation parameter includes controlling the image data generation such that a visibility of the feature in the image is increased.

For example, the controlling of the image data generation may cause the imaging section to adapt an exposure time, an exposure index, a pixel binning, a dynamic range, a focus or the like, as mentioned, such that an overexposure and/or underexposure of the feature may be avoided, that a contrast of the feature may be optimized, that a color of the feature may be recognized, that a size of the feature in the image data (frames) may be optimized, that a blurriness of the feature may be reduced, or the like.

For example, the controlling of the image data generation may cause a scene illuminator to adapt a brightness or (dot) density of emitted light, as mentioned, such that a contrast of the feature may be optimized, that a color of the feature may be recognized, or the like.

In some embodiments, the detecting of the feature includes determining an image portion, of the image, that corresponds to the feature; and the determining of the imaging control data is based on the determined image portion that corresponds to the feature.

The image portion may correspond to the ROI outputted by the artificial NN. The CIM module may generate the control data such that the imaging section generates the image data with an exposure time, exposure index, pixel binning, dynamic range, focus or the like, such that an image quality of the image data in the determined image portion may allow a detection of the feature.

In some embodiments, the image generation parameter indicates the image portion as a region- of-interest (ROI); and the controlling of the image data generation includes causing generation of subsequent image data in the indicated ROI.

For example, the image generation parameter may indicate a pixel row, a pixel column, a pixel group, a polygon of pixel coordinates, etc. that corresponds to a border of the ROI. The image generation parameter may also indicate multiple ROIs in which one or more features may be detected. As mentioned, the ROI may be determined and outputted by the artificial NN executed by the CIM circuitry. The imaging section may generate subsequent image data (e.g., image data generated at future points in time, after the imaging control section has configured the imaging section to generate image data in the indicated ROI) such that the subsequent image data correspond to the ROI. For example, parts of the imaging section that do not correspond to the ROI may be deactivated and/or not read out, such that an electrical power consumption may be reduced. For example, inside the one or more ROIs, sensor parameters may be changed (e.g., switching off binning to get higher resolution ROIs, illumination, focus etc.).

In some embodiments, the image generation parameter indicates an exposure time; and the controlling of the image generation includes controlling the image data generation to acquire subsequent image data with the exposure time indicated by the image generation parameter.

For example, as mentioned, the feature may be overexposed or underexposed. The CIM circuitry may determine, as the image generation parameter, an exposure time that may reduce an overexposure or underexposure, such that a detection of the feature in the subsequent image data may be facilitated.

In some embodiments, the image sensor circuitry includes an imaging substrate and at least one logic substrate; the imaging substrate is configured to perform the image data generation; the compute-in-memory circuitry is disposed on the at least one logic substrate; the at least one logic substrate is further configured to perform the controlling of image data generation; and the imaging substrate is stacked on the at least one logic substrate.

The imaging section may be disposed on the imaging substrate, and the imaging control section may be disposed on the at least one logic substrate. For example, the CIM circuitry and the imaging control section may be disposed on a same logic substrate, or on separate logic substrates that may be stacked.

The image sensor circuitry may have a stacked layout, in which two or more semiconductor substrates may be stacked. The imaging substrate and the at least one logic substrate may be connected by copper-to-copper (Cu-Cu) connections, through-silicon vias (TSVs), solder bumps, or the like.

The imaging substrate with the imaging section may be arranged at a side of the image sensor circuitry that is exposed to light, such that the imaging section may receive incident light.

The stacked layout may allow reducing an area occupied by the image sensor circuitry. The stacked layout may further allow shorter wirings between the imaging section, the CIM circuitry and/or the imaging control section, such that a faster processing (e.g., due to a reduced signal propagation time) and/or a reduced electrical power consumption (e.g., due to a reduced parasitic capacitance) may be possible.

In some embodiments, the CIM circuitry is based on static random-access memory (SRAM).

For example, as compared to analog in-memory computing (IMC), an SRAM-based CIM circuitry may allow an easier handling of multi-bit precision, an easier handling of a variability over analog IMC, and/or a higher memory density (such that costs of silicon area may be reduced).

In some embodiments, the image sensor circuitry includes a plurality of photosensitive elements for generating the image data.

The plurality of photosensitive elements may be arranged in a one-dimensional or two- dimensional array in the imaging section. Each of the photosensitive elements may include a photodiode and may generate photoelectric charges that may correspond to an amount of light incident on the photodiode. Each pixel of the image data may correspond to at least one photosensitive element and may have a value that may be based on the electric charges generated by the corresponding photosensitive element(s).

In some embodiments, the feature includes a body part of the user.

As mentioned, the body part may include hand, finger, arm, face, eye, head, leg, foot, etc. of a user, and the image sensor circuitry (e.g., the CIM circuitry) may be configured to detect the body part in the image.

In some embodiments, the body part includes a hand.

The image sensor circuitry may be configured to perform hand detection (e.g., determining that the image shows a hand, or determining a bounding box of the hand in the image), and/or may be configured to perform hand pose estimation (e.g., determining that the hand is held in a predetermined way and/or determining a perspective from which the hand is shown in the image).

The hand may perform a gesture, e.g., may show a predefined hand pose and/or may perform a predefined movement. The CIM circuitry may detect the hand and generate the image generation parameter such that subsequent image data show the hand at a suitable quality. The hand may then be tracked in the subsequent image data frames. The CIM circuitry may update the ROI indicated to the imaging control section if the hand moves out of the ROI. In some embodiments, the body part includes a face.

The image sensor circuitry may be configured to perform face detection (e.g., determining that the image shows a face, or determining a bounding box of a face in the image), and/or may be configured to perform face recognition (e.g., recognizing a specific (predetermined) person based on an image of their face).

The face may perform a gesture, e.g., may show a predefined facial expression and/or may perform a predefined movement. The CIM circuitry may detect the face and generate the image generation parameter such that subsequent image data show the face at a suitable quality. The face may then be tracked in the subsequent image data frames. The CIM circuitry may update the ROI indicated to the imaging control section if the face moves out of the ROI.

In some embodiments, the feature includes a gesture performed by the body part.

The gesture may correspond to a movement or pose of the body part. The gesture may be predetermined (e.g., trained by the artificial network and/or associated with a predetermined processing to be performed upon detection of the gesture, etc.). The image sensor circuitry may be configured to detect one or more specific (predetermined) gestures.

For detecting a gesture that includes a movement of the body part, the image sensor circuitry may track the body part across subsequent image frames and determine the movement based on the subsequent image frames on which the body part is tracked. The image sensor circuitry may be configured to detect a movement between subsequent image frames of the image data, and to start gesture detection processing based on the detected movement. The gesture detection processing may include causing the imaging section to generate a plurality of subsequent image frames that represent a ROI in which the movement is detected, and executing the artificial neural network with the plurality of subsequent image frames as input.

In some embodiments, the gesture includes a hand gesture.

The hand gesture may include a pose in which the hand is held and/or a movement performed by the hand. For example, the hand gesture may correspond to predefined angles in joints of the hand and/or of one or more fingers of the hand, to as movement (e.g., rotation, translation, waving, changing angles in joints of the hand or finger(s), etc.).

In some embodiments, the gesture includes a facial expression.

The facial expression may correspond to a shape of the face, of a mouth, of a lip, of a nose, of an eyelid, on an eye muscle, line of sight, of a facial muscle, of a tongue, or the like. The facial expression may include moving the face (e.g., changing the shape of the face, mouth, lip, nose, eyelid, eye muscle, line of sight, facial muscle, tongue, etc.; nodding, shaking a head, or the like).

In some embodiments, the image sensor circuitry further includes a communication section configured to output feature data that indicate the detected feature.

The feature data may include a text (e.g., name, label, description, etc.) that indicates the detected feature, an identifier (e.g., database key, pointer to a look-up table, unique identifier, etc.) of the detected feature, a numeric value (e.g., position, orientation anglejoint angle, feature size, movement speed, movement duration, movement range, etc.) that corresponds to the feature, and/or an embedding that may represent the feature for further processing by an artificial neural network. The feature data may include the indication of the determined feature that is outputted by the artificial NN.

The feature data may be generated, for example, by the CIM circuitry and/or by the communication section. The communication section may transmit the gesture data to an external device (e.g., to an application processor) via a communication interface such as CSI or any other interface provided by the MIPI Alliance, I²C, SPI, a proprietary protocol, or the like.

Some embodiments pertain to an electronic device that includes: image sensor circuitry that is configured to: generate image data based on incident light; receive the image data in compute-inmemory circuitry, wherein the compute-in memory circuitry is configured to: detect a feature in an image that is represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; wherein the image sensor circuitry is further configured to control image data generation based on the image generation parameter that is indicated by the imaging control data; and wherein the image sensor circuitry further includes a communication section that is configured to output feature data that indicate the detected feature; and an application processor that is configured to receive the feature data from the communication section and to perform processing according to the detected feature that is indicated by the detected feature.

The electronic device may include a HMD, smartglasses, a mobile phone (e.g., smartphone), a tablet, a notebook, or the like. For example, the electronic device may be configured as an AR/VR device. The electronic device may control the image sensor circuitry to detect the feature, generate the feature data, and transmit the feature data to the application processor.

The processing according to the feature, which the application processor may perform, may include controlling a game or other AR/VR application, starting a music playback, starting a navigation application, unlocking a user interface, navigating a user interface, initiating a function, starting a phone call, starting an application (e.g., a program), controlling a vehicle or a robot, controlling a camera (e.g., zooming, panning and/or rotating a camera that includes the image sensor circuitry such that the camera may further image the feature), or the like. The skilled person may find further processing that the application processor may perform according to the feature.

Due to detecting the feature in the image sensor circuitry, a load on the application processor may be reduced, and the application processor may have more processing resources available for other applications. Also, detecting the feature in the image sensor circuitry with the feedback loop may be more performant than in the application processor, such that a power consumption may be reduced and/or a processing speed may be increased.

It is noted that the features of the embodiments described herein may be combined in any suitable way.

Returning to Fig. 1, Fig. 1 illustrates image sensor circuitry 1 and an electronic device 2 according to an embodiment.

The image sensor circuitry 1 includes an imaging section 2, a compute-in-memory (CIM) section 3, an imaging control section 4 and a communication section 5.

The imaging section 2 includes a plurality of photosensitive elements for generating the image data, and is configured to generate image data based on incident light.

The CIM circuitry 3 is based on static random-access memory (SRAM). The CIM circuitry 3 is configured to receive image data from the imaging section 2 and to detect a feature in an image represented by the image data. The detecting of the feature includes executing an artificial neural network (NN) that is configured to detect a feature in an image inputted into the artificial NN.

The CIM circuitry 3 is further configured to determine, based on the detected feature, imaging control data that indicate an image generation parameter. The detecting of the feature includes determining an image portion, of the image, that corresponds to the feature; and the determining of the imaging control data is based on the determined image portion that corresponds to the feature.

The imaging control section 4 is configured to control image data generation by the imaging section 2 based on the image generation parameter indicated by the imaging control data.

For example, the image generation parameter indicates the image portion as a ROI; and the controlling of the image data generation includes causing generation (by the imaging section 2) of subsequent image data in the indicated ROI. For example, the controlling of the image data generation based on the image generation parameter includes controlling the image data generation (by the imaging section 2) such that an energy consumption of the image sensor circuitry is reduced.

For example, the controlling of the image data generation based on the image generation parameter includes controlling the image data generation (by the imaging section 2) such that a visibility of the feature in the image is increased.

For example, the image generation parameter indicates an exposure time; and the controlling of the image generation includes controlling the image data generation (by the imaging section 2) to acquire subsequent image data with the exposure time indicated by the image generation parameter.

The communication section 5 is configured to output the feature data that indicate the detected feature.

The image sensor circuitry 1 is included in an electronic device 6 that further includes an application processor 7. The application processor 7 is configured to receive the feature data from the communication section 5 and to perform processing according to the feature indicated by the feature data.

For example, the feature includes a body part of the user. For example, the body part includes a hand. For example, the body part includes a face.

For example, the feature includes a gesture performed by the body part. For example, the gesture includes a hand gesture. For example, the gesture includes a facial expression.

It is noted that, in some embodiments, the CIM circuitry 3 is based on DRAM, MRAM, RRAM, IGZO based FET, PCM, and/or any other suitable memory technology instead of or in addition to SRAM.

It is further noted that the feature is not limited to a hand gesture, facial expression, gesture in general, hand, face or body part in general.

Fig. 2 illustrates a hand tracking pipeline 10 according to an embodiment.

At 11, a sensor (e.g., the imaging section 2 of Fig. 1) outputs image data.

At 12, a hand detection is performed (e.g., by the artificial NN executed by CIM circuitry 3 of Fig. 1). Other objects (e.g., a face) may be detected as well.

At 13, a hand representation is obtained (e.g., by the artificial NN executed by CIM circuitry 3 of Fig. 1). The hand representation includes an indication of poses, key points, and a mesh. At 14, a hand interaction event detection is performed (e.g., by the artificial NN executed by CIM circuitry 3 of Fig. 1). The hand interaction event detection includes detecting and determining gestures or poses of the hand.

An output 15 of the hand tracking pipeline 10 includes an indication of the gestures events and/or hand poses that have been detected and determined at 14.

A further output 16 of the hand tracking pipeline 10 includes an indication of the hand representation obtained at 13.

The present disclosure pertains to implementing the hand tracking pipeline 10 on-sensor (e.g., on the image sensor circuitry 1 of Fig. 1) with an NN processing part executed in a compute-inmemory (CIM) section on-sensor.

The detected hand, the obtained hand representation, and the detected/determined gestures or poses of the hand are examples of a feature.

It is noted that the disclosure is not limited to hand tracking. The hand tracking pipeline 10 is provided for exemplary purposes only. The skilled person may appreciate that in some embodiments, a similar pipeline is performed for tracking other objects or body parts, e.g., a face, and outputs an indication of a facial expression, of facial expression events, and/or of a face representation, and that a similar pipeline may also be performed for detecting other features.

Fig. 3 illustrates a system diagram 20 according to an embodiment. A dashed box 21 indicates on-sensor processing.

At 22, a sensor (e.g., the imaging section 2 of Fig. 1) outputs image data. The image data may be outputted by one or more sensors (i.e., by one or more imaging sections provided in the image sensor circuitry). An example of the processing at 22 of Fig. 3 is the processing at 11 of Fig. 2.

At 23, sensor data streams (which include the image data outputted at 22) are transmitted to sensor data processing logic 24 via a proprietary interface. The sensor data processing logic 24 includes CIM elements, e.g., the CIM circuitry 3 of Fig. 1. The sensor data processing logic 24 processes the sensor data streams in both analog and digital domains.

At 25, gesture data (an example of feature data) that have been generated by the sensor data processing logic 24 are outputted, e.g., by the communication section 5 of Fig. 1. The gesture data indicate gestures events, hand poses, and a hand representation (examples of features). Alternatively or additionally, in embodiments where the sensor data processing logic 24 is configured to perform face detection, the gesture data indicate face events, facial expressions and a face representation (examples of features). The communication section 5 transmits the outputted gesture data via a standardized interface, such as MIPI (e.g., CSI).

At 26, off-chip processing of the gesture data is performed, e.g., in an application processor (such as the application processor 7 of Fig. 1) and/or in other logic provided off-chip.

It is noted that the transmission of sensor data streams at 23 is not limited to a proprietary interface, but is in some embodiments based on a standardized interface such as MIPI (e.g., CSI), I²C, SPI, or the like. The transmission of the gesture data at 25 may be based on another interface than MIPI/CSI, for example on I²C, SPI, or on a proprietary interface.

Fig. 4 illustrates image sensor circuitry 30 with two semiconductor substrates 31 and 32 according to an embodiment. The image sensor circuitry 30 is an example of the image sensor circuitry 1 of Fig. 1. The image sensor circuitry 30 includes a first wafer 31 and a second wafer 32. The first wafer 31 is stacked on the second wafer 32.

The first wafer 31 is an example of an imaging substrate and includes an array 33 of photosensitive elements (pixels), which is an example of the imaging section 2 of Fig. 1, such that the first wafer 31 is configured to perform image data generation.

The second wafer 32 is an example of a logic substrate and includes a CIM circuitry 34 (an example of the CIM circuitry 3 of Fig. 1), an SRAM section 35, on-chip memories 36 (e.g., based on SRAM), digital logic 37 including control registers (which further includes the imaging control section 4 of Fig. 1, such that the second wafer 32 is configured to perform controlling of image data generation), a microcontroller 38 (or, in some embodiments, hardwired control logic) which includes the communication section 5 of Fig. 1, and analog logic 39 (which includes an analog-to-digital converter (ADC)).

Black arrows indicate dedicated on-chip data and control interfaces. An arrow 40 indicates a dedicated on-chip control interface. An arrow 41 indicates an external data and control interface between the sensor chip 30 and an external device (e.g., the application processor 7 of Fig. 1), which is based on MIPI (e.g., CSI) or, in some embodiments, on I²C, SPI, or the like.

The first wafer 31 and the second wafer 32 are electrically connected via solder bumps and/or through-silicon vias (TSVs). For example, the CIM module 34 and the analog logic 39 are electrically connected to the array 33 of photosensitive elements.

Accordingly, the image sensor circuitry 30 is an example of a stacked-sensor architecture with digital in-memory-computation (DIMC), wherein the first wafer 31 includes a pixel layer, and the second wafer 32 includes an ISP, an ADC and the DIMC. Fig. 5 illustrates image sensor circuitry 50 with three semiconductor substrates 51, 52 and 53 according to an embodiment. The image sensor circuitry 50 is an example of the image sensor circuitry 1 of Fig. 1. The image sensor circuitry 50 includes a first wafer 51, a second wafer 52 and a third wafer 53. The first wafer 51 is stacked on the second wafer 52, and the second wafer 52 is stacked on the third wafer 53.

The first wafer 51 is an example of an imaging substrate and includes an array 54 of photosensitive elements (pixels), which is an example of the imaging section 2 of Fig. 1, such that the first wafer 51 is configured to perform image data generation.

The second wafer 52 is an example of a logic substrate and includes an image signal processor (ISP) 55 and analog logic 56 (which includes an ADC).

The third wafer 53 is a further example of a logic substrate and includes a CIM circuitry 57 (an example of the CIM circuitry 3 of Fig. 1), an SRAM section 58, on-chip memories 59 (e.g., based on SRAM), digital logic 60 including control registers (which further includes the imaging control section 4 of Fig. 1, such that the third wafer 53 is configured to perform controlling of image data generation), and a microcontroller 61 (or, in some embodiments, hardwired control logic) which includes the communication section 5 of Fig. 1.

Black arrows indicate dedicated on-chip data and control interfaces. An arrow 62 indicates an external data and control interface between the sensor chip 50 and an external device (e.g., the application processor 7 of Fig. 1), which is based on MIPI (e.g., CSI) or, in some embodiments, on I²C, SPI, or the like.

The first wafer 51, the second wafer 52 and the third wafer 53 are electrically connected via solder bumps and/or through-silicon vias (TSVs). For example, the ISP 55 and the analog logic 56 are electrically connected to the array 54 of photosensitive elements. For example, the ISP 55 is electrically connected to the CIM module 57. For example, the digital logic 60 is electrically connected to the analog logic 56.

Accordingly, the image sensor circuitry 50 is an example of a stacked-sensor architecture with DIMC, wherein the first wafer 51 includes a pixel layer, the second wafer 52 includes a signal treatment layer (or an ADC and an ISP), and the third wafer 53 includes a CMOS layer with standard memories and DIMC.

In the image sensor circuitry 30 of Fig. 4 and in the image sensor circuitry 50 of Fig. 5, sensor operation change decision logic may be implemented using on-sensor-stack-CIM, which may provide a low latency (<15 ms, without limiting the disclosure to this value) feedback loop for controlling a sensor operation.

It is noted that, although sections 35 and 36 in Fig. 4 as well as sections 58 and 59 in Fig. 5 are described as SRAM-based, corresponding sections in some embodiments are based on DRAM, MRAM, RRAM, an IGZO FET, PCM, or any other suitable memory technology.

Fig. 6 illustrates a processing flow performed by image sensor circuitry according to an embodiment. The processing is exemplarily described with respect to the image sensor circuitry 50 of Fig. 5, but corresponding processing is performed by the image sensor circuitry 30 of Fig. 4 and by the image sensor circuitry 1 of Fig. 1 as well.

For example, a goal of the processing is to read, from the array 54 of photosensitive elements, only a region-of-interest (ROI) that includes a detected hand (an example of a body part of a user and, thus an example of a feature), and not an image that corresponds to the whole array 54 of photosensitive elements.

At 71, a raw image captured by the array 54 of photosensitive elements is fed to the ISP 55 on the second wafer 52.

At 72, an ISP-processed image is fed from the ISP 55 to the CIM circuitry 57 as input.

At 73, an output (an indication of a hand ROI that corresponds to a hand detected in the ISP- processed image) of the CIM circuitry 57 is processed by the microcontroller 61 (or, in some embodiments, by hardwired control logic).

At 74, pixel coordinates of the hand ROI are written into control registers of the digital logic 60 that control the analog logic 56 of the imaging sensor circuitry 50.

At 75, the new hand ROI pixel coordinates are fed to image sensor readout circuitry in the analog logic 56.

At 76, when the array 54 of photosensitive elements captures a next frame of image data, only photosensitive elements that correspond to the pixel coordinates of the hand ROI are read, and not the whole array 54 of photosensitive elements.

Thus, the processing flow at 71 to 76 corresponds to a feedback loop from the CIM circuitry 57 to an imaging control.

Further, at 77, the communication section included in the microcontroller 61 (or, in some embodiments, hardwired control logic) transmits gesture data (an example of image data), which are generated by the CIM circuitry 57, and which indicate a gesture (a further example of a feature) performed by the detected hand, via the external data and control interface 62 to an external device (e.g., the application processor 7 of Fig. 1).

It is noted that the processing flow performed by the image sensor circuitry 50 (as well as corresponding processing flows performed by the image sensor circuitry 30 of Fig. 4 and by the image sensor circuitry 1 of Fig. 1) is not limited to detecting a hand and determining a gesture performed by the hand but, in some embodiments, includes detecting a face and determining a face gesture in addition or as alternative to detecting a hand and determining a hand gesture, and/or includes detecting any other feature such as another body part of a user (and/or a gesture thereof), a human, an animal, an object, etc. In some embodiments, also a plurality of features is detected in the image data.

Fig. 7 illustrates an example of a hardware platform 80. The hardware platform 80 may be configured as an AR/VR device or a mobile phone.

The hardware platform 80 includes a sensor 81. The sensor 81 includes, on a first wafer 81a, a pixel array 82 that is configured to capture image data, and, on a second wafer 81b, digital logic 83 that includes control registers, and analog logic 84 that includes an ADC. A black arrow indicates an on-chip data and control interface.

The hardware platform 80 further includes an application processor 85. The application processor 85 is configured to control sensor operations of the sensor 81 and to detect a feature based on the image data captured by the pixel array 82 of the sensor 81.

The sensor 81 and the application processor 85 are configured to communicate with each other via data and control interfaces 86. The data and control interfaces 86 may be based on MIPI (e.g., CSI), I²C, SPI, or the like.

Accordingly, in the hardware platform 80, feature detection is implemented outside the sensor 81 in an off-chip computational element (e.g., in the application processor 85). The off-chip computational element may optionally have a CIM part.

In the hardware platform 80, a sensor operation change decision logic is implemented in the application processor 85, which, however, may result in a much higher latency (50-100 ms, without limiting the disclosure to this value) feedback loop for controlling the sensor operation.

The image sensor circuitry 1, 30 and 50 of Fig. 1, 4 and 5, respectively, includes a CIM circuitry such that a detection of a feature (e.g., body part of a user, such as a hand and/or face, and optionally a determination of a gesture performed by the detected body part) are performed on- chip. Thus, a power consumption may be decreased, a latency may be improved, and a sensor data bandwidth requirement may be reduced in the image sensor circuitry 1, 30 and 50 as compared to the hardware platform 80.

As mentioned, some embodiments pertain to on-sensor feature detection (e.g., hand tracking and/or face detection) using CIM. Some embodiments allow for achieving low-power, low- latency feature detection by running feature detection CNNs on-sensor.

Some aspects of the disclosure include an image sensor chip that is configured for feature detection (e.g., hand tracking and/or face recognition) and that includes imaging circuitry that is configured to output images and an image processing part that is configured to process the output images and to provide a feature detection signal. The image processing part may include a memory with CIM features. The CIM features may include NN features that may be configured to determine a feature detection signal based on the sensor output images. The CIM element may use SRAM or any type of emerging memories. Some aspects also include a feedback loop from the CIM part to the imaging part for controlling an image quality and/or other sensor parameters like cropping parameters. The NN may be configured to determine a feature detection signal based on classification and segmentation of the output images from the sensor.

Please note that the division of the image sensor circuitry 1 into sections 2 to 5 is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the image sensor circuitry 1 could, at least in parts, be implemented by a respective programmed processor, field programmable gate array (FPGA) and the like. Also, the division of the image sensor circuitry 30 into wafers 31 and 32 and sections 33 to 41, and the division of the image sensor circuitry 50 into wafers 51, 52 and 53 and sections 54 to 62 are only provided for illustration purposes and the present disclosure is not limited to any specific division of functions in specific units. Further, the division of the hand tracking pipeline 10 into blocks 11 to 14 is provided for illustration purposes only, and the disclosure is not limited to any specific division of functions in specific blocks or to any specific ordering of the blocks.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

Note that the present technology can also be configured as described below.

(1) Image sensor circuitry, configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, the compute-in memory circuitry being configured to: detect a feature in an image represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; and the image sensor circuitry being further configured to control image data generation based on the image generation parameter indicated by the imaging control data.

(2) The image sensor circuitry of (1), wherein the detecting of the feature includes executing an artificial neural network configured to detect a feature in an image inputted into the artificial neural network.

(3) The image sensor circuitry of (1) or (2), wherein the controlling of the image data generation based on the image generation parameter includes controlling the image data generation such that an energy consumption is reduced.

(4) The image sensor circuitry of any one of (1) to (3), wherein the controlling of the image data generation based on the image generation parameter includes controlling the image data generation such that a visibility of the feature in the image is increased.

(5) The image sensor circuitry of any one of (1) to (4), wherein the detecting of the feature includes determining an image portion, of the image, that corresponds to the feature; and wherein the determining of the imaging control data is based on the determined image portion that corresponds to the feature.

(6) The image sensor circuitry of (5), wherein the image generation parameter indicates the image portion as a region-of- interest; and wherein the controlling of the image data generation includes causing generation of subsequent image data in the indicated region-of-interest.

(7) The image sensor circuitry of any one of (1) to (6), wherein the image generation parameter indicates an exposure time; and wherein the controlling of the image generation includes controlling the image data generation to acquire subsequent image data with the exposure time indicated by the image generation parameter. (8) The image sensor circuitry of any one of (1) to (7), wherein the image sensor circuitry includes an imaging substrate and at least one logic substrate; wherein the imaging substrate is configured to perform the image data generation; wherein the compute-in-memory circuitry is disposed on the at least one logic substrate; wherein the at least one logic substrate is further configured to perform the controlling of image data generation; and wherein the imaging substrate is stacked on the at least one logic substrate.

(9) The image sensor circuitry of any one of (1) to (8), wherein the compute-in-memory circuitry is based on static random-access memory.

(10) The image sensor circuitry of any one of (1) to (9), wherein the image sensor circuitry comprises a plurality of photosensitive elements for generating the image data.

(11) The image sensor circuitry of any one of (1) to (10), wherein the feature includes a body part of a user.

(12) The image sensor circuitry of (11), wherein the body part includes a hand.

(13) The image sensor circuitry of (11) or (12), wherein the body part includes a face.

(14) The image sensor circuitry of any one of (11) to (13), wherein the feature includes a gesture performed by the body part.

(15) The image sensor circuitry of (14), wherein the gesture includes a hand gesture.

(16) The image sensor circuitry of (14) or (15), wherein the gesture includes a facial expression.

(17) The image sensor circuitry of any one of (1) to (16), further comprising a communication section configured to output feature data indicating the detected feature.

(18) An electronic device, comprising: the image sensor circuitry of (17); and an application processor configured to receive the feature data from the communication section and to perform processing according to the detected feature indicated by the detected feature.

Claims

1. Image sensor circuitry, configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, the compute-in memory circuitry being configured to: detect a feature in an image represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; and the image sensor circuitry being further configured to control image data generation based on the image generation parameter indicated by the imaging control data.

2. The image sensor circuitry of claim 1, wherein the detecting of the feature includes executing an artificial neural network configured to detect a feature in an image inputted into the artificial neural network.

3. The image sensor circuitry of claim 1, wherein the controlling of the image data generation based on the image generation parameter includes controlling the image data generation such that an energy consumption is reduced.

4. The image sensor circuitry of claim 1, wherein the controlling of the image data generation based on the image generation parameter includes controlling the image data generation such that a visibility of the feature in the image is increased.

5. The image sensor circuitry of claim 1, wherein the detecting of the feature includes determining an image portion, of the image, that corresponds to the feature; and wherein the determining of the imaging control data is based on the determined image portion that corresponds to the feature.

6. The image sensor circuitry of claim 5, wherein the image generation parameter indicates the image portion as a region-of- interest; and wherein the controlling of the image data generation includes causing generation of subsequent image data in the indicated region-of-interest.

7. The image sensor circuitry of claim 1, wherein the image generation parameter indicates an exposure time; and wherein the controlling of the image generation includes controlling the image data generation to acquire subsequent image data with the exposure time indicated by the image generation parameter.

8. The image sensor circuitry of claim 1, wherein the image sensor circuitry includes an imaging substrate and at least one logic substrate; wherein the imaging substrate is configured to perform the image data generation; wherein the compute-in-memory circuitry is disposed on the at least one logic substrate; wherein the at least one logic substrate is further configured to perform the controlling of image data generation; and wherein the imaging substrate is stacked on the at least one logic substrate.

9. The image sensor circuitry of claim 1, wherein the compute-in-memory circuitry is based on static random-access memory.

10. The image sensor circuitry of claim 1, wherein the image sensor circuitry comprises a plurality of photosensitive elements for generating the image data.

11. The image sensor circuitry of claim 1, wherein the feature includes a body part of a user.

12. The image sensor circuitry of claim 11, wherein the body part includes a hand.

13. The image sensor circuitry of claim 11, wherein the body part includes a face.

14. The image sensor circuitry of claim 11, wherein the feature includes a gesture performed by the body part.

15. The image sensor circuitry of claim 14, wherein the gesture includes a hand gesture.

16. The image sensor circuitry of claim 14, wherein the gesture includes a facial expression.

17. The image sensor circuitry of claim 1, further comprising a communication section configured to output feature data indicating the detected feature.

18. An electronic device, comprising: image sensor circuitry configured to: generate image data based on incident light; receive the image data in compute-in-memory circuitry, the compute-in memory circuitry being configured to: detect a feature in an image represented by the image data; and determine, based on the detected feature, imaging control data that indicate an image generation parameter; the image sensor circuitry being further configured to control image data generation based on the image generation parameter indicated by the imaging control data; and the image sensor circuitry further comprising a communication section configured to output feature data indicating the detected feature; and an application processor configured to receive the feature data from the communication section and to perform processing according to the detected feature indicated by the detected feature.