EP4573394A1

EP4573394A1 - Methods and electronic devices

Info

Publication number: EP4573394A1
Application number: EP23744510.1A
Authority: EP
Inventors: Valerio CAMBARERI; Gianluca Agresti; Adriano SIMONETTO
Original assignee: Sony Depthsensing Solutions NV SA; Sony Europe BV; Sony Semiconductor Solutions Corp
Current assignee: Sony Depthsensing Solutions NV SA; Sony Europe BV; Sony Semiconductor Solutions Corp
Priority date: 2022-08-18
Filing date: 2023-07-26
Publication date: 2025-06-25
Also published as: WO2024037847A1

Abstract

A method is disclosed comprising performing material sensing based on an estimate (Z _direct ) of a direct component and/or an estimate (Z _global ) of a global component of ToF data (401, Z) acquired by a time-of-flight, ToF, sensor. Spot-iToF (401) data of a target object (203) are used as input data and infrared image and/or full frame image is used as auxiliary input data (402) for material sensing and 3D reconstruction (408). The ToF sensor may be an indirect time-of-flight (iToF) sensor or a direct time-of-flight (dToF) sensor. Performing material sensing may include performing material classification/segmentation. The material classification/segmentation information may comprise material type (such as wax, skin, etc.) or material properties (such as roughness, translucency, etc.). Raytraced synthetic data, realistic ground truth direct and global components and ground truth data may be used to train a machine learning (ML)-based material classification algorithm that annotates each output pixel with a material class based on features extracted from the direct and global images. It has been recognized that an ML-based approach for separating (403) the direct and the global components of the spot iToF data may improve the estimation of and correct the multipath interference in sparse indirect time-of-flight (spot-ToF) cameras. The approach may generate optically-accurate, raytraced synthetic data for providing improved ground truth direct and global components (ToF phasors) per scene under parametric or measured dot pattern illumination.

Description

METHODS AND ELECTRONIC DEVICES

TECHNICAL FIELD

The present disclosure generally pertains to the field of Time-of-Flight imaging, and in particular to methods and devices for Time-of-Flight image processing.

TECHNICAL BACKGROUND

A Time-of-Flight (ToF) camera is a range imaging camera system that determines the distance of objects by measuring the time of flight of a light signal between the camera and the object for each point of the image. Generally, a Time-of-Flight camera has an illumination unit that illuminates a re- gion of interest with modulated light, and a pixel array that collects light reflected from the same re- gion of interest.

In indirect Time-of-Flight (iToF) cameras a scene is illuminated with infrared light produced by an active illumination device, typically using a fixed-frequency amplitude modulated continuous wave- form. Three-dimensional (3D) images of the scene are captured by the iToF camera, which is also commonly referred to as “depth map”, or “depth image” wherein each pixel of the iToF image is attributed with a respective depth measurement. A depth measurement is measured by the delay of the return signal as it hits the scene and is reflected to the sensor. In iToF the delay is measured as a phase shift of correlation waveform samples computed from the return signal. The depth image can be determined directly from a phase image, which is the collection of all phase delays determined in the pixels of the iToF camera.

Although there exist techniques for determining depths images with an iToF camera, it is generally desirable to provide techniques which improve the determining of depths images with an iToF cam- era.

SUMMARY

According to a first aspect, the disclosure provides a method comprising performing material sens- ing based on an estimate of a direct component and/ or an estimate of a global component of ToF data acquired by a ToF sensor.

According to a second aspect, the disclosure provides a method for training a machine learning model for material sensing, the method comprising generating training data comprising ToF data and ground truth material labels. According to a third aspect, the disclosure provides an electronic device comprising circuitry config- ured to perform material sensing based on an estimate of a direct component and/ or an estimate of a global component of ToF data acquired by a ToF sensor.

According to a fourth aspect, the disclosure provides an electronic device comprising circuitry con- figured to generate training data comprising ToF data and ground truth material labels.

Further aspects are set forth in the dependent claims, the following description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to the accompanying drawings, in which:

Fig. 1 schematically shows the operational principle of an indirect Time-of-Flight imaging system, which can be used for depth sensing or providing a distance measurement;

Fig. 2 schematically shows a spot ToF imaging system which produces a spot pattern on a scene;

Fig. 3 schematically shows a general outline of a machine learning (ML)-based method for the re- gression of direct and global phasor images. The ML-based regression model is applied at inference on the phasor image Z of an iToF sensor with spot illumination;

Fig. 4 schematically shows in more detail process of material sensing and 3D reconstruction per- formed by an ML-based regression model, wherein spot-iToF data are used as input data and infra- red image and/ or full frame image is used as auxiliary input data;

Fig. 5 schematically shows an embodiment of meta data estimation based on the direct and global light components of iToF data;

Fig. 6 schematically shows an embodiment of material classification performed by a neural network to obtain material classification/ segmentation information;

Fig. 7 schematically shows an embodiment of 3D reconstruction performed by a neural network to obtain a 3D mesh of a target object or a scene;

Fig. 8 schematically shows, in an embodiment, how the light rays emitted by an iToF projector propagate inside objects, such as a face, with different material properties due to the different sub- surface scattering effects;

Fig. 9 shows in diagrams a representation of the uncorrected depths caused by the subsurface scat- tering of light in three objects with different material characteristics; Fig. 10 shows in diagrams a representation of the effect of the subsurface scattering on the iToF di- rect and global intensity component in objects with different material characteristics;

Fig. 11 schematically shows an embodiment of a process performed by an ML-based model regres- sion, wherein a deep neural network implementing an ML-based regression model which takes as input full-frame data of a full phasor frame;

Fig. 12 schematically shows an embodiment of a process performed by an ML-based regression model which takes as input a full-frame phasor image at the sensor resolution;

Fig. 13 schematically shows an embodiment of a process performed by an ML-based regression model, wherein concatenation is performed on the input data and a ML-based regression model which takes as input one or more neighborhoods around a center location;

Fig. 14 schematically shows an embodiment of a process performed by an ML-based regression model, wherein concatenation is performed on the input data and an ML-based regression model which is based on a global regression which takes as input one or more neighborhoods around a center location;

Fig. 15 schematically shows a schematic representation of a training data generation method. The method comprises determining a simulated phasor image (iToF raw data) and corresponding direct and global components from a physically-based rendering based on an iToF sensor model and a di- rect/ global separation;

Fig. 16(a) shows an exemplifying instance of the decomposition in the direct component, I direct of the ground truth phasor image (real part) as provided in a simulation of the data generation method- ology;

Fig. 16(b) shows an exemplifying instance of the decomposition in the global component, I_global, of the ground truth phasor image (real part) as provided in a simulation of the data generation method- ology;

Fig. 17 schematically describes the result of the separation as performed in direct/ global separation of Fig. 15;

Fig. 18 shows a flow diagram visualizing a method for training a neural network to separate the di- rect and the global components of the spot iToF image data to estimate the geometry of an object and to infer the material properties of it; and Fig. 19 schematically describes an embodiment of a ToF device that can implement the processes of performing DGS separation based on an ML-based regression model and of performing 3D recon- struction and a per-point material classification/ segmentation to estimate the geometry of an object and to infer the material properties of it.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 1 to Fig. 19, general expla- nations are made.

As indicated in the outset, it is generally known that the depth is measured by the delay of the return signal as it hits the scene and is reflected to the sensor. In iToF the delay is measured as a phase shift of correlation waveform samples computed from the return signal. Typically, the signal received from the iToF sensor pixels is generally comprised of a direct light component, i.e., the direct cam- era ray from the illuminator to the sensor as reflected by the target surface into the sensor pixel. In addition, a global light component is usually received at each sensor pixel. The global light compo- nent is a summation of multiple reflections and stray light that can be due to the sensor itself; the camera lens and optical filter stack; the scene, as caused by geometric scene features (e.g., corners, concave regions); the scene, as caused by material features (scattering, translucency).

These spurious contributions may be received by a sensor pixel and generally mix with the direct component. This may limit the accuracy of the depth measurement after processing, as the two components (direct and global) cannot be fully separated from the iToF phasors once mixed. The global light causes the Multipath Interference (MPI), as it mixes with the direct path at a certain pixel.

Among other ToF illumination patterns, spot ToF is known by which the ToF system may use an illuminator shining a sparse set of light beams, for determining a phase-shift, as well, and extracting other information on sparse scene locations captured this way.

It is known that measurement artifacts may be present, e.g., due to scattered light, MPI, or the like, which may contribute to systematic measurement error of iToF systems.

The task of MPI removal for full-field sensors has been widely tackled in the literature. The ap- proaches take as input e.g. iToF data at single or multiple frequencies and may estimate a depth im- age with reduced MPI distortion. The input information may consist of, but is not limited to, MPI distorted depth information, MPI distorted depth and amplitude, raw iToF measurements or phasor information computed from the raw data. The output depth images may be directly estimated by the network or may be computed from the estimated network output; in this case the output of the net- work can be an estimate of the direct light component, or a phasor image computed from the direct component alone.

The task of direct-global separation (DGS) is for example related to MPI correction. The known ap- proaches may take in input phasor iToF data and may produce an output where the direct and global light components have been separated either in the transient domain, which corresponds to the impulse scene response, or in the iToF domain.

As it is generally known, direct-global separation, DGS, is already established and implemented in the software pipeline (or datapath) of Spot-iToF systems. However, DGS may have several crucial limitations such as: i) the global component may be assumed to be spatially lowpass, so that it can be estimated from the signal in the valleys without significant recovery error. This may not be the case, in particular when the object or material presents highpass spatial elements such as discontinuities and non-uniformities caused by texture or sharp scene details, ii) the global component data may not be processed beyond removal in the small local neighborhood of a spot region, while special config- urations of materials and scene geometry may have much wider MPI than a single neighborhood, iii) the global component may be estimated locally for each spot region in the sensor array; however, this may limit the inference of useful information such as scene and material properties beyond the current spot, iv) there is no explicit or implicit material model being used by DGS, while a more general estimation procedure may learn DGS from such material models to extract salient proper- ties, v) the direct component may be assumed to be a sparse sampling of an unknown dense direct component, and may have sufficiently high spatial frequency so that fine details may be captured or may be retrieved by fusion with other modalities, e.g., by interpolation with guide data, vi) the dis- tinction between spots and valleys may be so that one does not use the full profile of the spot, as imaged on the sensor, for the estimation of the direct and global components, and vii) the subtrac- tion of valleys may add noise to the computed phasors, thus degrading the SNR; in other words, es- pecially for measurements with relatively low SNR on the direct component, the removal of systematic error caused by subtraction of the global component may not compensate for the addi- tional random noise already present in ToF data, and in fact systematic error may be buried under noise.

In view of the discussion above, it has been recognized that a machine learning (ML)-based ap- proach for separating the direct and the global components of the spot iToF data may improve the estimation of and correct the multipath interference in sparse indirect time-of-fEght (spot-ToF) cam- eras. Such an approach may generate optically-accurate, raytraced synthetic data for providing im- proved ground truth direct and global components (ToF phasors) per scene under parametric or measured dot pattern illumination.

Consequently, some embodiments pertain to a method comprising performing material sensing based on an estimate of a direct component and/ or an estimate of a global component of ToF data acquired by a ToF sensor.

The time-of-flight, ToF, sensor may be an indirect time-of-flight (iToF) sensor or a direct time-of- flight (dToF) sensor.

Performing material sensing may include performing material classification/ segmentation to obtain material classification/ segmentation information, or the like. The material classification/ segmenta- tion information may comprise for example, material type (such as wax, skin, etc.), material proper- ties (such as roughness, translucency, etc.), or the like.

For example, the phasor data may be single-frequency spot-iToF data represented as phasor image Z. The phasor image may be denoted with Z and may be obtained from raw data. The iToF meas- urements comprise two components, namely the in-phase component (I) and the quadrature com- ponent (Q), which are respectively the real and the imaginary part of the iToF phasor.

As an indirect time-of-flight (iToF) sensor, a sparse indirect time-of-flight sensor may be used, that may be detected by infrared camera or photo-diode recordings. For example, an ML-based model (requiring acceleration and on-device parameter storage) may be trained on a use-case specific da- taset which may not generalize to different use-cases or camera modes/ exposure settings. In this manner, the correction of otherwise difficult to correct geometric distortions induced by MPI in depth maps and meshes may be achieved.

In iToF cameras/ device, the direct and global light components may be mixed by the sensing pro- cess. Methodologies may exist to regress them with a combination of algorithms and system-level modifications to the measurement strategy, such as using multiple frequencies, or using dot patterns. Both direct and global components are relevant to assessing material properties; we assume them as inputs in our invention.

It should be noted that the present disclosure is not limited to an iToF sensor acquiring iToF data. Alternatively, a standard Time-of-Flight (ToF) camera/ device may be used. Alternatively, a spot camera/device may be used. Still alternatively, a Full-Field ToF camera/ device may be used. Still al- ternatively, a direct ToF (dToF) camera/device may be used by backscattering from the histogram. Still alternatively, any device or system that is configured to separate the acquired data to its direct and global components may be used. In some embodiments, performing material sensing may comprise obtaining material classification and/ or segmentation information based on an estimate of a direct component and/ or an estimate of a global component. The material classification and/ or segmentation information may be infor- mation obtained by performing material sensing on ToF data acquired by a ToF sensor. The mate- rial classification/ segmentation information may comprise for example, material class, material properties or the like.

For example, raytraced synthetic data, realistic ground truth direct and global components and ground truth data may be used to train an ML-based material classification algorithm that annotates each output pixel with a material class based on features extracted from the direct and global images. The material class may then be mapped to material templates in a use-case dependent lookup table e.g., “hair”, “skin”, “wax”, “plastic”, “cloth”. In parallel, 3D reconstruction may be performed using e.g. only the direct light component which is distortion-free and may therefore yield an accurate mesh which, due to the above material classification, that is provided with per-point/ per-voxel/ per- surface material annotation.

In some embodiments, performing material sensing may comprise material classification of a target object or of a part of the target object. The estimation may not be dense on the image grid. It should be noted that the present disclosure is not limited to a target object. Alternatively, performing mate- rial sensing may comprise material classification of a plurality of target objects or of a complete scene or the like.

In some embodiments, performing material sensing may comprise performing feature extraction on an estimate of the direct component and/ or an estimate of the global component on the ToF data to obtain direct and/ or global features. For example, the direct and/ or global features may include the global amplitude , or the direct amplitude - global amplitude (| Ẑ_direct| - | Ẑ_global|), their ratio or other combinations of Ẑ_dire _ct and Ẑ_global taking into account also their phases.

In some embodiments, performing material sensing may comprise performing material sensing per- point in the ToF data. For example, “point” may be each depth value that is comprised in the depth image.

In some embodiments, the method may comprise performing 3D reconstruction to obtain a 3D model and associating the 3D model with information obtained from the material sensing. For ex- ample, 3D model may be a 3D model of an object in the scene, or a 3D model a complete 3D scene. The reconstructed 3D model may for example be a 3D mesh. Associating the 3D model with infor- mation obtained from the material sensing may comprise annotating the 3D model with information obtained from the material sensing. Performing 3D reconstruction may for example be based on depth data comprised in the iToF, such as corrected depth data or uncorrected depth data. Infor- mation obtained from the material sensing may for example be material classification/ segmentation information, such as material class and or material properties, or the like. Thereby, an accurate 3D reconstruction with material sensing from direct and global Time-of-Flight data may be performed. The information may include 3D labels coming from the classification or from the semantic seg- mentation, or the like.

In some embodiments, the method may comprise performing 3D reconstruction based on the esti- mate of the direct component to obtain a 3D model and associating the 3D model with information obtained from the material sensing. Here, performing 3D reconstruction may for example be based on the direct light component that represents a corrected depth. Performing 3D reconstruction may for example be preferably based on corrected depth data comprised in the iToF data.

In some embodiments, the annotating of the 3D model may be further performed based on meta data comprising the material classification/ segmentation information of a target object and/or mate- rial properties of the target object. The meta data may for example comprise information obtained from the material sensing, such as material classification/ segmentation information, e.g., material class and or material properties, or the like.

For example, a machine learning (ML)-based pipeline may simultaneously estimate material proper- ties, material annotation, and 3D shape (e.g. 3D mesh) with correction of distortion caused by multi- path interference, which is a typical systematic error of indirect time-of- flight cameras /devices under patterned or full-field illumination.

In some embodiments, performing material sensing may comprise applying a machine learning- based regression model to the estimate of the direct component and/ or the estimate of the global component of the ToF data to obtain material classification/ segmentation information and/ or ma- terial properties of the target object. The ML-based model (requiring acceleration and on-device pa- rameter storage) may be trained on a use-case specific dataset which may not generalize to different use-cases or camera modes/ exposure settings. The correction of otherwise difficult to correct geo- metric distortions induced by MPI in depth maps and meshes may be achieved.

In some embodiments, the method may further comprise applying a machine learning-based regres- sion model to the ToF data to obtain the estimate of the direct light component of the ToF data and the estimate of the global component of the ToF data.

In some embodiments, the ToF data may be a phasor image. In some embodiments, the machine learning based regression model in addition to the ToF data captured by the ToF sensor may take auxiliary data as further input.

The auxiliary data may be data from other modes and frequencies, a full-frame infrared or grayscale image sampled by the same sensor or phasor images at higher or lower frequencies than the refer- ence one. Alternatively, the auxiliary data may be multi-channel image that stacks data from different channels.

In some embodiments, the auxiliary data may be an infrared image or a full frame image.

In some embodiments, the machine learning-based model regression may be pretrained based on one or more ground truth images obtained based on direct/ global separation of a transient image of a model scene.

The embodiments also disclose a method for training a machine learning model for material sensing, the method comprising generating training data comprising ToF data and ground truth material la- bels. For example, the ML-based model (requiring acceleration and on-device parameter storage) may be trained on a use-case specific dataset which may not generalize to different use-cases or cam- era modes/ exposure settings. In this manner, the correction of otherwise difficult to correct geo- metric distortions induced by MPI in depth maps and meshes may be achieved.

In some embodiments, the training data may further comprise auxiliary data.

The embodiments also disclose an electronic device comprising circuitry configured to perform ma- terial sensing based on an estimate of a direct component and/ or an estimate of a global component of ToF data acquired by a ToF sensor.

The electronic device may be for example a time-of-flight imaging device such as a camera, may be included in another device, may be an apparatus or system or the like and the illumination unit is adapted to function as an illumination unit for a time-of-flight imaging device. Alternatively, the electronic device may be for example an embedded device, a CPU, a GPU, or a cloud server.

Circuitry may include a processor, a memory (RAM, ROM or the like), a DNN unit, a storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g. liquid crystal, (organic) light emitting diode, etc.), loudspeakers, etc., a (wireless) interface, etc., as it is generally known for elec- tronic devices (computers, smartphones, etc.).

The sensor may be specifically designed for time-of-flight measurements and may be adapted to cap- ture a depth image of the illuminated scene as described herein. The sensor may be configured for direct ToF, where the time delay of the photons emitted by the illumination unit and reflected by the scene are detected, it may be configured for indirect ToF, where basically a phase shift of the light emitted by the illumination unit and reflected by the scene is detected, etc. The sensor may be based on at least one of the following: CMOS (complementary metal-oxide semiconductor), CCD (charge coupled device), SPAD (single photon avalanche diode), CAPD (current assisted photodiode) tech- nology or the like.

The embodiments also disclose an electronic device comprising circuitry configured to generate training data comprising iToF data and ground truth data. The electronic device may be for example an embedded device, a CPU, a GPU, or a cloud server.

Operational principle of an indirect Time-of-Flight imaging system (iToF)

Fig. 1 schematically shows the operational principle of an indirect Time-of-Flight imaging system, which can be used for depth sensing or providing a distance measurement. The iToF imaging sys- tem 101 includes an iToF camera, for instance the imaging sensor 102 and a processor (CPU) 105. The scene 107 is actively illuminated with amplitude-modulated infrared light LMS at a predeter- mined wavelength using the illumination unit 110, for instance with some light pulses of at least one predetermined modulation frequency generated by a timing generator 106. The amplitude-modu- lated infrared light LMS is reflected from objects within the scene 107. A lens 103 collects the re- flected light RL and forms an image of the objects onto an imaging sensor 102, having a matrix of pixels, of the iToF camera. In indirect Time-of-Flight (iToF) the CPU 105 correlates the reflected light RL with the demodulation signal DML which yields an in-phase component value (“I value”) for each pixel and quadrature component values (“Q-value”) for each pixel, so called I and Q values. Based on the I and Q values for each pixel a phase delay value may be calculated for each pixel which yields a phase image. Based on the phase image a depth value may be determined for each pixel which yields the depth image. Still further, based on the I and Q values an amplitude value, and a confidence value may be determined for each pixel which yields the amplitude image and the con- fidence image.

In a full field iToF system for each pixel of the image sensor 102 a phase delay value and a depth value may be determined. In a spot ToF system (see Fig. 2) a scene may be illuminated with spots by a spot illuminator and the phase a value and a depth value may only be determined for (a subset of) the pixels of the image sensor 102 which capture the reflected spots from the scene.

It should be noted that the signal received from the iToF sensor pixels is generally comprised of a direct light component, i.e., the direct camera ray from the illuminator to the sensor as reflected by the target surface into the sensor pixel. In addition, a global light component is usually received at each sensor pixel. The global light component is a summation of multiple reflections and stray light that can be due to the sensor itself, the camera lens and optical filter stack, the scene, as caused by geometric scene features (e.g., corners, concave regions), the scene, as caused by material features (scattering, translucency).

Spot Time-of-Flight Imaging (spot-ToF)

Spot-iToF (or Spot-ToF) cameras leverage a patterned active illuminator, so that the scene is illumi- nated with a few high-intensity regions following specific spatial distributions, instead of uniform illumination. This spatial diversity allows one to measure salient scene properties both on the high- intensity, actively-lit regions, whose main contribution is indeed direct light, as well as on the re- maining unlit regions where any signal that may be present is due to global light (and background ambient light signal). Generally, in Spot-ToF systems the light for illuminating the scene is concen- trated at the center location of light dots in a geometric, periodic pattern (e.g., a repetition of a basic triangle, square, or similar polygonal cell where the light dots are the vertices). This does not exclude the use of other patterns, such as diagonal, vertical, or horizontal line patterns (“light sheets”).

Fig. 2 schematically shows a spot ToF imaging system which produces a spot pattern on a scene.

The spot ToF imaging system comprises a spot illuminator 110, which produces a pattern 202 of spots 201 on a scene 107 comprising an object 203, here a face. An iToF camera 102 captures an im- age (e.g. raw image data) of the spot pattern on the scene 107. The pattern 202 of light spots 201 projected onto the scene 107 by illumination unit 110 results in a corresponding pattern of light spots in the amplitude image and depth image captured by the pixels of the image sensor (102 in Fig. 1) of iToF camera 102. The light spots will appear in the amplitude image produced by iToF camera 102 as a spatial light pattern including high-intensity areas 201 (the light spots), and low-in- tensity areas 202. The spot illuminator 110 and the camera 102 are a distance B apart from each other. This distance B is called baseline. The scene 107 has distance d. However, every object 203 or object point within the scene 107 may have an individual distance d from baseline B. The depth im- age of the scene captured by ToF camera 102 defines a depth value for each pixel of the depth im- age and thus provides depth information of scene 107 and object 203.

Typically, the pattern of light spots projected onto the scene 107, may result in a corresponding pat- tern of light spots captured on the pixels of the image sensor 102. In other words, spot pixel regions may be present among the plurality of pixels (and thus in the pixel values included in the obtained image data) and valley pixel regions may be present among the plurality of pixels (and thus in the pixel values included in the obtained image data). The spot pixel regions (i.e. the pixel values of pix- els included in the spot pixel regions) may include signal contributions from the light direcdy re- flected from the scene 107 but also from other reflections (i.e. multi-path interference) and background ambient light. A spot location is a pixel region including a plurality of pixels and the center of a spot location is the center of a spot pixel region including a plurality of pixels.

Focusing on the dot pattern illumination case, part of the light on the center location is reflected by the objects in the scene. A fraction of this light is correctly captured as direct light on the sensor ar- ray at the pixel location of the corresponding camera ray. Another part diffuses off-peak and into global light component of neighboring pixels (on an extended neighborhood depending on the type of MPI) due to geometric and material properties. The sparsity of the illuminator (when compared to uniform, flat illumination) is so that one can measure such geometric and material effects from the off-peak regions. This may lead to a limited use of the sensor array into only a few spot pixel re- gions, while the valley pixel regions may be used to infer multipath interference.

It should be noted that in the distinction between spots and valleys one may consider that each re- gion receives a mixture of direct light and global light: the spots receive primarily direct light, and the valleys primarily global light. Moreover, spots will receive typically higher light intensity than valley pixel regions, yielding better signal-to-noise ratio (SNR). Conversely, valleys receive typically lower light intensity and therefore lower SNR. Both regions are assumed to follow the well-known iToF noise model.

Direct-Global Separation (DGS)

In the Direct-Global Separation (DGS) method, the direct component Z_direct is estimated at the spot coordinate (the center of the spot), by measuring the phasor at the center location of the spot.

The phasor Z_spot at the center location of the spot comprises the direct component Z_direct, the global component Z_global, and a noise component Z_noise,spot: Z_spot≈ Z_direct + Z_global + Z_noise,spot •

The corresponding phasor Z_valley at the valley coordinates in a small neighborhood of the spot comprises the global component Z_global, and a noise component Z_noise,valley ^: Z_valley ≈ Z_global + Z_noise,valley

It follows that, in a single spot neighborhood (i.e., assuming the global is identical everywhere in the neighborhood), one can obtain the direct component Z_direct at the center location from the phasor Z_spot at the center location of the spot and from the phasor Z_valley at the valley coordinates in a small neighborhood of the spot according to: Z_direct ≈ Z_spot- Z_valley •

The quality of the estimation of Z_direct depends on: i) Z_global being a lowpass signal, so that the measurement MPI of the multi path interference at the valleys is consistent with that on the spots, ii) Z_spot being a sampling of the underlying scene content at sufficiently high spatial frequency, and iii) the noise contribution in the spot and valleys being removed or correctly accounted for in the esti- mation of Z_direct, for example, if the noise energy is larger than the direct signal energy one may not be able to measure the direct component via DGS, and this may result in injecting more noise in the resulting phasors.

Machine-learning (ML)-based regression method

The embodiments described below in more detail propose machine learning (ML) -based models to estimate and correct multipath interference in sparse indirect time-of-flight (spot-ToF) cameras. The embodiments may achieve optically-accurate, raytraced synthetic data generation to provide ground truth direct and global components (ToF phasors) per scene under parametric or measured dot pat- tern illumination. The generated data is then used to train several flavors of a ML-based regression algorithm that reconstructs sparse or dense, direct, or global phasor images from the raw phasor in- put as received from the ToF sensor. The technique may find application primarily where global and direct component estimation enables multipath correction and material sensing, i.e., classification and parametric material attributes estimation.

Fig. 3 schematically shows a general outline of a machine-learning (ML)-based method for the re- gression of direct and global phasor images. The ML-based regression model is applied at inference on the phasor image Z of an iToF sensor with spot illumination.

An iToF sensor 301 with spot illumination acquires a phasor image Z comprising phasor image data (see 401 in Fig. 4). The phasor image Z may for example be single-frequency spot-iToF data. Alter- natively, the ML based regression method may be trained using depth (D oc ZZ, depth is propor- tional to the phase ZZ of the phasor Z) and amplitude images (A = |Z|) as measured by the camera, which can be computed, for example, under spot illumination. Still alternatively, the ML based re- gression method can be also trained to yield the respective depth and amplitude images from the di- rect and global light components. This may be obtained, for example, by using the aforementioned relationships Optionally, data from other modes and frequencies 302 may be used as auxiliary inputs W . Auxiliary inputs IV may be a generally complex multi-channel image that stacks the auxiliary inputs. For ex- ample, other modes may be infrared under active or passive illumination. Additionally, multi-fre- quency spot-iToF data, for example, dual frequency spot-iToF data, can be provided to the method in the form of additional phasor images. Additionally, guide information from another capture mode using the same sensor, such as a full-frame infrared image without active light, can be provided to the method.

The phasor image Z and, the auxiliary inputs IV are transmitted as input to a deep neural network (DNN) implementing the ML-based regression model 303, e.g. Model_θ.

The ML-based regression model 303 operates based on a set of pre-trained parameters 304 obtained in a training phase to produce an estimate Ẑ_direct of the direct component, and an estimate Ẑ_global, of the global component at the output. The pre-trained parameters 304 may be for example the weights of a neural network. The parameters may be set by pre-training the ML-based regression model 303 with pairs of inputs and ground truth outputs (see examples of these outputs in Figs. 13a, b). The output of the ML based regression model 303 are estimates, Ẑ_direct, Ẑ_global, of the full- frame phasor images of the direct and global light components, that is the phasor image Z is decom- posed in direct and global components, namely as Z ≈ Ẑ_direct + Ẑ_global, up to the presence of additive noise.

In the embodiment of Fig. 3, the ML based regression method operates on the phasor image Z or functions of the latter, such as depth and amplitude, as primary input channel. As auxiliary channels IV may be (i) a full-frame infrared or grayscale image sampled by the same sensor, e.g., without ac- tive light; (ii) phasor images at higher or lower frequencies than the reference one.

Still further, in the embodiment of Fig. 3, the outputs of the ML based regression method are the estimates Ẑ_direct, Ẑ_global of the full-frame phasor images of the direct and global light components. The ML based regression method separates, i.e., unmix, the input contributions related to the direct light and the global light at each pixel measured by the sensor. Alternatively, the output of the ML based regression method may be sparse, i.e., the method outputs one phasor value per center loca- tion of each spot.

It should also be noted that a full-frame phasor image Z = I + jQ is received from the iToF sensor. The phasor image may be obtained for example, after phase correction to obtain equal and linear depth - phase characteristic over the whole sensor array, i.e., Z : = γ(Z_raw) where y is a generally per-pixel phase correction. This may consider iToF sensor calibration against cyclic error due to non-sinusoidal illumination and phase gradients due to lags in the propagation of the demodulation signal.

Alternatively, instead of performing phase correction, uncorrected raw data Z = Z_raw before phase correction may be used as input to the ML based regression model 303. Since the correction is gen- erally a fixed phase rotation per pixel, one may defer the calibration y to a point after direct and global estimation.

In this methodology a decomposition of the iToF in-phase component I = Re(Z) comprises I_direct ~ Re(Z_direct ), i.e., direct light (Fig. 16 (a)), and I_global = Re Z_global ), i.e., global light (Fig. 16 (b)). The decomposition holds for both, full-field illumination and spot illumination. Similar holds for the quadrature component Q = I (Z) which comprises Q direct ⁼ i-e., direct light, and, Q_global = Im( Z_global).

It is further noted that once the phasor image Z is provided in either of the above ways, it may or may not be processed further by denoising before providing it as input to the ML-based regression method described herein.

It is still further noted that under spot illumination, how the direct component is spatially high fre- quency as it is modulated by the dot pattern may observed. Conversely, the global component may be spatially low-pass because it is related to the light rays reflecting more than once in the scene, which generate this kind of effect in absence of specular reflectors, e.g., mirror-like objects.

3D reconstruction and material sensing

The embodiments described below in more detail propose machine learning (ML)-based models to estimate the geometry (e.g., the mesh) of an object and to infer the material properties of it. The em- bodiments may assume as input to the ML-based regression model iToF data that split into its direct and global light components, Ẑ_direct and Z_global.

Fig. 4 schematically shows in more detail process of material sensing and 3D reconstruction per- formed by an ML-based regression model, wherein spot-iToF data are used as input data and infra- red image and/ or full frame image is used as auxiliary input data.

Spot-iToF data 401, such as a full-frame phasor image Z of a target object 203, where Z = I + jQ, are acquired by an iToF sensor. The spot-ToF data 401 are processed by e.g., a DGS method to separate them to two direct-global separations 403, namely an estimate Ẑ_direct of the direct light component of the iToF data 401 and an estimate Ẑ_global of the global light component of the iToF data 401. The estimate Ẑ_direct of the direct light component and the estimate Ẑ_global of the global light component are then processed e.g., by a deep neural network, DNN, to obtain an image 408 with a 3D mesh of the target object 203 with material annotations (see 608 in Fig. 7) of the target object 203. Alternatively, the 3D mesh of the target object 203 may be obtained using methods not based on machine learning. For example, well-known 3D reconstruction pipelines may be used such as KinectFusion, as described by Izadi, Shahram, et al. in “Kinectfusion: real-time dynamic 3d sur- face reconstruction and interaction.” ACM SIGGRAPH 2011 Talks. 2011. 1-1.

Optionally, infrared images and/ or full frame images 402 may be used as auxiliary inputs IV. Auxil- iary inputs W may be a generally complex multi-channel image that stacks the auxiliary inputs. The phasor image Z and, the auxiliary inputs W are transmitted as input to a deep neural network (DNN) implementing the ML-based regression model 303, e.g. Model_θ . The implementation of the ML-based regression model 303 is described in more detail in Figs. 11 to 14 below.

In the embodiment of Fig. 4, the estimate Ẑ_direct of the direct light component and the estimate Ẑ_global °f the global light component are processed by a deep neural network, DNN. Alternatively, a different data driven algorithm may be used, such as a decision tree, a random forest, a support vector machine, Support Vector Machine (SVM), or the like. For example, it can be any ml-based pixel-level decision on a material label as long as the ground truth material label is known. Alterna- tively, the algorithm (DNN or not) may be trained using ground truth material annotations (e.g., la- bels) from simulated material model in the computer graphics engine. The simulated material model may include subsurface scattering simulation, reflective material simulation with raytracing, translu- cent medium simulation, and the like.

For example, this material annotation may be implemented by simulating training data e.g., by using any ml-based pixel-level decision on a material label as long as the ground truth material label is known.

In the embodiment of Fig. 4, as auxiliary inputs W may be used other modes may be infrared under active or passive illumination, without limiting the present disclosure in that regard. For example, data from other modes and frequencies, multi-frequency spot-iToF data, for example, dual fre- quency spot-iToF data, can be provided to the method in the form of additional phasor images. Ad- ditionally, guide information from another capture mode using the same sensor, such as a full-frame infrared image without active light, can be provided to the method.

Fig. 5 schematically shows an embodiment of meta data estimation based on the direct and global light components of iToF data. Spot-iToF data 401, such as a full-frame phasor image Z, Z = I + jQ, are acquired by an iToF sensor. The spot-ToF data 401 are processed namely a ML-based regression model 303, that per- forms direct and global separation (DGS) and separates the iToF data 401 into direct-global separa- tions. In this manner the ML-based regression model 303 estimates the direct and the global components, Ẑ_direct and Ẑ_global, as shown in diagram 403 of Fig. 4. Optionally, infrared images and/ or full frame images 402 may be used as auxiliary inputs W. Auxiliary inputs W may be a gener- ally complex multi-channel image that stacks the auxiliary inputs. The phasor image Z and, the auxil- iary inputs 147 are transmitted as input to a deep neural network (DNN) implementing the ML-based regression model 303, e.g., Model_θ . The implementation of the ML-based regression model 303 is described in more detail in Figs. 11 and 12 below.

The ML-based regression model 303 operates based on a set of pre-trained parameters 304 obtained in a training phase to produce an estimate Ẑ_direct of the direct light component, and an estimate Ẑ_global of the global component at the output. The pre-trained parameters 304 may be for example the weights of a neural network. The parameters may be set by pre-training the ML-based regression model 303 with pairs of inputs and ground truth outputs (see examples of these outputs in Figs. 16a, b). The output of the ML based regression model 303 are estimates Ẑ_direct,Ẑ_global of the direct and global light components of the full-frame phasor images. That is the phasor image Z is decomposed into two light components, namely as Z ~ Ẑ_direct + Ẑ_global, up to the presence of additive noise.

It should be noted, the direct-global separations may be used by two modules; the first may use the estimate Z(ji_rect, of the direct light component that is informative about the ideal ToF data without any MPI distortion, to estimate the 3D mesh of the target object (see Fig. 7); and the second module may use both the estimate Z_{d rect} of the direct light component and the estimate Ẑ_global, of the global light component or hand-crafted features extracted by combining them, to estimate the mate- rial class of each scene point (semantic segmentation) or of the entire object at ones (classification) (see Fig. 6).

It should be further noted that the direct and global data e.g., the estimate Ẑ_direct of the direct light component and the estimate Ẑ_global, of the global light component may be the output of a DGS method for Spot-ToF data, as shown in Figs. 11 to 15 below. Alternatively, the direct and global data may be the output of a DGS method for multi-frequency full-field iToF data, as proposed by A. Simonetto, G. Agresti, P. Zanuttigh, and H. Schafer, in the published paper “Lightweight Deep Learning Architecture for MPI Correction and Transient Reconstruction,” arXiv:2111.14396 [cs], Nov. 2021, Accessed: Nov. 30, 2021. Available: http://arxiv.org/abs/2111.14396. The estimate Ẑ_direct °f the direct light component and the estimate Ẑ_global, of the global light component may also be the measured ground truth (e.g. using a dToF sensor), or be the synthetic data produced by a raytracer.

Raytraced synthetic data generation may allow one to generate realistic ground truth direct and global components (ToF phasors) per scene under various illumination and camera settings. These may be used as input or training data for our method. Moreover, the same synthetic data generation may be done with modelled materials and their properties, which provide ground truth material an- notation per-pixel, per-point, or per-voxel depending on the chosen representation.

Fig. 6 schematically shows an embodiment of material classification performed by a neural network to obtain material classification/ segmentation information.

Feature extraction 601 is performed on the estimate Ẑ_direct of the direct light component and the estimate Ẑ_global, of the global light component to obtain direct and/ or global features 602. For ex- ample, the feature extraction 601 is performed on the estimate Ẑ_direct of the direct light component to obtain direct features and/or the feature extraction 601 is performed on the estimate Ẑ_global, of the global light component to obtain global features. The feature extraction 601 may be also per- formed on both the estimate Ẑ_direct of the direct light component and the estimate Z_giobcd, of the global light component to obtain a combination of direct and global features 602. A material classifi- cation/ segmentation 603, uses the direct and/ or global features 602, e.g., extracted by respectively the estimate Ẑ_direct of the direct light component and the estimate Ẑ_global, of the global light com- ponent, to infer a material classification/ segmentation information 604, e.g. material class and(or material properties, of the target object (see 203 in Fig. 2). Alternatively, the material classifica- tion/ segmentation 603 may be a per-point material classification/ segmentation and may estimate the material classification/ segmentation information 604 for each scene point (semantic segmenta- tion).

Salient properties of a material may be inferred from its global component, e.g., by extracting fea- tures from the estimate Z_giobcd of the global component of the phasor image Z. For example, the direct and/or global features 602 may include the global amplitude l Ẑ_globall, or the direct amplitude

- global amplitude (IẐ_directl - I Ẑ_global l), their ratio or other combinations of Ẑ_direct and Ẑ_global taking into account also their phases. These features are the direct and global features 602 and they are used because each material gives a different footprint to the direct component Ẑ_direct and global component Ẑ_global associated to an iToF measurement. This is because the light pene- trates at different depths in objects according to the density and the internal molecular structure of the material itself, as described in more detail in Fig. 8 below. In this manner, a material annotation (see 607 in Fig. 7) of a target object, such as a face 203, is estimated.

In the embodiment of Fig. 6, an ML-based regression model implemented by a deep neural network, DNN, is used to classify the materials having as inputs the estimate Ẑ_direct of the direct component, the estimate Ẑ_global of the global component and a combination of them. The ML solution, based on the embodiment, may be implemented with different techniques as AdaBoost, Decision Trees, Random Forests, Support Vector Machines, or deep neural networks.

It should be noted that the feature extraction 601 is optional. Alternatively, the estimate Ẑ_direct of the direct component, the estimate Ẑ_global of the global component and/ or a combination of them may be used as an input to the material classification/ segmentation 603 to obtain material classifica- tion/ segmentation information 604, e.g. to estimate the material class.

Fig. 7 schematically shows an embodiment of 3D reconstruction performed by a neural network to obtain a 3D mesh of a target object or a scene.

A 3D reconstruction 605 is performed on the estimate Ẑ_direct of the direct component, which en- codes a corrected depth of the iToF data to obtain a 3D model such as the 3D mesh 606. Annota- tion 607 is performed on the 3D mesh 606 and based on the material classification/ segmentation information 604 to obtain a 3D mesh with material annotations 608, e.g. on an image (see 408 in Fig. 4). The image 408 comprises a 3D geometry (mesh), e.g. of the target object 203 and a material annotation indicating e.g. the material class and respective material properties of the estimated mate- rial of the target object 203. The corrected depth 405 is e.g., extracted by the estimate Ẑ_direct of the direct component which is obtained as described in Fig. 4 above by performing direct and global separation (DGS).

Considering the correct geometry in the estimate Ẑ_direct of the direct component, and the spatial patterns in the estimate Ẑ_global of the global component, one may distinguish between scene multi- path, as in Fig. 13a, b, where the concave corners of the object cause high global component ampli- tude, and sub-surface scattering, wherein the distortion assumes patterns that are inconsistent with simple scene geometry-induced multipath. ML-based models may therefore classify materials using hand-crafted features from global component, e.g., amplitude, geometry, or learned features in DNN-based approaches. The problem may be framed as a Classification task and in this case the method will assign a single class for the whole frame of the direct component Ẑ_direct and the global component Ẑ_global, or the output may be a per pixel classification in case of a Semantic Segmenta- tion task. The estimated material classification may be directly projected on the 3D reconstruction by using texture mapping techniques.

It should be noted that an ML-based regression model is used to implement the processes of the 3D reconstruction module 605 and of the per-point material classification/ segmentation 603, described above. In other words, the ML-based regression model implemented by a DNN estimates material properties, material annotation, and 3D shape with correction of distortion caused by multipath in- terference.

It should be further noted that in the material sensing and 3D reconstruction process may also be used the direct and global component ( Ẑ_direct, Ẑ_global) estimated with other DGS methods. For ex- ample, a DGS method may be used that is not using single frequency Spot-ToF data as shown in Figs. 11 to 15 below. Alternatively, a DGS method using as input multi-frequency Full-Field iToF data may be used, as proposed by E. Buratto, A. Simonetto, G. Agresti, H. Sch fer, and P. Za- nuttigh, in the published paper “Deep Learning for Transient Image Reconstruction from ToF Data,” Sensors, vol. 21, no. 6, Art. no. 6, Jan. 2021, doi: 10.3390/s21061962. Additionally, the di- rect-global information does not necessarily need to come from a DGS method but may also consist of the measured ground truth.

The 3D reconstruction 406 in short-range scanning applications based on iToF data may be affected by multipath interference (MPI). However, the direct component Ẑ_direct, i.e. the corrected depth 405, being the output of the DGS, is an estimation of the ideal iToF data that is not affected by MPI and encodes correct depth information. The MPI free depth map may be estimated as

D ∞ < Ẑ_direct and may be used in well know 3D reconstruction pipelines to estimate accurate 3D point clouds/meshes from spot-ToF data irrespectively of the material, e.g., translucent, as the direct component Ẑ_direct may reject geometric distortion caused by the global component, Z_giobai.

Fig. 8 schematically shows, in an embodiment, how the light rays emitted by an iToF projector propagate inside objects, such as a face, with different material properties due to the different sub- surface scattering effects. The different light subsurface propagation may bring to different effects in the estimate Ẑ_direct of the direct component, the estimate Ẑ_global of the global component, since the more the material lets the light penetrate, the higher the magnitude of the iToF global compo- nent Ẑ_global. In the embodiment of Fig. 8, there are three different objects, namely a wax 610, a skin 611 and a Lambertian 612. For performing material sensing, an iToF sensor illuminates each object. The dot- ted lines show the path, and thus, its distance, followed by the global penetrating light component. The solid lines show the path, and thus, its distance, followed by the direct penetrating light compo- nent. In the case where the object is wax 610, the distance of the path followed by the penetrating light is longer than the distance of the path followed by the penetrating light in the case where the object is skin 611. Therefore, the global component in the wax 610 is smaller than the global com- ponent in the skin 611. The light penetrates deeper the wax 610 than the skin 611 or the Lambertian 612 and the light penetrates deeper the skin 611 than the Lambertian 612. The light subsurface scat- tering is medium in the case of the wax 610 and the skin 611, and there is none in the case of the Lambertian 612. As can be taken from Fig. 8, in the case where the object is Lambertian 612, there is no global component, only direct component.

It should be noted that the deeper the light penetrates the material the higher the global component Ẑ_global and thus, higher MPI may occur. Therefore, due to the deeper light penetration the distance that the light travels is longer and thus, the phase of the global component Ẑ_global is greater.

The subsurface scattering is described in more detail in Fig. 9 below and the effects of the subsur- face scattering on the direct and global components in the wax 610, skin 611 and Lambertian 612 are described in more detail in Fig. 10 below.

Fig. 9 shows in diagrams a representation of the uncorrected depths caused by the subsurface scat- tering of light in three objects with different material characteristics, namely wax, skin and Lamber- tian. The dotted lines represent the subsurface scattering of light on the global component while the solid lines represent the subsurface scattering of light on the direct component.

Fig. 10 shows in diagrams a representation of the effect of the subsurface scattering on the iToF di- rect and global intensity component in objects with different material characteristics.

The left diagram is a representation of the subsurface scattering on the iToF direct and global com- ponents in wax. The dotted line represents the global light component and the solid line represent the direct light component. The global component has intensity close to the intensity of the direct component, which means that the ratio of between the global and the direct component, may be close to 1, The light penetration into the wax is great, as described in

Fig. 7 above. The middle diagram is a representation of the subsurface scattering on the iToF direct and global components in skin. The dotted line represents the global light component and the solid line repre- sent the direct light component. The global component has intensity lower than the intensity of the direct component, which means that the direct component is greater than the global component,

I Ẑ_direct I > l Ẑ_global l, for example the ratio may be close to 0.2, The light pen- etration into the skin is lower than the wax, as described in Fig. 7 above.

The left diagram is a representation of the subsurface scattering on the iToF direct and global com- ponents in Lambertian. The dotted line represents the global light component and the solid line rep- resent the direct light component. The global component has intensity close to zero, which means that the light penetration into the skin is lower than the wax and the skin and is equal to zero, as de- scribed in Fig. 7 above. The direct component is greater than the global component, for example the ratio may be close to zero,

The ratio is mapped with a specific material. In this manner based on the ratio the material of the target object is obtained and therefore its material type as well. Such a mapping is shown in the table below:

From the table above, it can be taken that if the ratio of the global and direct component is in the range 1.0 + 0.2, the material of the target object is specified as being wax. If the ratio of the global and direct component is in the range 0.2 + 0.1 the material of the target object is specified as being skin. If the ratio of the global and direct component is in the range 0 +

O.lthe material of the target object is specified as being Lambertian.

As described above, both direct and global components Ẑ_direct, Ẑ_global are relevant to assessing material properties and are used as inputs to the DNN. Raytraced synthetic data generation allows one to generate realistic ground truth direct and global components (ToF phasors) per scene under various illumination and camera settings. These may be used as input or training data to the ML- based regression model implemented by a DNN. Moreover, the same synthetic data generation may be done with modelled materials and their properties, which provide ground truth material annota- tion per-pixel, per-point, or per-voxel depending on the chosen representation.

These input and ground truth data are used to train a ML-based material classification algorithm that annotates each output pixel with a material class based on features extracted from the direct and global images. The material class may then be mapped to material templates in a use-case dependent lookup table, e.g., “hair”, “skin”, “wax”, “plastic”, “cloth”.

Fig. 11 schematically shows an embodiment of a process performed by an ML-based model regres- sion, wherein a deep neural network implementing an ML-based regression model which takes as input full-frame data of a full phasor frame.

An iToF sensor 701 with sparse illumination acquires a full-frame phasor image Z, Z = I + jQ. The full-frame phasor image Z comprises IQ (In-phase and Quadrature) data. The full-frame phasor IQ data are data on the IQ domain, i.e., on the phasors as produced by the iToF sensor 701 after demodulation.

A full-frame phasor image Z = I + jQ is received from the iToF sensor, since direct and global phasors are provided at every pixel of the sensor. A deep neural network (DNN) 702 takes as input this full-frame phasor image data Z at the sensor resolution. The ML-based model learns to separate the direct and global components Ẑ_direct,Ẑ_global from the raw phasor image Z of an iToF sensor 701 with spot illumination.

The ML-based direct/ global regression model implemented by the DNN 702, using pre-trained pa- rameters, such as direct/ global ground truth images 703 (model parameters, e.g. weights), obtains the direct and global components (Ẑ_direct, Z_giObal) '•= Modelg(Z) . The DNN 702 outputs the estimates Ẑ_direct, Ẑ_global of the direct and global phasor image at the sensor resolution as full frame phasor images of direct and global components (i.e. regressed dense phasor data).

In the embodiment of Fig. 10, the full-frame phasor image Z is used as input to the regression. In alternative embodiments, some preprocessing may be applied to the full-frame phasor image Z and the regression may be based on the preprocessed data. Such preprocessing may for example com- prise performing a concatenation on the input data (see Fig. 13 below).

As indicated by the dashed arrows in Fig 11, the pre-trained parameters are obtained in a training phase based on direct and/ or global ground truth images 703, or global ground truth images 705. With the training of the DNN 702, a set of model parameters 9 are extracted. Therefore, during the ML-based model regression, the estimate Ẑ_direct of the direct component, the estimate Ẑ_global of the global component are obtained and the result are two dense phasor images describing the direct and global components at the full resolution of the sensor. In other words, the DNN 702 perform- ing global regression learns the joint separation and interpolation of the direct and global channels.

It should be noted that any data-driven method suitable for the solution of a regression task may be used, i.e., any method that learns model coefficients by model training with input-ground truth data pairs (Z, W ) → ( Z_direct, Z_global), wherein W is (optional) auxiliary input (see 302 in Fig. 3). Such auxiliary inputs, however, are not necessarily required, i.e., it is sufficient to process Ẑ_direct> Ẑ_global) ^:= Model_θ (Z) (as shown in Fig. 3).

Fig. 12 schematically shows an embodiment of a process performed by an ML-based model regres- sion, wherein a deep neural network implementing an ML-based regression model which is based on a global regression, and which takes as input a full-frame phasor image at the sensor resolution.

A full-frame phasor image Z = I + jQ is received from the iToF sensor, since direct and global phasors are provided at every pixel of the sensor. A deep neural network (DNN) 704 takes as input this full-frame phasor image data Z at the sensor resolution. The ML-based model learns to separate the direct and global components Ẑ_direct, Ẑ_global from the raw phasor image Z of an iToF sensor 701 with spot illumination.

The ML-based global regression model Model_θ (Z) implemented by the DNN 704, using pre- trained parameters 9, such as global ground truth image 705 (model parameters, e.g. weights), ob- tains the global component Ẑ_global := Model_θ (Z) in an inference phase. At 706, an estimate Ẑ_direct of the direct component is obtained based on the global component Ẑ_global provided by DNN 704 and the full frame phasor image Zdirect, global according to: Ẑ_direct := Ẑ_direct, Ẑ_global - Ẑ_global.

The estimates Ẑ_direct, Ẑ_global of the direct and global phasor image at the sensor resolution are then output as full frame phasor images of direct and global components (i.e. regressed dense phasor data).

As indicated by the dashed arrow in Fig 12, the pre-trained parameters are obtained in a training phase based on global ground truth images 705. With the training of the DNN 704, a set of model parameters 9 are extracted. Therefore, during the ML-based model regression, the estimate Ẑ_direct of the direct component, the estimate Ẑ_global of the global component are obtained and the result are two dense phasor images describing the direct and global components at the full resolution of the sensor (see 305, 306 in Fig. 3). In other words, the DNN 704 performing global regression learns the joint separation and interpolation of the direct and global channels.

It should be noted that any data-driven method suitable for the solution of a regression task may be used, i.e., any method that learns model coefficients by model training with input-ground truth data pairs (Z, W ) → ( Z_direct, Z_global), wherein IV is (optional) auxiliary input (see 302 in Fig. 3). Such auxiliary inputs, however, are not necessarily required, i.e., it is sufficient to pro- cess Ẑ_direct- Ẑ_global) '= Model_θ(Z) (as shown in Fig. 3).

In the embodiments of Fig. 11 and Fig. 12, the full-frame phasor image Z is used as input to the re- gression. In alternative embodiments, some preprocessing may be applied to the full-frame phasor image Z and the regression may be based on the preprocessed data. Such preprocessing may for ex- ample comprise performing a concatenation on the input data (see Figs. 13 and 14 below).

Fig. 13 schematically shows an embodiment of a process performed by an ML-based regression model, wherein concatenation is performed on the input data and a deep neural network implement- ing a ML-based regression model which takes as input one or more neighborhoods around a center location, i.e., a pixel illuminated by a sparse dot.

An iToF sensor 801 with sparse illumination acquires direct and global phasors e.g., sparse phasor data, per spot center location. The sparse phasor data are data on the IQ domain, i.e., on the phasors as produced by the iToF sensor after demodulation. A concatenate process 802 is performed on one or more neighborhoods to obtain concatenated phasor data Z_Ω.

A deep neural network (DNN) 803 takes as input the concatenated phasor data Z_Ω. The ML-based global regression model learns to separate the concatenated phasor data Z_Ω at the center location based on local or neighborhood pixels from the raw phasor image Z of an iToF sensor with spot il- lumination.

As already described in Fig. 11 above, an ML-based direct/ global regression model implemented by the DNN 803, using pre-trained parameters, such as direct/ global ground truth images 804 (model parameters, e.g. weights), obtains the direct and global components

( Ẑ_direct, Ẑ_global) := Model_θ(Z) . The DNN 803 outputs the estimates Ẑ_direct> Ẑ_global of the direct and global phasor image at the sensor resolution as full frame phasor images of direct and global components (i.e. regressed dense phasor data). Fig. 14 schematically shows an embodiment of a process performed by an ML-based model regres- sion, wherein concatenation is performed on the input data and a deep neural network implement- ing an ML-based regression model which is based on a global regression which takes as input one or more neighborhoods around a center location, i.e., a pixel illuminated by a sparse dot.

A deep neural network (DNN) 805 takes as input the concatenated phasor data Z_Ω. The ML-based global regression model learns to separate the concatenated phasor data Z_Ω at the center location based on local or neighborhood pixels from the raw phasor image Z of an iToF sensor with spot il- lumination.

An ML-based global regression model Model_θ(Z) implemented by the DNN 805, using pre-trained parameters 0, such as global ground truth image 806 (model parameters, e.g. weights), obtains the global component Ẑ_global '•= Model_θ (Z) in an inference phase. At 807, an estimate Ẑ_direct of the direct component is obtained based on the global component Ẑ_global provided by DNN 704 and the full frame phasor image Z_direct, _global according to:

It should also be noted that any possible combination of the inputs and outputs and the embodi- ments described in Figs. 11, 12, 13 and 14 may be used to implement the ML-based regression method.

Among such models, the most natural is a deep neural network (DNN) architecture such as a net- work comprised of multilayer perceptron or fully connected layers, or a network comprised of con- volutional layers, a kernel prediction network, or a vision transformer.

Among non-DNN models, non-linear regression with radial basis functions may be applied, or ran- dom forests for the embodiments of Figs. 11, 12, 13 and 14, wherein the embodiments of Fig. 10 may be computationally more demanding than the embodiments of Fig. 11. The common point of such models is that their training yields a set of parameters θ that can be used at inference to predict unknown, unobserved ( Ẑ_direct, Ẑ_global).

Training and data generation

The embodiments described herein in more detail propose machine learning (ML) -based models to be trained in a supervised way by using a training set comprising the iToF input and auxiliary inputs, and the ground truth material labels, e.g., wax, skin, wood, plastic, paper, and the like. The label may be unique for each frame in case of a classification task, or per-pixel in case of semantic segmenta- tion. The labels are the desired output of the proposed method ML-based model.

Regarding the material model labels, these may be defined during the set-up of the Computer-Gen- erated (CG) scenes using 3D modelling tools as Blender. The material model may define how the light is interacting with the 3D elements and can be implemented with analytical representation as the microfacet one, e.g., the GGX model by defining the diffuse and specular color, the roughness of the material. Another option is to use material models also emulating the light propagation inside the objects, e.g., causing subsurface effects in the iToF simulations. One such example of subsurface scattering model is provided in D. Bulczak and A. Kolb, “Efficient subsurface scattering simulation for time-of- flight sensors,” in Proceedings of the Conference on Vision, Modeling, and Visualiza- tion, Goslar, DEU, Oct. 2018, pp. 103—109. doi: 10.2312/vmv.20181259. Each of these material models is associated, vertex by vertex, to a given material class, e.g., wax, skin, wood, plastic, and pa- per. The ground truth for the proposed invention could be stored in a look-up table associating for each vertex of the 3D mesh the related material information encoded as a label of a set of a prefix set of classes.

The nature of this training set may be typically synthetic, i.e., obtained by rendering of 3D models with material information and assets using illuminator, lens, and sensor models to describe the iToF camera system. By this rendering, the direct and global components, as well as the corresponding input iToF phasor image and auxiliary images may be obtained. A schematic representation of the data generation is depicted in Fig. 12 below.

Fig. 15 schematically shows a schematic representation of a training data generation method. The method comprises determining a simulated phasor image (iToF raw data) and corresponding direct and global components from a physically-based rendering based on an iToF sensor model and a di- rect/ global separation. The simulated phasor image is a synthetic phasor image.

A 3D model or scene 902 illuminated by an illumination profile 901 is rendered by a transient ten- derer 903 to produce a transient image X, such as a histogram that depicts the intensity over time. The transient image X is (virtually) processed by an iToF sensor model and optics 905 to generate iToF raw data 906 (a full-frame phasor image of the scene 902), e.g. synthetic dataset 1005 for train- ing the ML-based models. The illumination profile 901 is for example described by means of para- metric or non-parametric radiant source intensity profile. The iToF sensor model and optics 905 models characteristics of an iToF sensor, e.g. the correlation of the signal produced by the incident light on a pixel with the demodulation signal (DML in Fig. 1).

The transient image X is obtained by the transient tenderer 903 using a raytracing technique. This transient tenderer 903 (raytracer) is the core common element of the approach described here which, given a 3D model 902 representing the geometry and the material properties of a target scene and given the illumination pattern 901 representing how the spot iToF camera is illuminating the scene in a sparse way, provides time-resolved, transient rendering describing light transport for each of the pixels. This produces a transient image X which represents the scene response to an ideal, in- finitely short (in time) light pulse as observed and illuminated from the iToF camera view. The tran- sient image describes at each pixel a histogram of times of arrival of photons (x axis) vs. photon counts (y axis). The iToF model generates the iToF response given this per-pixel histogram.

A direct/ global separation 904 is performed on the transient image to separate it in direct and global transient components by bandpass filtering (see also Fig. 16) on the transient image generated by physically-based, time-resolved rendering. The direct/ global separation 904 may for example obtain the direct transient component by limiting the raytracing to a maximal number of one reflection. Ac- cordingly, the direct/ global separation 904 may for example obtain the global transient component by limiting the raytracing to more than one reflection, i.e. by disregarding any (“direct”) rays that are reflected only once on the scene. The direct and global transient components are fed separately to the iToF sensor model 905 to obtain a direct ground truth phasor 907 and a global ground truth phasor 908.

It should be noted that the transient image includes light intensity information and light path length information. Using this information may be possible to recreate the transient image.

Figs. 16 a and b show two exemplifying instances of this decomposition in the real part of the direct and global components, I_direct, and I_global of the ground truth phasor image as provided in a simu- lation of the data generation methodology described below. The iToF phasor image is the summa- tion of the two.

Fig. 17 schematically describes the result of the separation as performed in direct/ global separation 904 of Fig. 15. In other words, Fig. 17 depicts a transient (a histogram) at one generic pixel as re- ceived on the sensor plane, before computing the iToF response. A (virtual) transient image 906 ob- tained by raytracing comprises a direct light component and a global light component. Each pixel of the transient image 906 is associated with a respective histogram of photon counts over time. The histogram comprises events related to the direct light component and events related to the global light component. On the transient image, H x W x B, where B is the number of histogram bins, H is the height of the histogram bins and W is the width of the histogram bins. The direct/ global separa- tion separates these components into two different transient images, namely a direct transient image 907, and a global transient image 908.

The iToF raw data 906 and the ground truth phasor 907 and the global ground truth phasor 908 are used as training data in a training process to obtain the pretrained parameters of the ML-based model. In particular during training, the iToF raw data 906 (phasor image Z in Fig. 10) is fed to the ML-based model (702 in Fig. 10) as input and the ML-based model produces respective direct and global components as output. The parameters of the ML-based model are optimized until the direct and global components obtained by ML-based model are as close as possible to the ground truth phasor 907 and the global ground truth phasor 908. In the training phase this optimization is typi- cally done with a larger set of transient images obtained from multiple scenes with different objects, object positions, camera orientations, and so forth.

In the embodiment of Fig. 15, the ML-based model may be trained in a supervised way by using a training set comprising the iToF input Z and (optionally) auxiliary inputs 147, and the ground truth direct and global phasors Z_direct, Z_global at the desired location. This realizes instances of the map- ping which is learnt by the ML-based regression model. The ground truth phasor 907 and/ or the global ground truth phasor 908 may be used as training data in a train- ing process to obtain the pretrained parameters of the ML-based model.

The nature of this training set is synthetic, i.e., obtained by rendering of 3D models and assets using illuminator, lens, and sensor models to describe the iToF camera system. By this rendering, the ground truth direct and global are obtained, as well as the corresponding input iToF phasor image Z and auxiliary images 147.

Alternatively, the training set described in Fig. 15 above may be realized, i.e., obtained by recording data via an iToF camera system, while the ground truth data may be obtained by recording data via another device such as a structured light or LiDAR 3D scanner, and annotating the resulting mesh data with known material models. Also in this case, the schematic representation of the data genera- tion described in Fig. 14 above may be used with inputs being 3D models and assets recorded by a ground truth device. Still alternatively, this training set may be realized, i.e., obtained by recording data via an RGB or IR camera system, e.g., iToF signal amplitude, providing 3D reconstruction by means of dense struc- ture-from-motion/ multiview synthesis methods and annotating the resulting mesh data with known material models. For example, B. Attal et al., propose such dense structure-from-motion/ multiview synthesis methods at the published paper “T6RF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis,” Advances in Neural Information Processing Systems, vol. 34, 2021. These dense structure-from-motion/ multiview synthesis methods are also known in the state of the art such as COLMAP, or KinectFusion.

Also in this case, the schematic representation of the data generation described in Fig. 12 above may be used with inputs being 3D models and assets generated by a 3D reconstruction algorithm from IR or RGB camera data. This last approach may be considered self-supervised, i.e., the algorithmic pipeline itself provides data to train the network for this regression task.

It should be noted that the above-described model may be implemented for spot iToF illumination as well as for full filed iToF illumination, as long as the illumination profile and light shading (illumi- nator model) is provided to transient tenderer.

It should further be noted that in the embodiments of Figs. 4 to 12, direct — global separation is per- formed from full-frame spot-iToF phasor images by means of a ML-based method, using data gen- eration by raytracing to obtain ground truth that enables precise training of the latter ML-based method parameters.

In the embodiment of Fig. 17 the transient image 906 (see X in Fig. 15) for a single iToF camera per pixel is shown, wherein the abscissa represents the travel time of the emitted light in ns and the ordi- nate represents the intensity of the emitted light. Its first component (vertical straight line) is the di- rect transient image component X^i_rect and it is related to the light rays bouncing only once in the scene and directly into the iToF camera. The second component is the global transient image com- ponent, and it is related to all the light rays bouncing multiple times in the scene and which is re- ferred as X_giobal.

The transient image X generated by the transient tenderer 903 is fed to the iToF sensor model implemented by the iToF sensor and optics 905, which estimates the iToF output from it by emulat- ing iToF camera modulation and demodulation signals. These are convolved with the transient im- age to obtain a realistic camera response given the modulation waveform. Moreover, sensor-related noise and distortion sources, for example, thermal noise, lens, and sensor scattering, tap imbalance are also part of the iToF sensor model. As an example, the sensor model may consist of a matrix with four rows and as many columns as the time bins of the transient image X. Each row of 0 is a cosine function with a different internal phase shift (p 6 [0, π /2 , T, 3 π / 2 ]. Through the matrix multiplication m = ΦX, which holds at every pixel in the sensor array, simulate the well-known four-taps sampling of an iToF camera. Other noise sources, such as shot noise, can then be generated on m. From m we can then build the corresponding iToF phasor Z = I + jQ as well known in basic iToF principles, reading where the subscripts denote the corresponding internal phase shift (p. The application of this model indeed yields the input iToF phasor image Z.

The same exact procedure can be applied to the direct and global phasor images. The transient im- age is bandpass-filtered in its direct and global transient components, and these may be fed into the sensor model 0, yielding the ground truth phasor images Z_direct and Z_global.

It should be further noted that the spot or pattern illuminator may be generally modelled by its radi- ant source intensity profile RS I (θ, Φ ) in polar coordinates (horizontal/ vertical angles). This profile may be parametric, e.g., it may be explicitly generated by a grid of Gaussian pulses based on an ele- mentary periodic cell. Alternatively, it may be non-parametric, i.e., as measured from a photo-goni- ometer set-up providing a discretization of the RS I (θ, Φ ).

It should be further noted that in a case where a dToF is used the direct/ global separation may be implemented as shown in Fig. 17.

Fig. 18 shows a flow diagram visualizing a method for training a neural network, such as for exam- ple a deep neural network (DNN) to separate the direct and the global components, Ẑ_direct and Ẑ_global, of the spot iToF image data to estimate the geometry (e.g., the mesh) of an object and to infer the material properties of it.

At 1200, an image sensor acquires spot iToF data e.g. by illuminating a scene having objects with different materials. At 1201, direct-global separation is performed on the spot iToF data based on a set of pre-trained parameters to obtain an estimate of direct and global light component, Ẑ_direct and Zgiobal- At 1202, a corrected depth is obtained based on the direct light component Ẑ_direct. At 1203, direct/ global features are extracted from the direct and the global light component, Ẑ_direct and Zgiobal- At 1204, 3D reconstruction is performed on the corrected depth to reconstruct the geome- try (mesh) of a target object. At 1205, a per-point material classification/ segmentation on the cor- rected depth is performed to infer the material properties of the target object. Fig. 19 schematically describes an embodiment of a ToF device that can implement the processes of performing DGS separation based on an ML-based regression model and of performing 3D recon- struction and a per-point material classification/ segmentation to estimate the geometry (e.g., the mesh) of an object and to infer the material properties of it. The electronic device 1300 may further implement all other processes of a standard ToF/ spot ToF system, like I-Q value determination, phase, and amplitude determination. The electronic device 1300 may further implement a DGS algo- rithm, a reflectance sharpening filter, or the like. The electronic device 1300 comprises a CPU 1301 as processor. The electronic device 1300 further comprises an ToF sensor 1306 and a deep neural network unit 1309 connected to the processor 1301. The electronic device 1300 further comprises a user interface 1307 that is connected to the processor 1301. This user interface 1307 acts as a man- machine interface and enables a dialogue between an administrator and the electronic system. For example, an administrator may make configurations to the system using this user interface 1307. The DNN 1309 may for example be an artificial neural network in hardware, e.g. a neural network on GPUs or any other hardware specialized for the purpose of implementing an artificial neural net- work. The DNN 1309 may thus be an algorithmic accelerator that makes it possible to use the tech- nique in real-time, e.g., a neural network accelerator. The DNN 1309 may for example implement the ML-based regression model that realizes the processes described with regard to Fig. 3, Fig. 4, Fig. 5, Fig. 6, and Fig. 7 in more detail. The DNN1309 may optionally be a software. The electronic device 1300 further comprises a Bluetooth interface 1304, a WLAN interface 1305, and an Ethernet interface 1308. These units 1304, 1305 act as 1/ O interfaces for data communication with external devices. For example, video cameras with Ethernet, WLAN or Bluetooth connection may be cou- pled to the processor 1301 via these interfaces 1304, 1305, and 1308. The electronic device 1300 fur- ther comprises a data storage 1302, which may be the calibration storage, and a data memory 1303 (here a RAM). The data storage 1302 is arranged as a long-term storage, e.g., for storing the algo- rithm parameters for one or more use-cases, for recording iToF sensor data obtained from the iToF sensor 1306 the like. The data memory 1303 is arranged to temporarily store or cache data or com- puter instructions for processing by the processor 1301.

It should be noted that the description above is only an example configuration. Alternative configu- rations may be implemented with additional or other sensors, storage devices, interfaces, or the like. It should be further noted that alternatively the electronic device 1300 may be implemented with a digital signal processor (DSP) or a graphics processing unit (GPU), without limiting the present dis- closure in that regard. It should be further noted that a ToF sensor, a processor, or an application processor may imple- ment the processes for long depth detection range measurement of a spot or a pixel in an iToF sys- tem.

It should also be noted that the division of the electronic device of Fig. 19 into units is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, at least parts of the circuitry could be implemented by a re- spectively programmed processor, field programmable gate array (FPGA), dedicated circuits, and the like.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example, on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

In so far as the embodiments of the disclosure described above are implemented, at least in part, us- ing software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a com- puter program is provided are envisaged as aspects of the present disclosure.

The methods as described herein are also implemented in some embodiments as a computer pro- gram causing a computer and/ or a processor to perform the method, when being carried out on the computer and/ or processor. In some embodiments, also a non-transitory computer-readable record- ing medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be per- formed.

It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. Changes of the ordering of method steps may be apparent to the skilled person.

The method of Fig. 18 can also be implemented as a computer program causing a computer and/ or a processor to perform the method, when being carried out on the computer and/ or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the pro- cessor described above, causes the method described to be performed.

***

Note that the present technology can also be configured as described below. (1) A method comprising performing material sensing (603) based on an estimate Ẑ_direct) of a direct component and/or an estimate ( Ẑ_global) of a global component of ToF data (401, Z) acquired by a ToF sensor (301).

(2) The method of (1), wherein performing material sensing comprises obtaining material classi- fication and/ or segmentation information (603) based on an estimate ( Ẑ_direct) of a direct compo- nent and/ or an estimate ( Ẑ_global) of a global component.

(3) The method of (1) or (2), wherein performing material sensing (603) comprises material clas- sification (603) of a target object (203) or of a part of the target object (203).

(4) The method of anyone of (1) to (3), wherein performing material sensing (603) comprises performing feature extraction (601) on an estimate (Zcu_rect) of the direct component and/or an esti- mate ( Ẑ_global) of the global component on the ToF data (401, Z) to obtain direct and/or global fea- tures (602).

(5) The method of anyone of (1) to (4), wherein performing material sensing (603) comprises performing material sensing (603) per-point in the ToF data (401, Z).

(6) The method of anyone of (1) to (5), wherein the method comprises performing 3D recon- struction (605) to obtain a 3D model (606) and associating (607) the 3D model (606) with infor- mation (604) obtained from the material sensing (603).

(7) The method of anyone of (1) to (6), wherein the method comprises performing 3D recon- struction (605) based on the estimate (Ẑ_direct)' of the direct component to obtain a 3D model (606) and associating (607) the 3D model (606) with information (604) obtained from the material sensing (603).

(8) The method of (6), wherein the annotating (607) of the 3D model (606) is further performed based on meta data comprising the material classification/ segmentation information (604) of a target object (203) and/ or material properties of the target object (203).

(9) The method of anyone of (1) to (8), wherein performing material sensing (603) comprises applying a machine learning-based regression model (Model_θ; 303) to the estimate (Ẑ_direct) of the direct component and/ or the estimate (Ẑ_global) of ^e global component of the ToF data (401, Z) to obtain material classification/ segmentation information (604) and/ or material properties of the tar- get object (203).

(10) The method of anyone of (1) to (9), wherein the method further comprises applying a ma- chine learning-based regression model (Model_θ; 303) to the ToF data (401, Z) to obtain the estimate (Ẑ_direct) of direct light component of the ToF data (401, Z) and the estimate ( Ẑ_global) of the global component of the ToF data (401, Z).

(11) The method of anyone of (1) to (10), wherein the ToF data (401, Z) is a phasor image (Z).

(12) The method of anyone of (1) to (11), wherein the machine learning based regression model ( Model_θ, 303) in addition to the ToF data (401, Z) captured by the ToF sensor (301) takes auxiliary data (W) as further input.

(13) The method of anyone of (1) to (12), wherein the auxiliary data (W) is an infrared image (402) or a full frame image (402).

(14) The method of anyone of (1) to (13), wherein the machine learning-based model regression (Model_θ(Z)) is pretrained based on one or more ground truth images (907, 908) obtained based on direct/ global separation (904) of a transient image (X) of a model scene (902).

(15) A method for training a machine learning model (Model_θ ; 303) for material sensing, the method comprising generating training data comprising ToF data (401) and ground truth material labels.

(16) The method of (15), wherein the training data further comprises auxiliary data (W ).

(17) An electronic device comprising circuitry configured to perform material sensing (603) based on an estimate ( Ẑ_direct) of a direct component and/ or an estimate ( Ẑ_global) of a global component of ToF data (401, Z) acquired by a ToF sensor (301).

(18) The electronic device of (17), wherein the circuitry is configured to perform material sensing comprises obtaining material classification and/ or segmentation information (603) based on an esti- mate (Ẑ_direct) of a direct component and/ or an estimate (Ẑ_global) of a global component.

(19) The electronic device of (17) or (18), wherein the circuitry is configured to perform material sensing (603) comprises material classification (603) of a target object (203).

(20) The electronic device of anyone of (17) to (19), wherein performing material sensing (603) comprises performing feature extraction (601) on an estimate ( Ẑ_direct) of the direct component and/ or an estimate (Ẑ_global) of the global component on the ToF data (401, Z) to obtain direct and/ or global features (602).

(21) The electronic device of anyone of (17) to (20), wherein performing material sensing (603) comprises performing material sensing (603) per-point in the ToF data (401, Z). (22) The electronic device of anyone of (17) to (21), wherein the circuitry is configured to per- form 3D reconstruction (605) to obtain a 3D model (606) and associating (607) the 3D model (606) with information (604) obtained from the material sensing (603).

(23) The electronic device of anyone of (17) to (21), wherein the circuitry is configured to per- form 3D reconstruction (605) based on the estimate Z₍n_rect)' of the direct component to obtain a 3D model (606) and associating (607) the 3D model (606) with information (604) obtained from the material sensing (603).

(24) The electronic device of (22), wherein the associating (607) of the 3D model (606) is further performed based on meta data comprising the material classification/ segmentation information (604) of a target object (203) and/ or material properties of the target object (203).

(25) The electronic device of anyone of (17) to (24), wherein performing material sensing (603) comprises applying a machine learning-based regression model (Model_θ ; 303) to the estimate direct component and/ or the estimate (Ẑ_global) of the global component of the ToF data (401, Z) to obtain material classification/ segmentation information (604) and/or material prop- erties of the target object (203).

(26) The electronic device of anyone of (17) to (25), wherein the circuitry is configured to apply a machine learning-based regression model (Model_θ; 303) to the iToF data (401, Z) to obtain the esti- mate (Ẑ_direct) of the direct light component of the ToF data (401, Z) and the estimate (Ẑ_global) of the global component of the ToF data (401, Z).

(27) The electronic device of anyone of (17) to (26), wherein the ToF data (401, Z) is a phasor im- age (Z).

(28) The electronic device of anyone of (17) to (27), wherein the machine learning based regres- sion model (Model_θ; 303) in addition to the ToF data (401, Z) captured by the ToF sensor (301) takes auxiliary data (147) as further input.

(29) The electronic device of (28), wherein the circuitry is configured to perform, wherein the auxiliary data (147) is an infrared image (402) or a full frame image (402).

(30) The electronic device of anyone of (17) to (29), wherein the machine learning-based model regression (Model_θ (Z)) is pretrained based on one or more ground truth images (907, 908) ob- tained based on direct/ global separation (904) of a transient image (X) of a model scene (902).

(31) An electronic device comprising circuitry configured to generate training data comprising ToF data (401) and ground truth material labels.

Claims

1. A method comprising performing material sensing based on an estimate of a direct compo- nent and/ or an estimate of a global component of ToF data acquired by a ToF sensor.

2. The method of claim 1, wherein performing material sensing comprises obtaining material classification and/ or segmentation information based on an estimate of a direct component and/ or an estimate of a global component.

3. The method of claim 1, wherein performing material sensing comprises material classifica- tion of a target object or of a part of the target object.

4. The method of claim 1, wherein performing material sensing comprises performing feature extraction on an estimate of the direct component and/ or an estimate of the global component on the ToF data to obtain direct and/ or global features.

5. The method of claim 1, wherein performing material sensing comprises performing material sensing per-point in the ToF data.

6. The method of claim 1, wherein the method comprises performing 3D reconstruction to ob- tain a 3D model and associating the 3D model with information obtained from the material sensing.

7. The method of claim 1, wherein the method comprises performing 3D reconstruction based on the estimate of the direct component to obtain a 3D model and associating the 3D model with information obtained from the material sensing.

8. The method of claim 6, wherein the associating of the 3D model is further performed based on meta data comprising the material classification/segmentation information of a target object and/ or material properties of the target object.

9. The method of claim 1, wherein performing material sensing comprises applying a machine learning-based regression model to the estimate of the direct component and/ or the estimate of the global component of the ToF data to obtain material classification/segmentation information and/ or material properties of the target object.

10. The method of claim 1, wherein the method further comprises applying a machine learning- based regression model to the ToF data to obtain the estimate of the direct light component of the ToF data and the estimate of the global component of the ToF data.

11. The method of claim 1, wherein the ToF data is a phasor image.

12. The method of claim 1, wherein the machine learning based regression model in addition to the ToF data captured by the ToF sensor takes auxiliary data as further input.

13. The method of claim 12, wherein the auxiliary data is an infrared image or a full frame image.

14. The method of claim 1, wherein the machine learning-based model regression is pretrained based on one or more ground truth images obtained based on direct/ global separation of a transient image of a model scene.

15. A method for training a machine learning model for material sensing, the method compris- ing generating training data comprising ToF data and ground truth material labels.

16. The method of claim 15, wherein the training data further comprises auxiliary data.

17. An electronic device comprising circuitry configured to perform material sensing based on an estimate of a direct component and/ or an estimate of a global component of ToF data acquired by a ToF sensor.

18. An electronic device comprising circuitry configured to generate training data comprising ToF data and ground truth material labels.