WO2025245641A1

WO2025245641A1 - System and method for determining a three-dimensional model of an object using neural structured light

Info

Publication number: WO2025245641A1
Application number: PCT/CA2025/050762
Authority: WO
Inventors: Kiriakos Neoklis Kutulakos; David Brian Lindell; Seyed Parsa MIRDEHGHAN; Maxx WU; Wenzheng Chen
Original assignee: University of Toronto
Current assignee: University of Toronto
Priority date: 2024-05-31
Filing date: 2025-05-30
Publication date: 2025-12-04
Anticipated expiration: 2026-11-30

Abstract

There is provided a system and method for determining a three-dimensional model of an object using a projector and a light sensor arranged in a stereo configuration. The method including: receiving pixel intensity values captured by the light sensor for a plurality of images of a scene containing the object, wherein the images of the scene each capture different projection patterns emitted by the projector onto the scene; determining, using the pixel intensity values from a set of the plurality of images, correspondence between a projector plane and a camera plane to form a three-dimensional model of the object, the correspondence determined using a trained artificial neural network model that uses a combination of volume rendering and geometric representation; and outputting the three-dimensional model of the object.

Description

SYSTEM AND METHOD FOR DETERMINING A THREE-DIMENSIONAL MODEL OF AN OBJECT USING NEURAL STRUCTURED LIGHT

TECHNICAL FIELD

[0001] The following relates generally to three-dimensional (3D) mapping or imaging, and more particularly to a system and method for determining a three-dimensional model of an object using neural structured light.

BACKGROUND

[0002] Structured-light (SL) triangulation is one of the oldest and most widely used techniques for precise and reliable 3D shape acquisition. Although there are many other depth-sensing modalities, SL remains the most accessible way to capture millimeter (mm) and sub-mm accurate geometry for a broad range of materials and appearances. Typically, all that is needed is to arrange one camera and a projector in a stereo configuration, adjust their baseline, and capture images while a sequence of patterns is projected onto a scene. With multi-megapixel cameras readily available and inexpensive projectors having 4K or more resolution, SL acquisition of sub- mm-accurate, multi-megapixel depth maps is possible. Unfortunately, accurate SL is typically slow. In such approaches, many patterns are projected onto a scene while it remains stationary, making 3D acquisition laborious and restrictive.

[0003] At the same time, passively learning 3D scene representations directly from 2D images has been growing in popularity. These approaches use just a camera for acquisition and typically need a relatively large set of images even for desktop scenes. Their reliance on ambient illumination, however, is not sufficient for accurate surface reconstruction, especially for scenes that lack surface texture or have complex reflectance.

SUMMARY

[0004] In an aspect of the present invention, there is provided a method for determining a three- dimensional model of an object using a projector and a light sensor arranged in a stereo configuration, the method comprising: receiving pixel intensity values captured by the light sensor for a plurality of images of a scene containing the object, wherein the images of the scene each capture different projection patterns emitted by the projector onto the scene; determining, using the pixel intensity values from a set of the plurality of images, correspondence between a projector plane and a camera plane to form a three-dimensional model of the object, the correspondence determined using a trained artificial neural network model that uses a combination of volume rendering and geometric representation; and outputting the three-dimensional model of the object.

[0005] In a particular case of the method, the geometric representation comprises a signed distance field representation.

[0006] In another case of the method, the geometric representation comprises a parameterization with three-dimensional gaussians.

[0007] In yet another case of the method, the light sensor comprises a camera and the plurality of images comprises a plurality of Red-Green-Blue images.

[0008] In yet another case of the method, the projection patterns comprise structured light projection patterns.

[0009] In yet another case of the method, the determination of the correspondence using the trained artificial neural network model comprises forming a rendered camera image by volumetric surface parameterization represented using the signed distance field that incorporates the structured light projection patterns and forming a rendered structured light pattern from the captured images.

[0010] In yet another case of the method, the artificial neural network model is trained by minimizing a light sensor loss and a projector loss, the light sensor loss penalizes the difference between rendered images and measured images, and a projector loss penalizes the difference between rendered projection patterns and measured projection patterns.

[0011] In yet another case of the method, the rendered images and the rendered projection patterns incorporate at least one of: a cosine factor based on a gradient of the SDF; a reflectance term parameterized by a two-dimensional (2D) mulitilayer perceptron; and a residuals component parameterized by a two-dimensional (2D) multilayer perceptron, and wherein the outputted three- dimensional model of the object comprises a three-dimensional surface and per-pixel surface normals and reflectances.

[0012] In yet another case of the method, the rendered camera image is convolved with an optimized blur kernel.

[0013] In yet another case of the method, the set of images comprises three or four images. [0014] In yet another case of the method, the method further comprising determining an uncertainty of depth at each pixel using reverse scanning through comparing front-to-back and back-to-front volume rendering.

[0015] In yet another case of the method, the light sensor and projector are moved in order to project the light projection patterns and capture corresponding images from different spatial positions.

[0016] In yet another case of the method, a deformation map is estimated for the captured images of a dynamic scene.

[0017] In yet another case of the method, the method further comprising optimizing the projection patterns.

[0018] In another aspect, there is provided a system for determining a three-dimensional model of an object using a projector and a light sensor arranged in a stereo configuration, the system comprising one or more processors and a data storage, the data storage comprising instructions for the processor to execute: an input module to receive pixel intensity values captured by the light sensor for a plurality of images of a scene containing the object, wherein the images of the scene each capture different projection patterns emitted by the projector onto the scene; a geometry module to determine, using the pixel intensity values from a set of the plurality of images, correspondence between a projector plane and a camera plane to form a three- dimensional model of the object, the correspondence determined using a trained artificial neural network model that uses a combination of volume rendering and geometric representation; and an output module to output the three-dimensional model of the object.

[0019] In a particular case of the system, the geometric representation comprises a signed distance field representation or a parameterization with three-dimensional gaussians.

[0020] In another case of the system, the projection patterns comprise structured light projection patterns.

[0021] In yet another case of the system, the determination of the correspondence using the trained artificial neural network model comprises forming a rendered camera image by volumetric surface parameterization represented using the signed distance field that incorporates the structured light projection patterns and forming a rendered structured light pattern from the captured images. [0022] In yet another case of the system, the artificial neural network model is trained by minimizing a light sensor loss and a projector loss, the light sensor loss penalizes the difference between rendered images and measured images, and a projector loss penalizes the difference between rendered projection patterns and measured projection patterns.

[0023] In yet another case of the system, the rendered images and the rendered projection patterns incorporate at least one of: a cosine factor based on a gradient of the SDF; a reflectance term parameterized by a two-dimensional (2D) mulitilayer perceptron; and a residuals component parameterized by a two-dimensional (2D) multilayer perceptron.

[0024] In yet another case of the system, the geometry module further determines an uncertainty of depth at each pixel using reverse scanning through comparing front-to-back and back-to-front volume rendering.

[0025] In yet another case of the system, the light sensor and projector are moved in order to project the light projection patterns and capture corresponding images from different spatial positions.

[0026] In yet another case of the system, a deformation map is estimated for the captured images of a dynamic scene.

[0027] These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of the system and method to assist skilled readers in understanding the following detailed description.

DESCRIPTION OF THE DRAWINGS

[0028] A greater understanding of the embodiments will be had with reference to the Figures, in which:

[0029] FIG. 1 is a diagram of an example implementation of the present embodiments showing examples of scene images and projection patterns;

[0030] FIG. 2 is a diagram of an example implementation of the present embodiments showing imaging geometry and a representation;

[0031] FIGS. 3A through 3D show an example experiment illustrating surface and normals, and disparity error, for pixelwise decoding (top row) compared to the decoder of the present embodiments (referred to as ‘TurboSL’) (bottom row) for pattern families, where n refers to the number of projection patterns;

[0032] FIG. 4 show an example experiment illustrating reconstruction of a statue from n = 3 SL images using a la carte projection patterns of maximum frequency 128;

[0033] FIG. 5 is a diagram showing four example ray-marching configurations for the present embodiments;

[0034] FIG. 6 shows the results of an example experiment illustrating a comparison of a signed difference between front-to-back and back-to-front disparity maps (left) and between front-to-back and ground-truth maps (middle) for a wedge-like surface reconstructed from two sets of SL images, one captured (top) and one with extra 1% Gaussian noise added to the captured set (bottom);

[0035] FIGS. 7A through 7D show example experimental results of the present embodiments for different scenes, namely, a statue, a plush toy, a fabric animal, and a fruit bowl, respectively;

[0036] FIG. 8 is a representative diagram of a system for determining a three-dimensional model of an object using a projector and a camera arranged in a stereo configuration, in accordance with an embodiment; and

[0037] FIG. 9 is a presentative flowchart of a method for determining a three-dimensional model of an object using a projector and a camera arranged in a stereo configuration, in accordance with an embodiment.

DETAILED DESCRIPTION

[0038] For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the Figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practised without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein. [0039] Various terms used throughout the present description may be read and understood as follows, unless the context indicates otherwise: “or” as used throughout is inclusive, as though written “and/or”; singular articles and pronouns as used throughout include their plural forms, and vice versa; similarly, gendered pronouns include their counterpart pronouns so that pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender. Further definitions for terms may be set out herein; these may apply to prior and subsequent instances of those terms, as will be understood from a reading of the present description.

[0040] Any module, unit, component, server, computer, terminal or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or nonremovable). Computer storage media may include volatile and non-volatile, removable and nonremovable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.

[0041] The present embodiments leverage recent advances on neural 3D scene modeling to make SL image acquisition much faster. Specifically, by recasting grayscale multi-pattern SL as an inverse rendering problem, and by using a neural signed distance field approach to solve it, it is shown that just three or four projection patterns are enough to obtain highly detailed geometric and photometric information about an unknown scene: (1) sub- pixel- accurate stereo correspondences, (2) surface normals and reflectances, (3) residual contributions from ambient and indirect illumination, and (4) a proxy for the depth uncertainty at each pixel. This approach does not have to use pre-trained models and can rely only on the captured SL images for supervision. [0042] In contrast to conventional settings for neural scene modeling, few-pattern SL is an extreme case that involves just one camera viewpoint and a very small number of input images — as few as three. In this extreme case, one does not have the benefit of the implicit geometric constraints provided by multi-viewpoint image datasets, which have already been employed for 3D shape acquisition in conjunction with time of flight.

[0043] From the perspective of SL imaging, the approach of the present embodiments can be thought of as addressing the problem of SL decoding: given the n intensities observed at a pixel for n known projection patterns, the goal is to “decode” this information into a stereo correspondence on the projector’s plane. Other approaches perform this decoding independently at each pixel. Such approaches yield dense and highly-accurate correspondences for sufficiently large n but introduces significant noise and outliers as n approaches the theoretical minimum of three patterns.

[0044] To tackle the data-starved regime of few-pattern SL, a global approach is taken instead. Intuitively, even though the intensities observed at a single pixel may constrain the pixel’s correspondence rather poorly, the SL images as a whole constrain the scene far more strongly; its 3D shape, its appearance, and even the optical properties of the projector illuminating it. This observation is exploited by formulating SL decoding as the joint inference of per-pixel correspondences, normals, reflectances, ambient and indirect contributions, and of a blur kernel for the projector’s imperfect optics. This problem is referred to as inverse structured light.

[0045] The present embodiments provide a number of substantial advantages. First, an unexplored instance of inverse rendering that can break the longstanding SL trade-off between speed, accuracy and density of 3D measurements. Second, it provides neural representations based on signed distance fields (SDFs), which are uniquely suited for this task. Specifically, in contrast to other approaches, alternative volumetric formulations can easily overfit the limited image data in few-pattern SL, and 2D neural representations are sensitive to the frequency content of projection patterns and thus unsuitable as a general backbone for SL. Third, precise modeling of SL image formation, from foreshortening of incident irradiance to imperfect projector optics, are useful for pushing SL performance to the sub-pixel accuracy regime. Fourth, inverse SL can produce particularly accurate reconstructions when paired with SL patterns that optimize the distinctiveness of adjacent pixels. These patterns have received little attention in traditional SL because they produce 3D point clouds with many outliers among their very accurate inliers, making downstream geometry processing a major challenge. The inductive bias of the SDF representation, on the other hand, acts as an effective mechanism for outlier rejection that takes full advantage of the patterns’ inherent accuracy. In embodiments of the present disclosure, these patterns are combined with a neural method for inverse SL. Fifth, the present embodiments provide the first experimental demonstration of acquiring megapixel-resolution disparity, normals, reflectance, and indirect/ambient contributions from three-pattern structured light, achieving subpixel accuracy with a standard projector and grayscale camera running at up to 60 fps.

[0046] Consider a projector and camera in a stereo configuration in front of an unknown scene (FIG. 1 , left). It is sought to compute correspondences between their planes by capturing n > 1 images under n grayscale projection patterns. Unlike single-pattern SL whose spatial- smoothness assumptions limit 3D accuracy and spatial resolution, multi-pattern SL can potentially compute the correspondence of each camera pixel individually.

[0047] Referring now to FIG. 8, a system 50 for determining three-dimensional (3D) geometry of an object using a projector and a camera arranged in a stereo configuration, in accordance with an embodiment, is shown. The system 50 can be run on any suitable computing device, for example, on a general-purpose computing device, on a purpose-built controller, on remotely located servers (such as in a cloud-based arrangement), or the like. In some embodiments, the components of the system 50 are stored by and executed on a single computer system or device. In other embodiments, the components of the system 50 are distributed among two or more computer systems or devices that may be locally or remotely distributed.

[0048] FIG. 8 shows various physical and logical components of an embodiment of the system 50. As shown, the system 50 has a number of physical and logical components, including a processing unit 52, a data storage 54, a user interface 56, a device interface 60 and a local bus 80 enabling the processing unit 52 to communicate with the other components. The processing unit 102 executes various conceptual modules, as described herein in greater detail. The data storage 54 provides responsive data storage to the processing unit 52, including computerexecutable instructions for implementing the modules, as well as any data used by these services. The user interface 106 enables a user to provide input via an input device, for example a touchscreen or button(s). The user interface 106 can also output information to output devices to the user, such as a display and/or speakers. The device interface 110 permits communication with various devices, such as a projector and light sensor (such as a camera), as described herein. [0049] In an embodiment, the processing unit 52 can execute a number of conceptual modules, which can include an input module 70, a geometry module 72, and an output module 74. In some cases, the functions and/or operations of the conceptual modules can be combined or executed on other modules.

[0050] Referring now to FIG. 9, a method 100 for determining three-dimensional (3D) geometry of an object using a projector and a light sensor (such as a camera) arranged in a stereo configuration, in accordance with an embodiment, is shown. At block 102, the input module receives captured pixel intensity values from the camera for a plurality of images of a scene containing the object. The images of the scene each capture different structured light projection patterns emitted by the projector. In some cases, the geometry module further determines an uncertainty of depth at each pixel using reverse scanning through comparing front-to-back and back-to-front volume rendering. At block 104, the geometry module determines , using the pixel intensity values from a set of the plurality of images, correspondence between a projector plane and a camera plane to form a three-dimensional model of the object, the correspondence determined using an artificial neural network model that uses a combination of volume rendering and signed distance field representation. At block 106, the output module outputs the three- dimensional model of the object.

[0051] Referring to FIG. 1 , shown is an example implementation of the system 50 for accurate and fast 3D sensing. From three consecutive grayscale SL video frames of a dynamic scene, the system 50 infers a detailed 3D surface; per-pixel surface normals and reflectances; and a disparity map with sub-pixel accuracy. It also estimates residual contributions for indirect and ambient light, and provides a no-reference proxy for the disparity error at each pixel.

[0052] Suppose the scene is within the depth of field of both the projector and the camera so that any patterns projected onto it are in focus. In this case, an infinitesimal patch c on the camera’s plane will receive contributions from three types of light paths: (1) direct reflections of a projected pattern p_fe, (2) indirect light originating from the projector, and (3) contributions from other light sources in the environment. The intensity measured at a discrete pixel [i,j] is an integral over the pixel’s footprint of these three contributions: noise (1) where p is the infinitesimal patch on the projector plane corresponding to c; the convolution * p_k models optical degradations of the projection pattern, if any, due to the projector’s imperfect optics; r(c) and n(c) are the reflectance and unit normal of the surface point projecting to c; the cosine factor n(c) • i(c) accounts for surface foreshortening relative to the incident light direction i(c); and g(p' c) represents indirect light transport from a general patch p' on the projector’s plane to patch c on the camera.

[0053] The geometric relation between corresponding patches on the camera and projector can be conveniently expressed as a 2D displacement by assuming, without loss of generality, a rectified stereo configuration: where z(c) is the depth of the point projecting to c and the scalar e depends on the SL system’s geometry (i.e., baseline, lens focal lengths and physical dimensions of camera and projector pixels). In the following, the displacement in Equation (2) is referred to as the pixel’s disparity.

[0054] Due to the limited observations at each pixel, prior work on multi-pattern SL considers a simplified form of Equation (1) that ignores indirect light and imperfect optics, and treats camera and projector pixels as infinitesimally small. This eliminates all integrals from the equation, leaving just three unknowns per pixel (Equation (T1) in TABLE 1): the depth z(c), a reflectance r(c) that absorbs surface foreshortening, and a residual term a(c) that also absorbs contributions from ambient and indirect light. Unfortunately, while pixelwise inversion of this simplified model is possible when n is large enough, significant challenges remain for small n.

[0055] TABLE 1 summarizes basic approaches, scene representations and image formation models for inverse SL, appreciating that e = [0e]^T , Vf() = Vf()/ll Vf() II :

TABLE 1 [0056] Regarding near-minimal decoding, in theory, three patterns suffice to invert Equation (T 1) as long as they can be expressed as a non-intersecting curve in ir³. In practice, however, few- pattern SL performs poorly even for highly-optimized pattern sequences. This is because SL patterns are designed to tackle several competing objectives: providing a large range of unambiguous disparities; ensuring the codes of nearby projector pixels are distinguishable with high probability in presence of noise; and making indirect light approximately constant across patterns so it can be absorbed by the ambient term. These objectives become mutually contradictory when n is small, leading to unavoidable pixelwise decoding errors regardless of pattern choice. The system 50 advantageously overcomes this problem with a neural approach that inverts the full Equation (1), except for the indirect light integral, over the whole image.

[0057] Regarding sub-pixel-accurate decoding, in theory, three patterns suffice to invert Equation (T1) as long as they can be expressed as a non-intersecting curve in ir³. In practice, however, few-pattern SL performs poorly even for highly-optimized pattern sequences. This is because SL patterns are designed to tackle several competing objectives: providing a large range of unambiguous disparities; ensuring the codes of nearby projector pixels are distinguishable with high probability in presence of noise; and making indirect light approximately constant across patterns so it can be absorbed by the ambient term. These objectives become mutually contradictory when n is small, leading to unavoidable pixelwise decoding errors regardless of pattern choice. The system 50 advantageously overcomes this problem with a neural approach that inverts the full Equation (1), except for the indirect light integral, over the whole image.

[0058] Regarding pattern-agnostic decoding, in previous approaches on SL, decoding algorithms have been designed in conjunction with the projection patterns themselves. Given the wide range of SL encoding schemes available, one consideration is how to distinguish the performance of the encoding scheme from the performance of the decoder itself. To that end, a ZNCC (Zero- Normalized Cross-Correlation) decoder treats SL decoding as a general optimization problem and can be optimal under an additive noise model; i.e., it returns the maximum-likelihood pixel correspondence for any given sequence of projection patterns. Despite its theoretical guarantees and state-of-the-art performance, few-pattern SL is a substantial challenge for this decoder (see FIGS. 3A through 3D, top). By formulating decoding as an inverse rendering problem, the approach used by the system 50 can be equally agnostic to the SL patterns used, but far more accurate (see FIGS. 3A through 3D, bottom).

[0059] It will be appreciated that FIGS. 3A through 3D show that all 3D meshes are poor in this setting. The system 50 has far better accuracy for a la carte SL patterns, and there is a relatively small difference between n = 3 and n = 4 for that family. [0060] Regarding normal estimation and geometry processing, pixelwise decoders conflate reflectance and orientation, outputting unoriented point clouds. Thus, normal estimation and surface extraction is left to downstream pipelines, a challenging task for few-pattern SL, whose noisy disparities and outliers can be quite significant. In contrast, the system 50 determines disparities and normals jointly, taking advantage of the strong relation between orientation, appearance, and sub-pixel disparity (Equation (1)) present in the raw SL images. This is in the spirit of end-to-end processing of raw sensor measurements for rendering and reconstruction.

[0061] Regarding quantifying spatial uncertainty, SL decoding errors can vary a lot across scenes and even from pixel to pixel within a scene. These errors generally occur for two reasons. First, the signal-to-noise ratio (SNR) of individual pixels is not constant because of differences in reflectance, orientation, projector distance, and incident ambient/indirect light. Second, SL encoding schemes introduce error non-uniformities, to some extent, because the n-dimensional code vector assigned to some projector pixels may be less distinctive than others. These nonuniformities are especially pronounced in few-pattern SL where very few degrees of freedom are available for encoding position. As a result, variations in depth uncertainty can be significant even among camera pixels with identical SNR. The question of how depth uncertainty can be estimated from the SL images themselves is poorly understood, yet this information can be potentially very useful for geometry processing downstream. This can be partially addressed by determining two distinct reconstructions as proxies for uncertainty.

[0062] Equation (1) provides a restricted version of the general rendering equation, where a projector is the scene’s main light source and the projection patterns are designed to create images highly informative of the scene’s 3D shape. From this perspective, pixelwise SL decoding can be viewed as an instance of inverse rendering, but solved independently at every camera pixel.

[0063] Neural multi-view reconstruction solves the inverse rendering problem through a global optimization of scene appearance and geometry, which are each parameterized using neural networks. The optimization generally follows an analysis-by-synthesis approach, wherein (1) an image formation model is used to render camera images based on the current estimates of scene appearance and geometry, (2) the rendered images are compared to the multi-view captured dataset, (3) gradients are computed to update scene geometry and appearance, and (4) the above steps are repeated iteratively to improve reconstruction fidelity. [0064] With respect to application to inverse SL, applying neural reconstruction involves choosing how to simplify Equation (1) and parameterize its various terms. TABLE 1 summarizes three general options.

[0065] A first option is to use a decoding approach based on 2D multi-layer perceptrons (MLPs) to invert the same image formation model employed by pixelwise-decoding methods (Equation (T2) in TABLE 1). In this case, the MLPs learn to map continuous-valued 2D input coordinates to the corresponding values of depth, reflectance, and residual contributions, with a global representation for geometry and appearance.

[0066] A second option is to rely on a 3D neural representation that employs MLPs or featurebased networks to map 3D input coordinates to values of an opaque density field, which models the differential probability of camera ray termination at each point in space. This allows use of a more general, albeit approximate, image formation model based on the volume rendering equation (Equation (T3) in TABLE 1). Implementing this equation to render the intensity of a camera pixel c involves marching along its 3D ray, retrieving the intensity of the corresponding projector pixel p_k(c - je) for each 3D point along the ray, and alpha-compositing these intensities based on the accumulated density. The result of this integration is scaled by the reflectance r(c) and combined with the residual term a(c), both of which are represented by 2D MLPs.

[0067] Neither of the above two options proved effective in the experiments conducted by the present inventors. In particular, contrary to their success in multi-view 3D reconstruction, 2D MLPs are found to fail to recover accurate geometry when applied to inverse SL, especially when high- frequency patterns are used for projection (FIG. 4, middle). Intuitively, using a depth map representation for inverse SL is challenging because it explicitly associates each camera pixel with a single depth and thus a single projector pixel. This fails to handle the ambiguities inherent in SL with high-frequency patterns, where image noise can make individual projector pixels less distinguishable. Volumetric approaches, on the other hand, are too flexible’, they can reproduce the (very few) SL images given as input very accurately, but extracting a surface from the volume involves accumulating density along individual camera rays and finding the expected ray termination distance, which yields inaccurate geometry (FIG. 4, right). The third option, used by the system 50, draws on the advantages of both 2D and volumetric representations by modeling surfaces explicitly while flexibly handling ambiguity. [0068] The system 50 uses a decoding approach that employs a hybrid representation based on volume rendering and a signed distance field representation. Geometry is represented using an efficient feature-based network to map input 3D coordinates to the values of an SDF. The SDF values are then converted to density; for example, as provided in Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction, in NeurlPS, 2021. Camera images are rendered by volume rendering according to Equation (T4). This representation provides an explicit model for surfaces, defined as the zero level set of the SDF.

[0069] The data-starved conditions in which the system 50 operates means exploiting the available constraints in the optimization for better 3D fidelity. To that end, forward and reverse rendering procedures that enforce consistency of the scene representation and the projector blur kernel with the SL input images are introduced, as well as the patterns that produced them.

[0070] In the forward-rendering direction, a camera image is reconstructed according to Equation (T4). Rays from the camera center of projection, through the image plane, are provided towards and into the scene (FIG. 5, bottom left). An artificial neural network, with 3D points sampled along the ray, is queried to recover the corresponding SDF values and density. Then, these points are projected into the projector plane, the projector pattern’s intensity is retrieved, and an alpha- composite of the resulting intensities along the ray is determined. A cosine factor, • i(c,z), is incorporated into the rendering along the ray by calculating the SDF’s gradient to query surface normals. The result of the integral is scaled by a learned reflectance term r(c) parameterized by a two-dimensional (2D) multilayer perceptron (MLP). An additional 2D MLP can be used to parameterize the residual component a(c), which absorbs ambient and global illumination and any other unmodeled contributions. The 2D and 3D artificial neural networks can be optimized to minimize differences between the rendered and captured camera images, and after optimization, a mesh can be extracted from the SDF.

[0071] To account for projector non-idealities, the known projection patterns can be convolved with a single kernel, b, before rendering the camera image. This kernel is optimized with the scene representation using just the SL input images.

[0072] In the reverse-rendering direction, the projector pattern is reconstructed according to Equation (T5) using the same 2D and 3D neural networks described above. Rays through each projector pixel are marched along and the 3D points along each ray are projected into the camera image plane to retrieve the corresponding camera measurement, i_fe(p + je) (FIG. 5, bottom right). The reflectance, cosine factor, and residual components can be incorporated, and the neural networks can be optimized to minimize the difference between the rendered and projected patterns.

[0073] To capture reconstruction uncertainty, two separate scene models can be optimized, one using bidrectional rendering and one by scanning the volume in a back-to-front order (FIG. 5, top). This approach is advantageous because, in SL, images are captured from a single viewpoint, and thus, each camera ray encounters one surface at most (it is assumed that there are no semitransparent surfaces). Effectively, the volume rendering approach finds projector-pattern correspondences by scanning along the epipolar line from one direction. As a camera ray is marched along in front-to-back volume scanning, points in one direction along an epipolar line on the projector plane are traced out (FIG. 5, bottom left). Back-to-front scanning proceeds in the same fashion, but starts from the other side of the epipolar line and scans in the opposite direction (FIG. 5, top left). This bidirectional scanning is unique to inverse SL and cannot be applied to conventional multi-view reconstruction where camera rays can pass through multiple reconstructed surfaces.

[0074] In empirical observations conducted by the present inventors, pixelwise subtraction of reconstructed front-to-back and back-to-front disparity maps provides a proxy for the actual disparity error; i.e., it is highly correlated with the pixelwise disparity error against an accurate reference (FIG. 6). This type of uncertainty modeling could be useful for downstream tasks and is reminiscent of reference-free 3D reconstruction metrics. For example, in structural biology and other fields, the Fourier shell correlation metric similarly assesses correlation between two independently reconstructed volumes. This proxy is used to remove low-confidence pixels in the experiments of this disclosure.

[0075] FIG. 6 shows that the difference cross-sections show agreement to within a fraction of a pixel, and the scatter-plots demonstrate strong correlation across all disparity errors. The SL patterns used were the same as shown in FIG. 4.

[0076] The decoder used by the system 50 can be trained by minimizing the following losses with respect to the parameters of the 2D and 3D neural networks. A camera loss, £_Cam. penalizes the difference between the rendered and measured camera images, and a projector loss, £_proj , operates similarly for the projector patterns: -^cam ^— Sfc,c II ifc(^c) lfc(^c) II > -^proj ^— Sfc,p H (B * Pfc)(p) Pfc(P) II- (3)

[0077] A loss _mask is incorporated to remove density along camera rays corresponding to shadowed parts of the scene and to enforce that rays passing through points illuminated by the projector are completely attenuated by density (to prevent transparent surfaces). A binary mask M(c) for illuminated camera pixels is calculated by applying a threshold on the difference between the brightest and darkest pixel values across patterns (shadowed pixels show little variability across patterns). The loss is computed as where BCE is the binary cross entropy loss, and the integral argument computes the total attenuation along the ray. Last, an eikonal penalty is incorporated to enforce that the gradient of the SDF be equal to one, and an L1 penalty is added on densities sampled along the rays to avoid regions of spurious non-zero density:

Aeik = Zz,c (II Vf(c, z) ||₂- l)², _sp = Sz,c exp(-|f(c,z)|). (5)

In practice, both L1 and L2 penalties for the projector and camera losses can be applied, and a weighted combination of all losses can be determined.

[0078] In some cases, the system 50 can be bult upon the NerfAcc framework, augmented to include the hybrid SDF and volumetric surface parameterization. Each model can be trained by sampling rays from both the camera and projector and performing forward and backward rendering. In example experiments, training for n = 3 input images took 12 minutes and used 10GB of VRAM on an RTX6000 GPU, on par with state-of-the-art neural rendering methods. To avoid numerical instability because of the division in Equation (T5), only camera loss was used for the first 1000 iterations and the projector loss was added until convergence. To capture uncertainty, two models for each scene were trained using front-to-back and back-to-front volume scanning, and all surfaces are rendered using the zero level set of the front-to-back-trained model. To model projector blur, two separable one-dimensional filters of size 11 pixels were optimized, and their outer product was used as the blur kernel. While the camera mask can be estimated from input images, the projector mask is unknown. Randomly sampling the projector rays with a constant “background” intensity leads to the creation of spurious densities. To avoid this, it is assumed that scene points at infinite depth along the ray through a projector pixel have reflectance equal to the projection pattern’s intensity at that pixel. This allows the projector loss to carve out the space.

[0079] FIG. 1 shows several results from two SL system configurations, one geared toward 60 fps image acquisition (IDS UI-3240CP camera) and one for 30 fps acquisition (Prosilica GT1920c camera). The same off-the-shelf projector (LG PH550 LED mini projector) and the same baseline were used for both configurations. To demonstrate the rapid image acquisition capabilities of the system 50, live video data was used to reconstruct a dynamic scene exhibiting non-rigid motion. Three frequency-128 a la carte patterns were repeatedly projected onto a hand-held book, while capturing synchronized 30 fps video with the camera. FIG. 1 (row 4) shows three snapshots of the reconstructions.

[0080] An ablation study of several components of the decoder of the system 50 was conducted and shown in FIG. 7A. SL images of a statue with three a la carte patterns of maximum frequency 32 were captured, and the scene was reconstructed using only the camera loss. Without the other components, the approach fails to accurately model the captured images, and spatial structures from the projector patterns leak into the reconstructed surface. Adding the projector loss and cosine factor improved the reconstructed surface and eliminated some of the minor banding artifacts from the geometry. Finally, incorporating the learnable projector blur kernel yielded a low image reconstruction error across most of the statue, and the recovered geometry closely matched the ground truth.

[0081] Although the system 50 was able to achieve sub-pixel-level accuracy with as few as three SL patterns (FIG. 3), increasing the number of patterns improves the accuracy of the recovered geometry. FIG. 7B shows a plush toy captured using 4, 5, and 6 a la carte patterns with maximum frequency 32. While using four patterns already reconstructs geometry at the sub-pixel level, additional patterns improved the smoothness of the surface normals and revealed additional wrinkles and fine details in the fabric of the toy. Inspecting the disparity error maps also shows that the reconstruction improves in many areas, albeit with some outliers.

[0082] FIG. 7C shows the outputs of the system’s 50 bidirectional rendering scheme, including surfaces and normals recovered using front-to-back and back-to-front rendering. The scene was reconstructed using four patterns with a maximum frequency of 64. While both rendered surfaces capture similar details, the reconstructions disagree in areas such as the legs and ears. As can be seen, the no-reference error map computed by subtracting the front-to-back and back-to-front disparity maps helped identify regions of high ground-truth error. Although the no-reference error map overestimated the ground-truth error in some regions (e.g. the right ear), it contained very few false negatives, i.e., points where the ground-truth error was underestimated. By using a threshold on this conservative no-reference error map, an inlier mask that identified the high- confidence 3D points and normals was obtained.

[0083] Scenes exhibiting strong indirect lighting effects are a substantial challenge for SL systems. FIG. 7D shows results of the experiments using the system 50 for a bowl filled with a pumpkin and two clementines, using three SL patterns of maximum frequency 64. The impact of indirect light was particularly strong near the objects’ boundaries deep inside the bowl, where diffuse interreflections contribute the most. The residual term does capture their contributions (e.g. lower edge of pumpkin, boundary of uppermost clementine), along with unmodeled stripelike contributions from the projector. Despite these effects, the no-reference metric allowed automatic exclusion of geometry near the boundary where depth errors were most severe, yielding high quality geometry despite them. A similar behavior can be observed in the results of FIG. 1 , near the book’s crease.

[0084] In summary, FIG. 7A shows the impact on image error of including the camera loss, projector loss, cosine factor, and projector blur terms for three patterns of maximum frequency 32. Including each term in sequence improves image and 3D reconstruction fidelity (arrows). FIG. 7B shows that increasing the number of patterns up to 6 further improves reconstruction accuracy. FIG. 7C shows reconstructions using bidirectional rendering have strong agreement in some areas, less so in others. Regions where the reconstructions differ correlate with large ground-truth error, providing a conservative estimate of reconstruction uncertainty. FIG. 7D shows that the system 50 captures fine geometric details in this challenging scene with indirect lighting; areas affected by indirect light can be observed in the residuals image (secarrows).

[0085] Using neural inverse rendering from the perspective of SL yields unique advantages. While much recent work in inverse rendering focuses on what can be achieved using images captured from tens to hundreds of viewpoints, SL imaging using the system 50 provides a number of advantages. For example, careful camera-projector modeling and artificial neural network rendering techniques can enable robust, sub-mm 3D reconstruction from a single viewpoint with commodity hardware. In this way, the present disclosure provides an approach to turn a noisy and fragile active triangulation technique, three-pattern structured light with a grayscale camera, into a fast and powerful tool for 3D capture. This approach is able to output sub-pixel accurate disparities at megapixel resolution, along with reflectance, normals, and a no-reference estimate of its own pixelwise 3D error. To achieve these advantages, the system 50 formulates structured- light decoding as a neural inverse rendering problem. Despite having just, for example, three or four input images, which can be all from the same viewpoint, the example experiments show that this problem can be tractably solved using the system 50. The approach implemented by the system 50 can combine: (1) a precise image formation model, (2) a signed distance field scene representation, and (3) projection pattern sequences optimized for accuracy instead of precision. The system 50 can be used to reconstruct a variety of complex scenes from images captured at up to 60 fps with a camera and a common projector. The example experiments highlight the system’s 50 ability for dense and highly-accurate 3D acquisition from data captured in fractions of a second.

[0086] In some cases, given a fixed set of structured light patterns, the system 50 can optimize the geometry representation and the appearance representations. The geometry representation can be characterized with a neural signed distance field, in its original form, or other representations, such as 3D Gaussian Splatting. The appearance representations can be characterized with two MLPs for per-pixel reflectances, and residual contributions, in its original form, or other representations, such as per-3d-point BRDF. In some cases, the system 50 can optimize the projection patterns simultaneously, or in conjunction, with the determination of the three-dimensional model. This can be performed by repeatedly projecting patterns, reconstructing the scene, evaluating the loss function, and updating the patterns and scene representations accordingly. Such an approach can be implemented by synthesizing the measurements in a simulation environment, or by incorporating the imaging system in the optimization loop.

[0087] The following provides an example implementation of the system 50 and some results of associated example experiments; however, it is to be understood that the described example implementation is not intended to constrain the subject matter of the present embodiments.

[0088] In a particular case, the machine learning model can be built using NeuS implementation, which incorporates the NerfAcc backbone for efficient sampling. The model can consist of three Instant-NGP networks (MLPs with multi-resolution hash encodings) for the signed distance function (SDF), reflectance, and residual contributions of indirect or ambient illumination. The SDF network takes as input a 3D point in space and outputs its SDF. The reflectance and residual networks query 2D points on the image plane and output the points’ reflectance and residual contributions. For all three networks, the number of hash feature grids and the feature size for each level can be set to 16 and 2, respectively. The resolution of the coarsest grid can be 16, and each level can have a scale factor of 1.45. The base MLPs for all three networks can have 2 hidden layers with width 64 and a ReLLI activation. The reflectance and residual networks employ sigmoid activations at the output to constrain the values between zero and one. The SDF network output is unconstrained.

[0089] The NerfAcc backbone maintains an occupancy grid to omit empty regions for efficient sampling. The grid’s resolution can be set to 128 in each dimension, and the density threshold for pruning can be set to 10-3. 1024 samples can be used per ray to render each pixel, where some of these samples are skipped if they fall within empty regions of the occupancy grid. All the scenes can be scaled and optimized within a bounding box of radius 1.5.

[0090] The loss function used to train the model can include five main components: (1) the camera loss that compares the rendered image with the input structured light (SL) images; (2) the projector loss that compares the rendered pattern with the input projection patterns; (3) the sparsity loss that encourages sparse reconstruction leading to fewer regions of spurious density; (4) the mask loss that encourages the model to converge to an opaque surface for any ray inside the mask and empty space for rays outside the mask; and (5) the Eikonal loss to regularize the SDF. The camera and projector losses are both weighted sums of L1 and L2 penalty functions, where the weights can be set to 1.0 and 10, respectively. Adding the L1 loss improves the reconstruction in low reflectance regions. For the mask loss, the mask can be estimated by thresholding the difference between the minimum and maximum pixel intensities across all the captured images. If the difference is higher than a certain threshold, the pixel can be considered to be visible to the projector and set the mask value to one (and otherwise to zero). The weights of the camera loss, projector loss, sparsity loss, Eikonal loss, and mask loss can be set to 1.0, 1.0, 0.01 , 0.1 , and 0.1 , respectively.

[0091] During experiments, the models usually converged in 10k iterations which takes around 12 minutes on the GPU used in the experiment. Initially, in each iteration, there was a random sampling of 256 rays from the camera and projector planes, and render their respective images. As the optimization progresses, the number of effective samples per ray decreases from the original 1024 samples because pruned-out regions in the occupancy grid are no longer sampled. Therefore, the number of rays were dynamically increased during each training iteration to keep the total number of samples constant over the course of training (without exceeding a maximum of 8192 rays). [0092] In most implementations, there are challenges to incorporating projector loss. For example, accounting for noise when rendering the projector pattern. While the input projector patterns are noise-free, the camera images include measurement noise. Rendering the projector pattern with reverse rendering can result in a noisy pattern. Further, this noise is divided by the product of the reflectance and a cosine similarity term. Therefore, the system 50 can multiply the per-pixel mean squared error loss of the rendered projector rays by the square of this scale factor. The per-pixel L1 loss of the rendered projector rays can be multiplied by the absolute value of this scale factor. Scaling the loss functions in this way puts more weight on rays corresponding to higher albedo (defined as reflectance multiplied by the cosine similarity term) and reduces the contribution of noisy pixels to the reconstruction

[0093] In another example, there can be numerical instability of the gradients. Rendering the projector rays involves a division by a scale factor that includes the reflectance and the cosine similarity term. This division, particularly in the initial stage of optimization, leads to exploding gradients and inhibits convergence during optimization. To address this, the system 50 can use two solutions. First, clamping the denominator to a minimum of 0.1 and maximum of 1 . This zeros the gradient when the denominator is smaller or greater than the threshold. Second, the system 50 can, for example, begin the optimization using only the camera loss for 1000 iterations before switching to use both the camera and projector losses for the remaining 9000 iterations.

[0094] In another example, there can be a lack of mask for projector image plane. Unlike the camera image, there is no convenient way to estimate a mask for the projector. Naively sampling from the projector plane and evaluating the loss between the input projector pattern and the rendered one leads to creation of density in empty space along each projector ray. To avoid this, the system 50 can composite the rendered image onto a background that is identical to the projector pattern. This ensures that zero density regions are not penalized, leading to non-zero density only in regions that affect the camera image.

[0095] In some cases, the system 50 can optimize the projector blur kernel. In theory, the projector can be described using a pinhole model. However, in practice, the mapping from the projector input to output is a manufacturer-dependent black-box function and involves a series of operations such as blurring or sharpening. The projector can be adjusted to make it in focus and calibrate the projector to reduce the lens distortion. On top of that, a blur kernel b can be optimized to account for any additional black-box processing. The blur kernel can be defined as a KX K matrix and applied to the whole projector pattern, assuming it is spatially invariant. To reduce the number of parameters, the system 50 can optimize two separable one-dimensional filters b_x, b_y e ]R^K, and use their outer product as the blur kernel b: b = b_xb

[0096] The kernel generally has extra constraints: first, the center of mass should be located in the center of the kernel; and second, the sum of the kernel should be equal to one to conserve the energy. Therefore, for each one-dimensional filter, the system 50 can apply re-weighting and normalization to satisfy the constraints. Specifically, the filter can be broken down into 3 parts: left half vector, center element, and right half vector:

[0097] where K is an odd number and b_t is the i -th element of filter b_x for i e {-m, -1,0,1, . . . , m}. The system 50 learns only the right and left parts and adjusts the center element to satisfy the summation constraint. More specifically, the system 50 rescales b to match b^ to keep the kernel centered each time it is computed, b™ can be defined to be equal to one minus the sum of b^ and b to constrain the filter sum to be equal to one. These additional update rules are given as:

[0098] where {b^, b"¹, bj} are the normalized left, middle and right parts of b_x. The system 50 computes b_y in the same fashion.

[0099] In some cases, a mesh can be extracted from depth predictions based on pixelwise decoding and the SDF predicted using the presently described approaches. Meshing the pixelwise decoder output can include, first, generating a depth map based on the pixelwise decoder outputs. Then, the system 50 backprojects the depth map to calculate point cloud points based on the calibrated camera intrinsics. The point cloud points are converted to a triangle mesh by adding edges to connect neighboring points (based on their positions in the depth map). Any point cloud points that fall outside the mask can be omitted from the meshing procedure used to compute the mask loss.

[0100] In other cases, to mesh the SDF predicted by the system 50, a per-pixel depth can be estimated using the expected ray termination distance. The system 50 can then backproject the depth map points to recover point cloud points using the calibrated camera intrinsics, and add edges to connect neighboring points based on the point positions in the depth map. In this case, the system 50 does not not mesh any points falling outside the mask used for the mask loss.

[0101] In some cases, the system 50 can use the above depth-based meshing procedure and a Blender Tenderer to create the surface visualizations. Experimental results demonstrate that the SDF predicted by the system 50 can be meshed directly without the depth map intermediary. Specifically, using marching cubes with a resolution of 512³ to extract the zero level set of the predicted SDF. This extracts some spurious surfaces that are in occluded regions not visible to the camera or projector. To remove these surfaces, the system 50 can iterate over all vertices in the extracted mesh, project a ray from the vertex to the camera center of projection, then check for intersections with the extracted mesh. If the ray intersects the extracted mesh, it indicates that the vertex is occluded and is invisible. All such occluded vertices and all triangle faces that include at least one of the removed vertices can be removed. Further, vertices and faces whose projections onto the camera image plane fall outside the mask used for the mask loss can be removed.

[0102] In some cases, the system 50 can determine a second reconstruction that can be used to compute an error proxy for the front-to-back reconstruction. The present inventors determined a correlation between the actual disparity error and the error proxy, and that the error proxy can be used to remove low-confidence regions in the final reconstruction. The error proxy can be determined as the difference between the front-to-back and back-to-front reconstructed disparities. Intuitively, it can be expected that for accurately predicted pixels, the outputs of both rendering schemes should agree (i.e., they predict the same disparity). The present inventors determined that this intuition holds, especially for a la carte patterns for which the error proxy and the actual disparity error are highly correlated. [0103] The example experiments illustrated that both front-to-back and back-to-front renderings show a strong correlation between the reconstruction error and the error proxy, and this correlation is particularly strong for a la carte patterns, which produce the most accurate results. The error proxy can be used to reliably identify erroneous pixels by setting an error threshold; whereby, the error proxy may be useful for outlier rejection, i.e. , by identifying regions in the error proxy with the largest magnitude.

[0104] The example experiments determined that there are horizontal and vertical structures in scatter plots of the error proxy versus disparity error. The vertical structures correspond to pixels whose actual disparity error is close to zero; in this case the disparity error is small but the front- to-back and back-to-front disparity predictions disagree. Hence, the vertical structures correspond to false negatives when using the error proxy to identify accurately predicted disparities. The horizontal structures comprise a very sparse set of pixels that are actually reconstructed incorrectly, but the back-to-front and front-to-back disparity predictions show close agreement. Thus, these points correspond to false positives when using the error proxy to identify accurately predicted disparities. Overall, the horizontal and vertical structures contain a much sparser set of pixels compared to the dominant diagonal structures, which indicate the strong correlation between the error proxy and the actual disparity error.

[0105] Additionally, the example experiments determined that there was bias in the back-to-front and front-to-back disparity predictions. Comparing the correlation plots and the signed disparity errors across different patterns illustrated that each reconstruction tends to be biased towards one side of the ground truth surface. The back-to-front predictions were biased towards the back of the ground truth surface (since the disparity error is mostly positive), and the front-to-back predictions were biased towards the front side of the ground truth surface. This inherent bias distinguished these two rendering schemes from two random trials of the same optimization.

[0106] The error proxy is a strong indicator of the actual reconstruction errors. Therefore, it can be used to identify pixels with accurate disparity predictions and to remove uncertain regions from the reconstructed surface. For example, the error proxy can be used to define an inlier mask for the confident pixels, and that mask is used to remove the uncertain regions. The system 50 can determine an inlier mask by evaluating the median error proxy value m within the mask used for the mask loss. The mask is given by thresholding the error proxy values to those that are less than 100m for a la carte patterns and less than 20m for other patterns. The threshold can be calibrated to keep inlier predictions with a disparity error of less than 10 pixels [0107] While the embodiments described herein generally refer to a static camera and projector pair, it should be understood by a person skilled in the art that the approach of the present embodiments can be used to output 360-degree 3D reconstruction of the scene. Such reconstruction can be performed by moving camera-projector pairs, continuously projecting patterns, and capturing structured light images. Similar bidirectional rendering and neural scene representations, as described herein, can be employed to accurately estimate the scene geometry and appearance from the stream of data. In its simplest form, as an example, this can be implemented by parameterizing a single signed distance field network for geometry, and neural representations for per-frame reflectance and residual maps. The appearance model can be further extended to a single neural representation for BRDF function for every 3D point, enabling capture of relightable 3D assets. In the context of a moving cam era- projector pair, extra scene parameters, for example, a deformation map for dynamic scenes, can be modeled and estimated from the captured stream of structured light images.

[0108] Additionally, while the embodiments described herein generally use neural signed distance field representation for the geometry and 2D MLPs or estimating reflectance and residual contributions, it is understood that these representations can be replaced by other suitable representations; for example, 3D Gaussian Splatting, multi-layer perceptrons for modeling the surface BRDF function, or the like.

[0109] Additionally, the model used by the present embodiments for geometry and appearance can be initialized using any suitable techniques; for example, initialized with noisy estimates from pixel-wise correspondence matching using ZNCC to potentially speed up training and convergence.

[0110] In some cases, in addition to the camera-projector pair, there can be other types of sensors or illumination sources for capturing auxiliary data used for optimizing the geometry and appearance.

[0111] In some cases, instead of a single camera and projector pair, there can be multiple cameras and projectors, collectively optimizing a unique geometry representation.

[0112] While the above disclosure generally refers to cameras, it should be understood that any suitable light sensors can be used instead of the camera. Further, while the above disclosure generally refers to light projectors, it should be understood that any suitable illumination source can be used to project the pattern, such as one or more lasers; whereby such illumination sources can generate any suitable projection pattern which may or may not be a structured light pattern. [0113] Although the foregoing has been described with reference to certain specific embodiments, various modifications thereto will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the appended claims.

Claims

1. A method for determining a three-dimensional model of an object using a projector and a light sensor arranged in a stereo configuration, the method comprising: receiving pixel intensity values captured by the light sensor for a plurality of images of a scene containing the object, wherein the images of the scene each capture different projection patterns emitted by the projector onto the scene; determining, using the pixel intensity values from a set of the plurality of images, correspondence between a projector plane and a camera plane to form a three- dimensional model of the object, the correspondence determined using a trained artificial neural network model that uses a combination of volume rendering and geometric representation; and outputting the three-dimensional model of the object.

2. The method of claim 1, wherein the geometric representation comprises a signed distance field representation.

3. The method of claim 1, wherein the geometric representation comprises a parameterization with three-dimensional gaussians.

4. The method of claim 1, wherein the light sensor comprises a camera and the plurality of images comprises a plurality of Red-Green-Blue images.

5. The method of claim 1, wherein the projection patterns comprise structured light projection patterns.

6. The method of claim 1, wherein the artificial neural network model is trained by minimizing a light sensor loss and a projector loss, the light sensor loss penalizes the difference between rendered images and measured images, and a projector loss penalizes the difference between rendered projection patterns and measured projection patterns.

7. The method of claim 6, wherein the rendered images and the rendered projection patterns incorporate at least one of: a cosine factor based on a gradient of the SDF; a reflectance term parameterized by a two-dimensional (2D) mulitilayer perceptron; and a residuals component parameterized by a two-dimensional (2D) multilayer perceptron, and wherein the outputted three-dimensional model of the object comprises a three- dimensional surface and per-pixel surface normals and reflectances.

8. The method of claim 6, wherein the rendered camera image is convolved with an optimized blur kernel.

9. The method of claim 1 , wherein the set of images comprises three or four images.

10. The method of claim 1, further comprising determining an uncertainty of depth at each pixel using reverse scanning through comparing front-to-back and back-to-front volume rendering.

11. The method of claim 1, wherein the light sensor and projector are moved in order to project the light projection patterns and capture corresponding images from different spatial positions.

12. The method of claim 1, wherein a deformation map is estimated for the captured images of a dynamic scene.

13. The method of claim 1, further comprising optimizing the projection patterns.

14. A system for determining a three-dimensional model of an object using a projector and a light sensor arranged in a stereo configuration, the system comprising one or more processors and a data storage, the data storage comprising instructions for the processor to execute: an input module to receive pixel intensity values captured by the light sensor for a plurality of images of a scene containing the object, wherein the images of the scene each capture different projection patterns emitted by the projector onto the scene; a geometry module to determine, using the pixel intensity values from a set of the plurality of images, correspondence between a projector plane and a camera plane to form a three-dimensional model of the object, the correspondence determined using a trained artificial neural network model that uses a combination of volume rendering and geometric representation; and an output module to output the three-dimensional model of the object.

15. The system of claim 14, wherein the geometric representation comprises a signed distance field representation or a parameterization with three-dimensional gaussians.

16. The system of claim 14, wherein the projection patterns comprise structured light projection patterns.

17. The system of claim 14, wherein the artificial neural network model is trained by minimizing a light sensor loss and a projector loss, the light sensor loss penalizes the difference between rendered images and measured images, and a projector loss penalizes the difference between rendered projection patterns and measured projection patterns.

18. The system of claim 17, wherein the rendered images and the rendered projection patterns incorporate at least one of: a cosine factor based on a gradient of the SDF; a reflectance term parameterized by a two-dimensional (2D) mulitilayer perceptron; and a residuals component parameterized by a two-dimensional (2D) multilayer perceptron.

19. The system of claim 14, wherein the geometry module further determines an uncertainty of depth at each pixel using reverse scanning through comparing front-to-back and back- to-front volume rendering.

20. The system of claim 14, wherein the light sensor and projector are moved in order to project the light projection patterns and capture corresponding images from different spatial positions.

21. The system of claim 14, wherein a deformation map is estimated for the captured images of a dynamic scene.