WO2022158890A1

WO2022158890A1 - Systems and methods for reconstruction of dense depth maps

Info

Publication number: WO2022158890A1
Application number: PCT/KR2022/001084
Authority: WO
Inventors: Yingen Xiong; Christopher A. Peri
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2021-01-21
Filing date: 2022-01-21
Publication date: 2022-07-28

Abstract

A method by an extended reality (XR) display device includes obtaining image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device. The method further includes determining a plurality of sets of feature points for a current image frame of the plurality of image frames, constructing a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points, and generating a dense depth map corresponding to the current image frame based on an evaluation of the cost function. The method thus includes rendering the current image frame on the one or more displays of the XR display device based on the dense depth map.

Description

SYSTEMS AND METHODS FOR RECONSTRUCTION OF DENSE DEPTH MAPS

This disclosure relates generally to database and file management within network environments, and in particular relates to reconstruction of dense depth maps for extended reality (XR) systems.

An extended reality (XR) system may generally include a computer-generated environment and/or a real-world environment that includes at least some XR artifacts. Such an XR system (or world) and associated XR artifacts typically include various applications (e.g., video games, e-commerce, social media, videoconferencing, messaging, and so forth), which may allow users to utilize these XR artifacts by manipulating their presence in the form of a computer-generated representation (e.g., avatar). In typical XR systems, image data may be rendered on, for example, a robust head-mounted display (HMD) that may be coupled through a physically wired connection or wireless connection to a base graphics generation device or cloud based service responsible for generating the image data.

However, in some instances, in which the HMD includes, for example, lightweight XR glasses or spectacles as opposed to more robust headset devices, the XR glasses or spectacles may, in comparison, include reduced processing power, low-resolution/low-cost cameras, and/or relatively simple tracking optics. Additionally, due to the smaller architectural area, the XR glasses or spectacles may also include reduced power management (e.g., batteries, battery size) and thermal management (e.g., cooling fans, heat sinks) electronics. This may often preclude such devices from maximizing performance while reducing power consumption. For instance, certain lightweight XR systems may re-project 3D renderings in which the edges or boundaries of certain objects include blurred or washed out pixels. This may prevent the lightweight XR system from accurately reconstructing and re-projecting 3D scenes and objects (e.g., 3D scene reconstruction and comprehension). It may be thus useful to provide techniques to improve lightweight XR systems.

According to an embodiment of the present disclosure, reconstruction of dense depth maps based on sparse depth points and corresponding image frames is available, and thus possible to provide improved 3D scene reconstruction and comprehension for extended reality (XR) systems.

According to a first aspect of the present invention there is provided a dense depth map reconstruction method comprising: obtaining image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device; determining a plurality of sets of feature points for a current image frame of the plurality of image frames; constructing a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points; generating a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and rendering the current image frame on the one or more displays of the XR display device based on the dense depth map.

According to a second aspect of the present invention there is provided a dense depth map reconstructing extended reality (XR) display device comprising: one or more displays; one or more computer-readable storage media including instructions; and one or more processors coupled to the display and the storage media. The one or more processors is configured to execute the instructions to: obtain image data and sparse depth points corresponding to a plurality of image frames to be displayed on the one or more displays; determine a plurality of sets of feature points for a current image frame of the plurality of image frames; construct a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points; generate a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and render the current image frame on the one or more displays based on the dense depth map.

According to a third aspect of the present invention there is provided a computer-readable medium comprising instructions that, when executed, cause one or more processors of an XR display device to perform any one of the disclosed methods.

FIG. 1 illustrates an example extended reality (XR) system.

FIG. 2A illustrates an example real-world scene.

FIG. 2B illustrates an example depth map of the real-world scene.

FIG. 3 illustrates a diagram of a workflow process for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension.

FIG. 4 illustrates a diagram of a workflow process for performing camera tracking and extraction of static object contour feature points and dynamic object contour feature points.

FIG. 5 illustrates a diagram of a workflow process for performing image edge extraction from motion.

FIG. 6A illustrates an example depth map of a real-world scene.

FIG. 6B illustrates another example depth map of the real-world scene.

FIG. 7 illustrates is a flow diagram of a method for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension.

FIG. 8 illustrates an example computer system.

Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below.

In a first aspect, a method includes: accessing image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device; determining a plurality of sets of feature points for a current image frame of the plurality of image frames; constructing a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points; generating a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and rendering the current image frame on the one or more displays of the XR display device based on the dense depth map.

Alternatively, the plurality of sets of feature points comprises a set of static edge feature points, a set of dynamic edge feature points, and a set of object contour feature points.

Alternatively, the plurality of sets of feature points further comprises one or more of a set of static object feature points, a set of dynamic object feature points, a set of static object contour feature points, or a set of dynamic object contour feature points.

Alternatively, the method may further include: prior to constructing the cost function, determining one or more weights respectively corresponding to the plurality of sets of feature points.

Alternatively, the method may further include: constructing the cost function based on the determined one or more weights, the sparse depth points, and a depth coherence between neighboring image frames with respect to the plurality of image frames.

Alternatively, the depth coherence between neighboring image frames with respect to the plurality of image frames is determined based on a dense depth map corresponding to a previous image frame of the plurality of image frames, and wherein the previous image frame was previously displayed on the one or more displays of the XR display device.

Alternatively, the dense depth map is generated utilizing one or more linear solvers to determine depth points corresponding to each of a plurality of pixels of the current image frame.

Alternatively the one or more linear solvers comprise one or more of a full multigrid (FMG) solver, a geometric multigrid (GMG) solver, or an algebraic multigrid (AMG) solver.

In a second aspect, an extended reality (XR) display device includes one or more displays; one or more non-transitory computer-readable storage media including instructions; and one or more processors coupled to the display and the storage media. The one or more processors is configured to execute the instructions to: access image data and sparse depth points corresponding to a plurality of image frames to be displayed on the one or more displays of the XR display device; determine a plurality of sets of feature points for a current image frame of the plurality of image frames; construct a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points; generate a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and render the current image frame on the one or more displays of the XR display device based on the dense depth map.

Alternatively, the instructions further comprise instructions to: prior to constructing the cost function, determine one or more weights respectively corresponding to the plurality of sets of feature points.

Alternatively, the instructions further comprise instructions to construct the cost function based on the determined one or more weights, the sparse depth points, and a depth coherence between neighboring image frames with respect to the plurality of image frames.

Alternatively, the one or more linear solvers comprise one or more of a full multigrid (FMG) solver, a geometric multigrid (GMG) solver, or an algebraic multigrid (AMG) solver.

In a third aspect, a computer-readable medium comprise instructions that, when executed by one or more processors of an extended reality (XR) display device, cause the one or more processors to: access image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device; determine a plurality of sets of feature points for a current image frame of the plurality of image frames; construct a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points; generate a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and render the current image frame on the one or more displays of the XR display device based on the dense depth map.

The present embodiments are directed toward techniques for reconstructing dense depth maps based on sparse depth points and corresponding image frames that may be utilized to improve 3D scene reconstruction and comprehension capabilities of extended reality (XR) display devices. In particular embodiments, an XR display device may access image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device. In particular embodiments, the XR display device may then determine a plurality of sets of feature points for a current image frame of the plurality of image frames. For example, in particular embodiments, the plurality of sets of feature points may include one or more of a set of static object feature points, a set of static edge feature points, a set of static object contour feature points, a set of dynamic object feature points, a set of dynamic edge feature points, or a set of dynamic object contour feature points.

In particular embodiments, XR display device may then construct a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points. For example, in particular embodiments, prior to constructing the cost function, the XR display device may determine one or more weights respectively corresponding to the plurality of sets of feature points. In particular embodiments, the XR display device may construct the cost function based on the determined one or more weights, the sparse depth points, and a depth coherence between neighboring image frames with respect to the plurality of image frames. For example, in particular embodiments, the depth coherence between neighboring image frames with respect to the plurality of image frames is determined based on a dense depth map corresponding to a previous image frame of the plurality of image frames, in which the previous image frame was previously displayed on the display of the XR display device. In particular embodiments, the XR display device may then generate a dense depth map corresponding to the current image frame based on an evaluation of the cost function. For example, in particular embodiments, the dense depth map may be generated utilizing one or more linear solvers to determine depth points corresponding to each of a plurality of pixels of the current image frame. For example, in particular embodiments, the one or more linear solvers may include one or more of a full multigrid (FMG) solver, a geometric multigrid (GMG) solver, or an algebraic multigrid (AMG) solver.

In this way, the present techniques for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension in XR display devices may improve the users' overall experience in utilizing XR display devices and XR applications. Indeed, by providing the dense depth maps based on sparse depth points and corresponding image frames, the processing times for depth map reconstruction may be reduced and the accuracy and precision of depth map reconstruction (e.g., with respect to improving smoothness and perceptibility of re-projected 3D objects, 3D object boundaries, 3D object edges, 3D object contours, and so forth) may be improved so as to be more suitable for 3D scene reconstruction and comprehension in XR applications.

As used herein, "extended reality" may refer to a form of electronic-based reality that has been manipulated in some manner before presentation to a user, including, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, simulated reality, immersive reality, holography, or any combination thereof. For example, "extended reality" content may include completely computer-generated content or partially computer-generated content combined with captured content (e.g., real-world images). In some embodiments, the "extended reality" content may also include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional (3D) effect to the viewer). Further, as used herein, it should be appreciated that "extended reality" may be associated with applications, products, accessories, services, or a combination thereof, that, for example, may be utilized to create content in extended reality and/or utilized in (e.g., perform activities) in extended reality. Thus, "extended reality" content may be implemented on various platforms, including a head-mounted device (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing extended reality content to one or more viewers.

Furthermore, as used herein, a "current image frame" may refer to a most recent image frame or a most recent set of image frames that may be either sent (e.g., pushed) to an XR display device from a computing platform or requested or loaded (e.g., pulled) from the computing platform by the XR display device for image or depth processing and analysis. For example, in particular embodiments, the "current image frame" may simply correspond to the latest image frame (of an N set of RGB-D image frames being rendered by the computing platform) that may be pushed to, or pulled by, the XR display device. That is, all of the image frames of the N set of RGB-D image frames may not be provided from the computing platform to the XR display device, and thus the "current image frame" may simply refer to the most recent image that is indeed sent from the computing platform to the XR display device.

As used herein, a "previous image frame" may refer to a penultimate most recent image frame or a penultimate most recent set of image frames that may have been either previously sent (e.g., pushed) to an XR display device from a computing platform or previously requested or loaded (e.g., pulled) from the computing platform by the XR display device for image or depth processing, analysis, and RGB-D re-projection. For example, in particular embodiments, the "previous image frame" may simply correspond to the penultimate image frame (of an N set of RGB-D image frames being rendered by the computing platform) that may have been pushed to, or pulled by, the XR display device.

FIG. 1 illustrates an example extended reality (XR) system 100 that may be suitable for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension, in accordance with presently disclosed embodiments. In particular embodiments, the XR system 100 may include an XR display device 102, a network 104, and a computing platform 106.

The XR display device 102 is a device capable of displaying extended reality (XR), and may include XR glasses in the shape of glasses worn by a user on the face, a head mounted display (HMD) worn on the head, a virtual reality headset (VRH), or an XR helmet. In the case of a head-mounted device, by arranging a display in front of the user's eyes, a super-large screen can be provided to the user, and a realistic virtual world can be provided since the screen moves according to the user's movement.

In particular embodiments, a user may wear the XR display device 102 that may display visual extended reality content to the user. The XR display device 102 may include an audio device that may provide audio extended reality content to the user. In particular embodiments, the XR display device 102 may include one or more cameras which can capture images and videos of environments. The XR display device 102 may include an eye tracking system to determine the vergence distance of the user. In particular embodiments, the XR display device 102 may include a lightweight head-mounted display (HMD) (e.g., goggles, eyeglasses, spectacles, a visor, and so forth). In particular embodiments, the XR display device 102 may also include a non-HMD device, such as a lightweight handheld display device or one or more laser projecting spectacles (e.g., spectacles that may project a low-powered　laser　onto a user's retina to project and display image or depth content to the user).

In an embodiment, the XR display device 102 may provide an XR service that outputs at least one virtual object to overlap an area determined by the user's field of view (FOV). For example, the area determined by the user's field of view is an area determined to be recognized by the user wearing the XR display device 102 through the XR display device 102, and may include the entire or at least a portion of the display of the XR display device 102. In an embodiment, the XR display device 102 may include a plurality of transparent members corresponding to each of both eyes of the user.

In one embodiment, the XR display device 102 may include a display, a camera, an audio output unit, and a support unit.

The camera may take an image corresponding to the user's field of view or measure a distance to an object. In one embodiment, the camera may be used for head tracking and spatial recognition. In addition, the camera may recognize the user's movement.

In an embodiment, the camera may further include an 'eye tracking (ET) camera' in addition to a camera used for detecting an image corresponding to the user's field of view, for example, motion of an object or for spatial recognition. In one embodiment, the ET camera may be used for detecting and tracking a user's pupils. The ET camera may be used for adjusting the center of the virtual image projected on the XR display device 102 to be positioned according to the direction in which the eyes of the user wearing the XR display device 102 gaze. For example, in an ET camera, a GS (Global shutter) camera may be used to detect a pupil and to track a rapid pupil movement without delay. The ET camera may separately include a left-eye camera and a right-eye camera.

In an embodiment, the virtual object output through the display may include information related to an application program executed on the XR display device 102 or information related to an external object located in an actual space corresponding to an area determined by the user's field of view. For example, the XR display device 102 may identify an external object included in at least a portion of the user's field of view among image information related to an actual space acquired through a camera. The XR display device 102 may output a virtual object related to an external object identified at least in part of the user's field of view through the display areas of the XR display device 102. External objects may include objects that exist in real space.

In one embodiment, the display may include a condensing lens or waveguide in a transparent member. For example, the transparent member may be formed of a glass plate, a plastic plate, or a polymer, and may be completely transparent or translucent. In an embodiment, the transparent member may include a first transparent member facing the right eye of a user wearing the XR display device 102 and a second transparent member facing the left eye of the user. When the display is transparent, the display may be disposed at a position facing the user's eyes and may display image.

The waveguide can deliver light generated by the light source of the display to the user's eyes. For example, the waveguide may be positioned at least partially on a part of the transparent member. According to an embodiment, light emitted from the display may be incident on one end of the waveguide, and the incident light may be transmitted to the user's eyes through total internal reflection within the waveguide. The waveguide may be made of a transparent material such as glass, plastic, or polymer, and may include a nano-pattern formed on one surface inside or outside, for example, a polygonal or curved grating structure. In one embodiment, the incident light may be propagated or reflected inside the waveguide by a nano-pattern to be provided to the user's eyes. In one embodiment, the waveguide includes at least one of a diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (e.g., a mirror). In an embodiment, the waveguide may guide the display light emitted from the light source unit to the user's eyes using at least one diffractive element or a reflective element.

In one embodiment, the display may include a display panel or a lens (e.g., glass). For example, the display panel may include a transparent material such as glass or plastic. In one embodiment, the display may be formed of a transparent element, and the user may see the actual space behind through the display. The display may display the virtual object on at least a portion of the transparent element so that the user sees that the virtual object is added to at least a portion of the real space.

In one embodiment, the support unit may include a printed circuit board (PCB) for transmitting electrical signals to components of the XR display device 102, a speaker for outputting an audio signal, or a battery for supplying power. For example, in the spectacle-type XR display device 102, the support unit may be disposed on the spectacle leg. The speaker may include a first speaker for transmitting an audio signal to the user's left ear and a second speaker for transmitting an audio signal to the user's right ear.

In particular embodiments, the network 104 may include, for example, any of various wireless communications networks (e.g., WLAN, WAN, PAN, cellular, WMN, WiMAX, GAN, 6LowPAN, and so forth) that may be suitable for communicatively coupling the XR display device 102 to the computing platform 106.

In particular embodiments, the computing platform 106 may include, for example, a standalone host computing system, an on-board computer system integrated with the XR display device 102, a mobile device, or any other hardware platform that may be capable of providing extended reality content to the XR display device 102. In particular embodiments, the computing platform 106 may include, for example, a cloud-based computing architecture (including one or more servers 110 and data stores 112) suitable for hosting and servicing XR applications or experiences executing on the XR display device 102. For example, in particular embodiments, the computing platform 106 may include a Platform as a Service (PaaS) architecture, a Software as a Service (SaaS) architecture, and an Infrastructure as a Service (IaaS), or other similar cloud-based computing architecture.

As it may be appreciated, in particular embodiments in which the XR display device 102 includes lightweight devices, such as goggles, eyeglasses, spectacles, a visor, and so forth, the XR display device 102 may, due to the smaller architectural area, include reduced power management (e.g., batteries, battery size) and thermal management (e.g., cooling fans, heat sinks, and so forth) electronics.

FIG. 2A illustrates an example real-world scene 200A (e.g., indoor office room) and FIG. 2B illustrates an example depth map 200B of the real-world scene (e.g., indoor office room). As illustrated by FIG. 2A, the real-world scene 200A may include objects (e.g., table, seating chairs, and so forth). However, as illustrated by FIG. 2B, the 3D objects (e.g., as generally highlighted by the dashed circle 202) of the depth map 200B may be rendered with the pixels corresponding to the boundaries and edges of the objects (e.g., seating chairs, table) appearing blurred or washed out. As may be appreciated, because the depth map 200B may be rendered with the pixels corresponding to the boundaries and edges of the 3D objects (e.g., seating chairs, table) appearing blurred or washed out, a user wearing the XR display device 102 may not be able to perceive and determine the respective 3D objects themselves or the locations of the respective 3D objects within the real-world scene 200A. This may further preclude XR artifacts that may be rendered on the respective 3D objects (e.g., seating chairs, table, and so forth) of the depth map 200B of the real-world scene from appearing desirably and appropriately perceptible to the user, and would thus reduce the efficacy of the overall XR experience of the user. Thus, as will be further appreciated with respect to FIGs. 3 through 7, the present embodiments are directed toward techniques for reconstructing dense depth maps based on sparse depth points and corresponding image frames that may be utilized to improve 3D scene reconstruction and comprehension capability of the XR display device 102 of FIG. 1.

FIG. 3 illustrates a diagram of a workflow process 300 for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension, in accordance with the presently disclosed embodiments. In particular embodiments, the workflow process 300 may begin at block 302 and with "Frame_ID = 0" with the XR display device 102 accessing a (R)ed, (G)reen, (B)lue current image frame, sparse depth points at block 304, and a dense depth map of an RGB-(D)epth previous image frame at block 306 (e.g., as illustrated by the feedback loop). For example, in particular embodiments, the current image frame and the previous image frame may include, for example, one or more still image data sequences, one or more video data sequences, one or more graphics interchange format (GIF) image data sequences, or other bitmap image data sequence. In particular embodiments, the workflow process 300 may then continue with the XR display device 102 performing, concurrently or sequentially, extraction of static object edge feature points at block 308, extraction of static object contour feature points at block 310, and extraction of dynamic object edge feature points and dynamic object contour feature points at block 312. For example, in particular embodiments, the extractions of the static object edge feature points may be performed on each individual image frame of an image frame sequence, which may include detailed features of 3D objects. In particular embodiments, the XR display device 102 may perform image segmentation to obtain 3D object areas and ranges, which may be then utilized to obtain the object contour feature points. Similarly, in particular embodiments, the extractions of dynamic object edge feature points and dynamic object contour feature points may be performed from video data captured, for example, by one or more cameras of the XR display device 102 while the XR display device 102 moves (e.g., by a user of the XR display device 102 changing her head pose or other movements). For example, in particular embodiments, the dynamic object edge feature points and dynamic object contour feature points may be detected and extracted with coherence of neighboring image frames in the current image frame sequence utilizing, for example, one or more image feature tracking processes.

In particular embodiments, the XR display device 102 may perform depth coherence computation at block 314 based on the dense depth map of an RGB-D previous image frame. For example, in particular embodiments, the XR display device 102 may determine the depth coherence between neighboring image frames with respect to the current image frame sequence based on the dense depth map of an RGB-D previous image frame. In particular embodiments, the XR display device 102 may then generate at block 316 one or more respective weights for the static object edge feature points, static object contour feature points, and dynamic object edge feature points and dynamic object contour feature points. In particular embodiments, the XR display device 102 may then utilize the generated one or more respective weights for the static object edge feature points, static object contour feature points, and dynamic object edge feature points and dynamic object contour feature points and the depth coherence between neighboring image frames with respect to the current image frame sequence to construct a cost function at block 318. In particular embodiments, XR display device 102 may construct the cost function to propagate the sparse depth points corresponding to the current image frame based on, for example, the generated one or more respective weights (e.g., which may be iteratively adjusted to alter one or more effects of different object features) for the static object edge feature points, static object contour feature points, and dynamic object edge feature points and dynamic object contour feature points and the depth coherence between neighboring image frames with respect to the current image frame sequence. For example, in particular embodiments, the XR display device 102 may construct the cost function to reconstruct a dense depth map from the propagated sparse depth points for each pixel of the current image frame.

For example, in particular embodiments, the cost function may be analytically expressed as:

arg min COST (depth) = coefficient_sparse * sum(weight_sparse * depth_difference_sparse(depth)) + coefficient_coherence * sum(weight_coherence * depth_difference_coherence(depth)) + coefficient_static_features * sum(weight_static_features * depth_difference_static_features(depth)) + coefficient_dynamic_features * sum(weight_dynamic_features * depth_difference_dynamic_features(depth))

In the cost function as analytically expressed above, coefficient_sparse, coefficient_coherence, coefficient_static_features, and coefficient_dynamic_features may represent coefficients that may be utilized for balancing and unifying certain effects of the static object edge feature points, static object contour feature points, dynamic object edge feature points, and dynamic object contour feature points; the depth coherence between neighboring image frames with respect to the current image frame sequence; and the sparse depth points. Referring again to the cost function as analytically expressed above, weight_sparse, weight_coherence, weight_static_features, and weight_dynamic_features may represent the determined weights for determining relative importance or significance with respect to the static object edge feature points, static object contour feature points, dynamic object edge feature points, and dynamic object contour feature points; the depth coherence between neighboring image frames with respect to the current image frame sequence; and the sparse depth points. Lastly, in the cost function as analytically expressed above, depth_difference_sparse(depth) may represent the depth difference between a current pixel(s) and corresponding sparse depth points(s), depth_difference_coherence(depth) may represent the depth difference between the depth coherence with respect to the current image frame and the depth coherence with respect to the previous image frame, depth_difference_static_features(depth) may represent the depth difference between the current pixel(s) and the corresponding pixel(s) of one or more static features, and depth_difference_dynamic_features(depth) may represent the depth difference between the current pixel(s) and the corresponding pixel(s) of one or more dynamic features.

In particular embodiments, the XR display device 102 may then solve the cost function to generate the dense depth map at block 320 and feedback the dense depth map for subsequent iterations of the workflow process 300. For example, in particular embodiments, the XR display device 102 may solve the cost function utilizing, for example, a full multigrid (FMG) solver, a geometric multigrid (GMG) solver, an algebraic multigrid (AMG) solver, or other solver that may provide a solution to the cost function that is both computationally inexpensive and has a reduced processing time (e.g., as compared to conventional techniques). In particular embodiments, the workflow process 300 may be performed iteratively until a dense depth map sequence from an image frame sequence is reconstructed, in which each depth map in the depth map sequence corresponds to a respective image in the image frame sequence. For example, as further depicted by FIG. 3, "Frame_ID ++" may indicate the iterative loop through the image frame sequence until the dense depth map sequence is reconstructed. In this way, the present techniques for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension may improve the users' overall XR experience in utilizing the XR display device 102 and associated XR applications. Indeed, by providing the dense depth maps based on sparse depth points and corresponding image frames, the processing times for depth map reconstruction may be reduced and the accuracy and precision of depth map reconstruction (e.g., with respect to improving smoothness and perceptibility of re-projected 3D objects, 3D object boundaries, 3D object edges, 3D object contours, and so forth) may be improved so as to be more suitable for 3D scene reconstruction and comprehension in XR applications.

FIG. 4 illustrates a diagram of a workflow process 400 for performing camera tracking and extraction of static object contour feature points and dynamic object contour feature points, in accordance with the presently disclosed embodiments. In particular embodiments, the workflow process 400 may begin at block 402 with the XR display device 102 determining a group of keyframes, which may include best possible 3D reconstruction information for the current keyframe at block 404. For example, in particular embodiments, the XR display device 102 may perform the workflow process 400 to extract camera motion for each keyframe of the group of keyframes for the current keyframe. In particular embodiments, the XR display device 102 may then determine an image keyframe based on the current keyframe and the group of keyframes at block 406. In particular embodiments, the XR display device 102 may then perform camera motion extraction and tracking at block 408. For example, in particular embodiments, for the current keyframe, the XR display device 102 may track and extract (e.g., utilizing a gradient based process or optical flow process to compute camera motion of the current frame) camera motion with neighboring frames of the group of keyframes. In particular embodiments, the XR display device 102 may store the motion-to-motion sequence for the current keyframe at block 410. In particular embodiments, the XR display device 102 may then iteratively loop through

blocks

406, 408, and 410 until all keyframes of the group of keyframes are computed. In particular embodiments, the XR display device 102 may then output a motion sequence for the current image frame corresponding to each keyframe of the group of the keyframes at block 412.

FIG. 5 illustrates a diagram of a workflow process 500 for performing image edge extraction from motion, in accordance with the presently disclosed embodiments. In particular embodiments, the workflow process 500 may begin at block 502 with the XR display device 102 determining a motion sequence for a group of keyframes for the current image frame at block 504. In particular embodiments, the XR display device 102 may then determine a keyframe motion based on the keyframe motion sequence and current image frame at block 506. For example, in particular embodiments, the XR display device 102 may determine camera motion with image features and gradients, in which the edges and boundaries of 3D objects can be expressed, for example, as magnitudes of gradients. In particular embodiments, the XR display device 102 may then extract edges and boundaries of objects from the motion sequence and compute reliabilities for the extracted edges and boundaries at block 508.

For example, in particular embodiments, the XR display device 102 may utilize magnitudes of the edges and boundaries of objects to represent the reliabilities, which may, for example, indicate a confidence score for the edges and boundaries of objects. In particular embodiments, the XR display device 102 may then perform edge filtering and noise reduction at block 510 to filter out any false positive edges or "weak" edges (e.g., edges having a lower confidence score) from the "strong" edges (e.g., edges having a higher confidence score). In particular embodiments, the XR display device 102 may then store the extracted edges and boundaries for the current image frame at block 512. In particular embodiments, the XR display device 102 may then iteratively loop through

blocks

506, 508, 510, and 512 with another keyframe motion of the keyframe motion sequence until edges and boundaries are extracted for all motion frames in the motion sequence. In particular embodiments, the XR display device 102 may then finalize the edges and boundaries extracted for the current frame together with the reliabilities (e.g., corresponding to "strong" edges having a higher confidence score) at block 514. In particular embodiments, the XR display device 102 may then output the extracted edges and boundaries extracted for the current image frame at block 516.

FIG. 6A illustrates a first example depth map 600A of a real-world scene (e.g., indoor office room) and FIG. 6B illustrates a second example depth map 600B of the real-world scene. In particular embodiments, the first example depth map 600A may correspond to a depth map generated utilizing conventional depth map reconstruction techniques. In contrast, the second example depth map 600B may correspond to a dense depth map generated utilizing the present embodiments for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension. As can be observed, the 3D objects (e.g., as generally highlighted by the dashed circle 602) of the conventional depth map 600A may be rendered with the pixels corresponding to the boundaries and edges of the objects (e.g., seating chairs, table) appearing blurred or washed out. On the other hand, as can be further observed, the 3D objects (e.g., as generally highlighted by the dashed circle 604) of the depth map 600B may be rendered with the pixels corresponding to the boundaries and edges of the objects (e.g., seating chairs, table) appearing smooth and with increased accuracy and perceptibility, in accordance with the presently disclosed embodiments.

FIG. 7 illustrates is a flow diagram of a method 700 for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension, in accordance with the presently disclosed embodiments. The method 700 may be performed utilizing one or more processing devices (e.g., computing platform 106) that may include hardware (e.g., a general purpose processor, a graphic processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), or any other processing device(s) that may be suitable for processing image data), software (e.g., instructions running/executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

In step 702, the one or more processing devices (e.g., XR display device 102) may access image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device.

In step 704, the one or more processing devices (e.g., XR display device 102) nay determine a plurality of sets of feature points for a current image frame of the plurality of image frames.

In step 706, the one or more processing devices (e.g., XR display device 102) may construct a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points.

In step 708, the one or more processing devices (e.g., XR display device 102) may generate a dense depth map corresponding to the current image frame based on an evaluation of the cost function.

In step 710, the one or more processing devices (e.g., XR display device 102) may render the current image frame on the one or more displays of the XR display device based on the dense depth map.

In this way, the present techniques for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension capabilities of XR display devices may improve the users' overall experience in utilizing XR display devices and XR applications. Indeed, by providing the dense depth maps based on sparse depth points and corresponding image frames, the processing times for depth map reconstruction may be reduced and the accuracy and precision of depth map reconstruction (e.g., with respect to improving smoothness and perceptibility of re-projected 3D objects, 3D object boundaries, 3D object edges, 3D object contours, and so forth) may be improved so as to be more suitable for 3D scene reconstruction and comprehension in XR applications.

FIG. 8 illustrates an example computer system 800 that may be utilized for reconstructing dense depth maps based on sparse depth points and corresponding image frames for providing improved 3D scene reconstruction and comprehension, in accordance with the presently disclosed embodiments. In particular embodiments, one or more computer systems 800 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 800 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 800 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 800. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems 800. This disclosure contemplates computer system 800 taking any suitable physical form. As example and not by way of limitation, computer system 800 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these. Where appropriate, computer system 800 may include one or more computer systems 800; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks.

Where appropriate, one or more computer systems 800 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example, and not by way of limitation, one or more computer systems 800 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 800 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.

In particular embodiments, computer system 800 includes a processor 802, memory 804, storage 806, an input/output (I/O) interface 808, a communication interface 810, and a bus 812. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 802 includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, processor 802 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 804, or storage 806; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 804, or storage 806. In particular embodiments, processor 802 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 802 including any suitable number of any suitable internal caches, where appropriate. As an example, and not by way of limitation, processor 802 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 804 or storage 806, and the instruction caches may speed up retrieval of those instructions by processor 802.

Data in the data caches may be copies of data in memory 804 or storage 806 for instructions executing at processor 802 to operate on; the results of previous instructions executed at processor 802 for access by subsequent instructions executing at processor 802 or for writing to memory 804 or storage 806; or other suitable data. The data caches may speed up read or write operations by processor 802. The TLBs may speed up virtual-address translation for processor 802. In particular embodiments, processor 802 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 802 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 802 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 802. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory 804 includes main memory for storing instructions for processor 802 to execute or data for processor 802 to operate on. As an example, and not by way of limitation, computer system 800 may load instructions from storage 806 or another source (such as, for example, another computer system 800) to memory 804. Processor 802 may then load the instructions from memory 804 to an internal register or internal cache. To execute the instructions, processor 802 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 802 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 802 may then write one or more of those results to memory 804. In particular embodiments, processor 802 executes only instructions in one or more internal registers or internal caches or in memory 804 (as opposed to storage 806 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 804 (as opposed to storage 806 or elsewhere).

One or more memory buses (which may each include an address bus and a data bus) may couple processor 802 to memory 804. Bus 812 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 802 and memory 804 and facilitate accesses to memory 804 requested by processor 802. In particular embodiments, memory 804 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 804 may include one or more memory devices, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage 806 includes mass storage for data or instructions. As an example, and not by way of limitation, storage 806 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage 806 may include removable or non-removable (or fixed) media, where appropriate. Storage 806 may be internal or external to computer system 800, where appropriate. In particular embodiments, storage 806 is non-volatile, solid-state memory. In particular embodiments, storage 806 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 806 taking any suitable physical form. Storage 806 may include one or more storage control units facilitating communication between processor 802 and storage 806, where appropriate. Where appropriate, storage 806 may include one or more storages 806. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 808 includes hardware, software, or both, providing one or more interfaces for communication between computer system 800 and one or more I/O devices. Computer system 800 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system 800. As an example, and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 808 for them. Where appropriate, I/O interface 808 may include one or more device or software drivers enabling processor 802 to drive one or more of these I/O devices. I/O interface 808 may include one or more I/O interfaces, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 810 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 800 and one or more other computer systems or one or more networks. As an example, and not by way of limitation, communication interface 810 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 810 for it.

As an example, and not by way of limitation, computer system 800 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 800 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system 800 may include any suitable communication interface 810 for any of these networks, where appropriate. Communication interface 810 may include one or more communication interfaces 810, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

In particular embodiments, bus 812 includes hardware, software, or both coupling components of computer system 800 to each other. As an example, and not by way of limitation, bus 812 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 812 may include one or more buses 812, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

Herein, a computer-readable storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

Herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Moreover, "and" is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated otherwise by context.

Herein, "automatically" and its derivatives means "without human intervention," unless expressly indicated otherwise or indicated otherwise by context.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, may be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However, any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) may be claimed as well, so that any combination of claims and the features thereof are disclosed and may be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which may be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.

The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.

Claims

A method comprising, by an extended reality (XR) display device:

obtaining image data and sparse depth points corresponding to a plurality of image frames to be displayed on one or more displays of the XR display device;

determining a plurality of sets of feature points for a current image frame of the plurality of image frames;

constructing a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points;

generating a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and

rendering the current image frame on the one or more displays of the XR display device based on the dense depth map.
The method of Claim 1, wherein the plurality of sets of feature points comprises a set of static edge feature points, a set of dynamic edge feature points, and a set of object contour feature points.
The method of Claim 2, wherein the plurality of sets of feature points further comprises one or more of a set of static object feature points, a set of dynamic object feature points, a set of static object contour feature points, or a set of dynamic object contour feature points.
The method of Claim 1, further comprising:

prior to constructing the cost function, determining one or more weights respectively corresponding to the plurality of sets of feature points.
The method of Claim 4, further comprising:

constructing the cost function based on the determined one or more weights, the sparse depth points, and a depth coherence between neighboring image frames with respect to the plurality of image frames.
The method of Claim 5, wherein the depth coherence between neighboring image frames with respect to the plurality of image frames is determined based on a dense depth map corresponding to a previous image frame of the plurality of image frames, and wherein the previous image frame was previously displayed on the one or more displays of the XR display device.
The method of Claim 1, wherein the dense depth map is generated utilizing one or more linear solvers to determine depth points corresponding to each of a plurality of pixels of the current image frame.
An extended reality (XR) display device, comprising:

one or more displays;

one or more computer-readable storage media including instructions; and

one or more processors coupled to the display and the storage media, the one or more processors configured to execute the instructions to:

obtain image data and sparse depth points corresponding to a plurality of image frames to be displayed on the one or more displays;

determine a plurality of sets of feature points for a current image frame of the plurality of image frames;

construct a cost function configured to propagate the sparse depth points corresponding to the current image frame based on the plurality of sets of feature points;

generate a dense depth map corresponding to the current image frame based on an evaluation of the cost function; and

render the current image frame on the one or more displays based on the dense depth map.
The XR display device of Claim 8, wherein the plurality of sets of feature points comprises a set of static edge feature points, a set of dynamic edge feature points, and a set of object contour feature points.
The XR display device of Claim 8, wherein the instructions further comprise instructions to:

prior to constructing the cost function, determine one or more weights respectively corresponding to the plurality of sets of feature points.
The XR display device of Claim 10, wherein the instructions further comprise instructions to construct the cost function based on the determined one or more weights, the sparse depth points, and a depth coherence between neighboring image frames with respect to the plurality of image frames.
The XR display device of Claim 11, wherein the depth coherence between neighboring image frames with respect to the plurality of image frames is determined based on a dense depth map corresponding to a previous image frame of the plurality of image frames, and wherein the previous image frame was previously displayed on the one or more displays.
The XR display device of Claim 8, wherein the dense depth map is generated utilizing one or more linear solvers to determine depth points corresponding to each of a plurality of pixels of the current image frame.
The XR display device of Claim 13, wherein the one or more linear solvers comprise one or more of a full multigrid (FMG) solver, a geometric multigrid (GMG) solver, or an algebraic multigrid (AMG) solver.
A computer-readable medium comprising instructions that, when executed, cause one or more processors of an extended reality (XR) display device to perform method of any one of claims 1 to 7.