TW202316239A - Frame extrapolation with application generated motion vector and depth - Google Patents

Abstract

In one embodiment, a method includes receiving a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application, calculating a current three-dimensional position corresponding to the current frame of an object presented in the rendered image using the depth map, calculating a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map, estimating a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object, and generating an extrapolated image corresponding to the future frame by reprojecting the object presented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object.

Description

Frame extrapolation with application-generated motion vectors and depth

The present invention relates generally to artificial reality systems, and in particular, to external insets.

This application is asserted under 35 U.S.C. § 119(e) in U.S. Provisional Patent Application No. 63/254476, filed October 11, 2021, and U.S. Nonprovisional Patent Application No. 17/590682, filed February 01, 2022 interests, which are incorporated herein by reference.

Artificial reality is a form of reality that has been adjusted in some way before being presented to the user, which may include, for example, virtual reality (VR), augmented reality (augmented reality, AR), mixed reality ( mixed reality; MR), mixed reality, or a combination and/or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with captured (eg, real world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of these may be presented in a single channel or in multiple channels (such as stereoscopic video that creates a three-dimensional effect on the viewer). Artificial reality may relate to, for example, applications, products, accessories, services or some combination thereof for creating content in artificial reality and/or for use in artificial reality (e.g., performing activities in artificial reality) couplet. Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, or an artificial Any other hardware platform on which the reality content is provided to one or more viewers.

Certain embodiments described herein relate to systems and methods for generating high quality frame extrapolation and reprojection by using motion vectors and depth maps generated by applications. In order to provide users with a comfortable artificial reality experience, it is necessary to present high-resolution frames at a high frame rate. However, this is a great challenge for mobile HMDs due to the limitation of computing power of the hardware. Traditional time warp solutions for artificial reality systems have several limitations: the solution only corrects for rotation but not translational movement, and does not address animation shutter. Traditional time warping solutions simply rotate the 2D RGB image to accommodate the user's new viewpoint. Traditional frame extrapolation solutions are based on lower quality motion vectors due to estimating motion vectors from 2D images. The novel frame extrapolation solution disclosed in this paper can consider translational movement as well as rotational movement. The novel frame extrapolation solution utilizes motion vector and depth information generated by the application based on rendered objects.

In certain embodiments, a computing system associated with a wearable device can receive a rendered image, motion vector data, and depth map corresponding to a current frame generated by an application. Motion vector data and depth maps can be generated based on 3D objects rendered by the application. The motion vectors in the motion vector data can be three-dimensional. The computing system can process the received motion vector data and depth map such that regions corresponding to the foreground of the rendered image are expanded. The computing system can use the depth map to calculate the current three-dimensional position of objects present in the rendered image corresponding to the current frame. To calculate the current three-dimensional position of the object, the computing system may back-project the depth map onto the three-dimensional space from the current viewpoint associated with the current frame. The current viewpoint can be associated with the position and orientation of the wearable device at the instant in time at which the current frame is presented. The computing system can use the motion vector data and the depth map to calculate the past three-dimensional position of the object corresponding to the past frame. To calculate the past three-dimensional position of the object, the computing system can generate an estimated depth map corresponding to the past frame by subtracting the motion vector from the depth map. The computing system may back-project the estimated depth map onto three-dimensional space from a past viewpoint associated with the past frame. The computing system can estimate the future three-dimensional position of the object corresponding to the future frame based on the past three-dimensional position and the current three-dimensional position of the object. The future three-dimensional position of the object may be estimated based on the assumption that the object moves at a constant speed from a time instant corresponding to a past frame to a time instant corresponding to a future frame. The computing system can perform linear interpolation to estimate the future three-dimensional position of the object. After estimating the future 3D position of the object, the computing system can generate a distorted mesh by projecting the estimated future 3D position of the object onto a future viewpoint The object is reprojected to a future viewpoint associated with the future frame to generate an extrapolated image corresponding to the future frame. To generate the extrapolated image, the computing system may apply a distortion grid to the rendered image.

The specific examples disclosed herein are examples only, and the scope of the invention is not limited to these specific examples. A particular embodiment may include all, some, or none of the components, elements, features, functions, operations or steps of the embodiments disclosed above. Embodiments according to the present invention are especially disclosed in the appended claims for a method, storage medium, system and computer program product, wherein any feature mentioned in one claim category (such as a method) may also be described in another asserted in the claim item category (eg system). Dependencies or back references in the appended claims are selected for formality reasons only. However, any subject matter arising from a reverse deliberate reference to any preceding claim, particularly in terms of multiple dependencies, may likewise be claimed such that any combination of the claims and their features are disclosed and may not be related to the It is asserted with the dependency selected in the scope of the application. Claimable subject matter includes not only combinations of features as set forth in the appended claims but also any other combination of features in the claims, where each feature mentioned in a claim can be combined in any other feature or claim Combinations of other features. Furthermore, any of the embodiments and features described or depicted herein may be included in a separate claim and/or in connection with any embodiment or feature described or depicted herein or with features of the appended claims. asserted in any combination of any of them.

FIG. 1A illustrates an example artificial reality system 100A. In a specific example, the artificial reality system 100A may include a headset 104 , a controller 106 and a computing system 108 . The user 102 can wear a headset 104 that can display visual artificial reality content to the user 102 . The headset 104 can include an audio device that can provide audio artificial reality content to the user 102 . Headset 104 may include one or more cameras that capture images and video of the environment. The headset 104 may include an eye tracking system to determine the convergence distance of the user 102 . Headset 104 may include a microphone to capture voice input from user 102 . Headset 104 may be referred to as a head-mounted display (HMD). Controller 106 may include a trackpad and one or more buttons. Controller 106 can receive input from user 102 and relay the input to computing system 108 . The controller 106 can also provide tactile feedback to the user 102 . The computing system 108 can be connected to the headset 104 and the controller 106 via a cable or a wireless connection. The computing system 108 can control the headset 104 and the controller 106 to provide artificial reality content to and receive input from the user 102 . Computing system 108 may be a stand-alone mainframe computing system, an onboard computing system integrated with headset 104, a mobile device, or any other hardware capable of providing artificial reality content to and receiving input from user 102 platform.

FIG. 1B illustrates an example augmented reality system 100B. The augmented reality system 100B may include a head-mounted display (HMD) 110 (eg, glasses) including a frame 112 , one or more displays 114 , and a computing system 108 . The display 114 can be transparent or translucent, allowing the user wearing the HMD 110 to see the real world through the display 114 while simultaneously displaying visual artificial reality content to the user. HMD 110 may include an audio device that may provide audio artificial reality content to a user. HMD 110 may include one or more cameras that can capture images and video of the environment. The HMD 110 may include an eye-tracking system to track the vergent movements of a user wearing the HMD 110 . HMD 110 may include a microphone to capture voice input from the user. The augmented reality system 100B may further include a controller including a track pad and one or more buttons. The controller can receive input from a user and relay the input to the computing system 108 . The controller can also provide tactile feedback to the user. Computing system 108 may be connected to HMD 110 and the controller via a cable or wireless connection. The computing system 108 can control the HMD 110 and the controller to provide augmented reality content to the user and to receive input from the user. Computing system 108 may be a stand-alone host computer device, an on-board computer device integrated with HMD 110, a mobile device, or any other hardware platform capable of providing artificial reality content to and receiving input from the user.

Figure 2 illustrates an example overview of frame extrapolation and reprojection using motion vector and depth information generated by an application. In the example illustrated in FIG. 2 , the application 210 renders images of video frames at 36 frames per second (FPS). Therefore, application 210 presents image 211 for frames N and N+2. The application also generates motion vectors and depth information 213 corresponding to each image 211 . By using the frame extrapolation solution proposed herein, the operating system 220 of the computing system 108 presents the frame to the user at 72 FPS. The operating system 220 presents the image 211 presented by the application 210 for frame N and frame N+2 221 to a display associated with the operating system 108 . Based on the rendered image 211 , and the motion vector and depth information 213 generated along with the rendered image 211 , the operating system 220 generates an image 223 for frames N+1 and N+3. The operating system 220 presents the generated images 223 for frames N+1 and N+3 to the display. Although high-quality frame extrapolation using motion vectors and depth information 213 can be important, the delay caused by sampling half can often exert a significant impact on user experience. When latency is kept high, the user may experience significant black pulling when rotating the headset 104 or significant lag when moving their controls. Several techniques are proposed herein to reduce potential delays: (1) The operating system 220 can delay the start of the frame to reduce the time instant when the application 210 finishes rendering the image and the time instant the image spends on the display interval between. (2) The operating system 220 can then extract the poses of the headset 104 and the controller 106 to fill the time gap between presenting the image and displaying the presented image to the display. (3) The operating system 220 can resample the camera pose immediately before time warping, and reproject pixels based on both self-camera rotation and camera translation considerations. This technique is called positional time warping (PTW). The depth map generated by the application can be used for this PTW. In certain embodiments, without using frame extrapolation, the head pose latency with PTW can be even lower than that in an equivalent full frame rate application. In certain embodiments, a runtime system specifically responsible for frame extrapolation may function in place of the operating system 220 . Although this disclosure describes frame extrapolation at a particular rate, this disclosure contemplates frame extrapolation at any suitable rate.

In a particular embodiment, operating system 220 of computing system 108 associated with HMD 110 may receive a rendered image, motion vector data, and depth map corresponding to a current frame generated by application 210 . In certain embodiments, computing system 108 may include a runtime system specifically responsible for frame extrapolation. In such cases, the runtime system may replace operating system 220 for the programs disclosed herein. Unlike previous methods that estimate motion vectors based on comparisons between 2D images, motion vector data and depth maps can be generated based on 3D objects rendered by an application. The motion vectors in the motion vector data can be three-dimensional. Motion vector data may be generated using motion blurring techniques, temporal anti-aliasing techniques, or any suitable technique. Since the depth buffer is always used for motion vector calculations, the overhead for generating the depth map can be small. In a particular embodiment, operating system 220 of computing system 108 may process the received motion vector data and depth map such that regions corresponding to the foreground of the rendered image are expanded. Although this disclosure describes receiving rendered images, motion vector data, and depth maps in a particular manner, this disclosure contemplates receiving rendered images, motion vector data, and depth maps in any suitable manner.

In certain embodiments, the operating system 220 of the computing system 108 may use the depth map to calculate the current three-dimensional position of an object present in the rendered image corresponding to the current frame. To calculate the current three-dimensional position of the object, the operating system 220 of the computing system 108 may back-project the depth map from the current viewpoint associated with the current frame onto the three-dimensional space. The current viewpoint may be associated with the position and orientation of the wearable device at the instant in time at which the current frame is presented. Figure 3 illustrates an example data flow for frame extrapolation. By way of example and not limitation, illustrated in FIG. 3 , the operating system 220 of the computing system 108 can access a UV depth map 311 associated with a rendered image corresponding to frame N . The UV depth map 311 may be a mapping of depth information to 2D screen positions. In certain embodiments, UV depth map 311 may be a two-dimensional map on UV coordinates. By performing backprojection of the UV depth map 311 from the viewpoint associated with frame N onto the three-dimensional space, the operating system 220 of the computing system 108 may calculate the three-dimensional position 314 of an object in the rendered image corresponding to frame N . By applying the inverse of the view projection matrix 313 corresponding to frame N to the UV depth map 311 , backprojection can be performed. Although this disclosure describes calculating the current three-dimensional position of an object represented in a rendered image in a particular manner, the present invention contemplates computing the current three-dimensional position of an object represented in a rendered image in any suitable manner.

In certain embodiments, the operating system 220 of the computing system 108 may use the motion vector data and the depth map to calculate past three-dimensional positions of objects corresponding to past frames. To calculate the past three-dimensional position of the object, the operating system 220 of the computing system 108 may generate an estimated depth map corresponding to the past frame by subtracting the motion vector from the depth map. The operating system 220 of the computing system 108 may back-project the estimated depth map from the past viewpoint associated with the past frame onto the three-dimensional space. By way of example and not limitation, continuing with the previous example illustrated in FIG. 3 , by subtracting the motion vector 312 corresponding to frame N from the UV depth map 311 corresponding to frame N , the operating system 220 of the computing system 108 may Depth map 321 corresponding to frame N -1 is estimated. The operating system 220 of the computing system 108 may calculate the The three-dimensional position 324 of the N -1 objects. By applying the inverse of the view projection matrix 323 corresponding to frame N-1 to the estimated UV depth map 321 corresponding to frame N -1, backprojection may be performed. Although this disclosure describes computing the past 3D position of an object corresponding to a past frame in a particular manner, the present invention contemplates computing the past 3D position of an object corresponding to a past frame in any suitable manner.

In certain embodiments, based on the past three-dimensional position and the current three-dimensional position of the object, the operating system 220 of the computing system 108 may estimate the future three-dimensional position of the object corresponding to the future frame. Estimating the future three-dimensional position of the object based on the past three-dimensional position and the current three-dimensional position of the object can be called space warp. Estimating the future three-dimensional position of an object may be performed based on the assumption that the object moves at a constant speed from a time instant corresponding to a past frame to a time instant corresponding to a future frame. The operating system 220 of the computing system 108 can perform linear interpolation to estimate the future three-dimensional position of the object. By way of example and not limitation, continuing with the previous example described in FIG. 3, based on the calculated three-dimensional position 314 of the object corresponding to frame N , and the estimated three-dimensional position 324 of the object corresponding to frame N -1, the calculation The operating system 220 of the system 108 can estimate the three-dimensional position 334 of the object. Although this disclosure describes estimating the future 3D position of an object corresponding to a future frame based on the object's past and current 3D positions in a particular manner, the present invention contemplates estimating the object's past and current 3D positions based on the object's past and current 3D positions in any suitable manner. A future three-dimensional position of an object corresponding to a future frame is estimated.

4 illustrates an example estimate of a future location of an object based on its past location and current location . In the specific example illustrated in FIG. 4, the object is located at a three-dimensional position x1 at time t1, and is located at a three-dimensional position x2 at time t2. The operating system 220 of the computing system 108 can estimate the three-dimensional position x3 at the time t3 by performing linear interpolation, where x3=Ler(x1, x2, (t3-t1)/(t2-t1)). Although this disclosure describes linearly interpolating to predict a three-dimensional position of an object based on previous positions in a particular manner, this disclosure contemplates performing linear interpolation based on previous positions in any suitable manner to predict the three-dimensional position of an object.

In certain embodiments, the operating system 220 of the computing system 108 may generate the distorted mesh by reprojecting the estimated future three-dimensional position of the object onto a future viewpoint. By way of example and not limitation, continuing the previous example described in FIG. 3, by reprojecting the estimated three-dimensional position 334 of the object corresponding to frame N+1 onto the viewpoint corresponding to frame N +1, the computation The operating system 220 of the system 108 can generate the distortion grid 337 . Reprojection can be performed by applying the view projection matrix 335 corresponding to frame N +1 to the estimated three-dimensional position 334 of the object. The view projection matrix 335 can be obtained by re-extracting the pose of the headset 104 . Although this disclosure describes generating a distorted mesh by reprojecting the estimated future 3D position of an object onto a future viewpoint in a particular manner, this disclosure contemplates generating a distorted mesh by reprojecting the estimated future 3D position of an object onto the future in any suitable manner. view point up to produce a distorted mesh.

In certain embodiments, the operating system 220 of the computing system 108 can generate a corresponding future image by using the future three-dimensional position of the object to reproject the object represented in the rendered image to a future viewpoint associated with the future frame. The extrapolated image of the frame. To generate the extrapolated image, operating system 220 of computing system 108 may apply a distortion grid to the rendered image. By way of example and not limitation, continuing with the previous example illustrated in FIG. 3, by applying the distortion grid 337 to the rendered image (not shown) corresponding to frame N , the operating system 220 of the computing system 108 can An image (not shown) corresponding to frame N+1 is generated. The operating system 220 of the computing system 108 can display the generated image corresponding to frame N+1 to a display associated with the headset 104 . Although the present invention describes generating the extrapolated image corresponding to the future frame in a particular manner, the present invention contemplates generating the extrapolated image corresponding to the future frame in any suitable manner.

5 illustrates an example comparison of time budget per frame between a full frame rendering application and a half frame rendering application. In FIG. 5 , (a) illustrates an application that renders images at 72 FPS, and (b) illustrates an application that renders images at 36 FPS. For the application in (b), an additional 36 frames per second can be extrapolated using the invention disclosed herein. For the application in (a), the total budget per frame may be 13.9 ms, which may need to be split between the application 210 and the operating system 220 . For each frame, the operating system 220 may perform composition work to push the rendered image to the screen in the backend. Since the application program 210 and the operating system 220 can share the same Graphics Processing Unit (GPU), if the operating system 220 occupies 1.3 ms per vsync, the application program 210 can have 12.6 ms to use. Meanwhile, the application in (b) may have 27.8ms per frame when the application is rendered at 36 FPS. Due to frame extrapolation, the operating system 220 may consume more time for vsync, eg 1.8ms as illustrated in FIG. 5 . Furthermore, the application in (b) may take additional time for generating motion vectors, eg 2.5 ms as illustrated in FIG. 5 . The application in (b) may have 21.7 ms of GPU time per frame, which is 71% larger than the budget of the application in (a).

Applications may need to render transparent objects. For example, an application may render a transparent object moving to the left on top of an opaque object moving to the right. For a pixel containing two objects, the motion vector can be ambiguous because the pixel moves in two directions. However, the problem may be less pronounced. When the transparent surface is far from the camera, the projection movement can be minimal between frames. Also, for particle effects, little movement jitter may not be noticeable, transparency presents a greater use case since this effect often occurs with fast animations, such as explosions. A problematic situation using frame extrapolation and reprojection may be the case where near-field fast moving objects are transparent. An example of a near field fast moving object would be the controller 106 . Therefore, the object associated with the controller 106 and any child objects of the controller 106 may need to be opaque.

Frame extrapolation and reprojection can cause some image distortion, especially on backgrounds. Distortion may not be noticeable when the background has a rich textured pattern. However, distortions caused by frame extrapolation and reprojection may be noticeable to the user when objects are moving on a clear background. Specific considerations are required to make backgrounds more friendly to frame extrapolation.

Frame extrapolation and reprojection can cause pixel distortion artifacts around the object when the object is rotated rapidly. Imagine the cube rotating at about 100 revolutions per second. The orientation of the cube from frame to next frame may appear to be more or less random, since the movement vector may not be constructed accurately. To alleviate this problem, when the application detects high-speed rotation during the motion vector generation phase, the application can disable the portion of the motion vector associated with the object's rotation.

6 illustrates an example method 600 for outlining an inset based on motion vectors and depth maps generated by an application. The method may begin at step 610, where the operating system of the computing system 108 may receive a rendered image, motion vector data, and depth map corresponding to the current frame generated by the application. At step 620, the operating system of the computing system 108 may use the depth map to calculate the current three-dimensional position corresponding to the current frame of the object represented in the rendered image. At step 630, the operating system of the computing system 108 may calculate the past three-dimensional position of the object corresponding to the past frame using the motion vector data and the depth map. In step 640, the operating system of the computing system 108 may estimate the future three-dimensional position of the object corresponding to the future frame based on the past three-dimensional position and the current three-dimensional position of the object. At step 650, the operating system of the computing system 108 may generate a corresponding future frame by reprojecting the object represented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object. extrapolated image. Certain embodiments may repeat one or more steps of the method of FIG. 6 where appropriate. Although this disclosure describes and illustrates certain steps of the method of FIG. 6 as occurring in a particular order, this disclosure contemplates that any suitable steps of the method of FIG. 6 occur in any suitable order. Furthermore, although this disclosure describes and illustrates an example method for outlining an inset frame from motion vectors and depth maps generated by an application, which includes certain steps of the method of FIG. Any suitable method of producing motion vectors and depth maps out of inset frames, including any suitable steps, which may include, where appropriate, all, some, or all of the steps of the method of FIG. 6 None. Furthermore, although this disclosure describes and illustrates particular components, devices or systems for carrying out particular steps of the method of FIG. 6 , this disclosure contemplates any suitable combination of any suitable components, devices or systems for carrying out any suitable steps of the method of FIG. 6 .

System and method

FIG. 7 illustrates an example computer system 700 . In certain embodiments, one or more computer systems 700 perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems 700 provide the functionality described or illustrated herein. In certain embodiments, software running on one or more computer systems 700 performs one or more steps of one or more methods described or illustrated herein, or provides functions described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 700 . Herein, references to computer systems may encompass computing systems, and vice versa, where appropriate. In addition, a reference to a computer system may encompass one or more computer systems, where appropriate.

The present invention contemplates any suitable number of computer systems 700 . The invention contemplates computer system 700 taking any suitable physical form. By way of example and not limitation, the computer system 700 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (single-board computer system; SBC) (such as a computer- on-module; COM) or module system (system-on-module; SOM)), desktop computer system, laptop or notebook computer system, interactive public information query station, mainframe computer, network of computer systems cell phone, personal digital assistant (PDA), server, tablet computer system, or a combination of two or more of these. Where appropriate, computer system 700 may comprise one or more computer systems 700; be standalone or distributed; span multiple locations; span multiple machines; span multiple data centers; , the cloud may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 700 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitation. By way of example and not limitation, one or more computer systems 700 may perform one or more steps of one or more methods described or illustrated herein in real-time or in batch mode. Where appropriate, one or more computer systems 700 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

In a specific example, computer system 700 includes processor 702 , memory 704 , storage 706 , input/output (I/O) interface 708 , communication interface 710 , and bus 712 . Although this disclosure describes and illustrates a particular computer system with a particular number of particular components in a particular configuration, this disclosure contemplates any suitable computer system with any suitable number of any suitable components in any suitable configuration.

In certain embodiments, processor 702 includes hardware for executing instructions, such as those making up a computer program. By way of example and not limitation, to execute instructions, processor 702 may retrieve (or fetch) instructions from internal scratchpad, internal cache, memory 704, or storage 706; decode and execute the instructions; And then write one or more results to internal registers, internal cache, memory 704 or storage 706 . In certain embodiments, processor 702 may include one or more internal cache memories for data, instructions, or addresses. The invention contemplates processor 702 including any suitable number of any suitable internal cache memory, where appropriate. By way of example and not limitation, processor 702 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the I-cache may be copies of instructions in memory 704 or storage 706 , and the I-cache may speed up the retrieval of those instructions by the processor 702 . The data in the data cache memory can be a copy of the data in the memory 704 or the storage 706 for the operation of the instructions executed at the processor 702; for the subsequent instructions executed at the processor 702 to access or write the results of previous instructions executed at processor 702; or other suitable data, into memory 704 or storage 706. The data cache can speed up read or write operations performed by the processor 702 . The TLB can speed up virtual address translation for the processor 702 . In certain embodiments, processor 702 may include one or more internal registers for one or more of data, instructions, or addresses. The invention encompasses processor 702 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 702 may include one or more arithmetic logic units (ALU); be a multi-core processor; or include one or more processors 702 . Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In certain embodiments, memory 704 includes main memory for storing instructions for execution by processor 702 or data against which processor 702 operates. By way of example and not limitation, computer system 700 may load instructions into memory 704 from storage 706 or from another source, such as another computer system 700 . The processor 702 can then load the instructions from the memory 704 into an internal register or an internal cache. To execute the instructions, processor 702 may retrieve and decode the instructions from internal register or internal cache. During or after execution of instructions, processor 702 may write one or more results (which may be intermediate or final results) to internal registers or internal cache memory. Processor 702 may then write one or more of those results to memory 704 . In certain embodiments, processor 702 executes only one or more instructions in internal scratchpad or internal cache memory or in memory 704 (as opposed to storage 706 or elsewhere), and only for one or more internal scratchpad or internal cache memory or in memory 704 (as opposed to storage 706 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 702 to memory 704 . As described below, busses 712 may include one or more memory buses. In certain embodiments, one or more memory management units (MMUs) reside between the processor 702 and the memory 704 and facilitate accesses to the memory 704 requested by the processor 702 . In a particular embodiment, memory 704 includes random access memory (random access memory; RAM). Where appropriate, this RAM can be volatile memory. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Furthermore, this RAM may be a single-port or multi-port RAM, where appropriate. This invention encompasses any suitable RAM. Memory 704 may include one or more memories 704, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In certain embodiments, storage 706 includes mass storage for data or instructions. By way of example and not limitation, storage 706 may include a hard disk drive (HDD), floppy disk, flash memory, optical disk, magneto optical disk, magnetic tape, or a Universal Serial Bus (Universal Serial Bus; USB) drive, or a combination of two or more of these. Storage 706 may include removable or non-removable (or fixed) media, where appropriate. Storage 706 may be internal or external to computer system 700, as appropriate. In a particular embodiment, storage 706 is a non-volatile solid-state memory. In a particular embodiment, storage 706 includes read-only memory (ROM). When appropriate, this ROM can be masked programmable ROM, programmable ROM (programmable ROM; PROM), erasable PROM (erasable PROM; EPROM), electrically erasable PROM (electrically erasable PROM; EEPROM ), electrically alterable ROM (electrically alterable ROM; EAROM), or flash memory or a combination of two or more of these. The invention contemplates mass storage 706 taking any suitable physical form. Storage 706 may include one or more storage device control units that facilitate communication between processor 702 and storage 706, where appropriate. Storage 706 may include one or more storages 706, where appropriate. Although this disclosure describes and illustrates a particular storage, this disclosure contemplates any suitable storage.

In certain embodiments, I/O interface 708 includes hardware, software, or both, providing one or more interfaces for communication between computer system 700 and one or more I/O devices. Computer system 700 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between the individual and the computer system 700 . By way of example and not limitation, I/O devices may include keyboards, keypads, microphones, monitors, mice, printers, scanners, speakers, still cameras, stylus, tablets, touch screens, trackballs , a video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. The present invention contemplates any suitable I/O device and any suitable I/O interface 708 therefor. I/O interface 708 may include one or more devices or software drivers, where appropriate, enabling processor 702 to drive one or more of these I/O devices. I/O interface 708 may include one or more I/O interfaces 708, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In certain embodiments, communication interface 710 includes hardware, software, or both, providing one or more interfaces for communication between computer system 700 and one or more other computer systems 700 or one or more networks ( such as packet-based communication). By way of example and not limitation, communication interface 710 may include a network interface controller (network interface controller; NIC) for communicating with Ethernet or other wire-based networks, or for communicating with wireless networks such as WI- FI network) communication wireless NIC (wireless NIC; WNIC) or wireless adapter. The present invention contemplates any suitable network and any suitable communication interface 710 therefor. By way of example and not limitation, the computer system 700 can be connected to ad hoc networks, personal area networks (PAN), local area networks (LAN), wide area networks (WAN), metropolitan One or more parts of a metropolitan area network (MAN) or the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, the computer system 700 can communicate with wireless PAN (wireless PAN; WPAN) (such as Bluetooth WPAN), WI-FI network, WI-MAX network, cellular telephone network (such as Global System for Mobile Communications (Global System for Mobile Communication; GSM) network), or other suitable wireless network or a combination of two or more of these communications. Computer system 700 may include any suitable communications interface 710 for any of these networks, where appropriate. Where appropriate, the communication interface 710 may include one or more communication interfaces 710 . Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

In certain embodiments, bus 712 includes hardware, software, or both that couple components of computer system 700 to each other. By way of example and not limitation, bus 712 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus ; FSB), Hypertransport (HYPERTRANSPORT; HT) interconnect, Industry Standard Architecture (Industry Standard Architecture; ISA) bus, INFINIBAND interconnect, low pin count (low-pin-count; LPC) bus, memory bus , Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI Express (PCI-Express; PCIe) bus, serial advanced attachment technology (serial advanced technology attachment (SATA) bus, Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of such buses. Bus bars 712 may include one or more bus bars 712, where appropriate. Although this disclosure describes and illustrates a particular busbar, this disclosure contemplates any suitable busbar or interconnect.

Herein, the computer-readable non-transitory storage medium or the medium may include one or more semiconductor-based or other integrated circuit (integrated circuit; IC) (such as field-programmable gate array (field-programmable gate array; FPGA) ) or application-specific IC (application-specific IC; ASIC)), hard disk drive (HDD), hybrid hard drive (hybrid hard drive; HHD), optical disc, optical disc drive (optical disc drives; ODD), magnetic Optical disc, magnetic optical drive, floppy disk, floppy disk drive (FDD), magnetic tape, solid-state drive (SSD), RAM drive, secure digital card or drive, any other suitable computer The non-transitory storage medium may be read, or any suitable combination of two or more of these. Computer readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, as appropriate.

other

Herein, unless expressly indicated otherwise or the context dictates otherwise, "or" is inclusive and not exclusive. Thus, herein "A or B" means "A, B, or both" unless expressly indicated otherwise or the context dictates otherwise. Further, "and" means both jointly and each unless expressly indicated otherwise or the context dictates otherwise. Thus, herein "A and B" means "A and B, jointly or separately," unless expressly indicated otherwise or the context dictates otherwise.

The scope of the present invention encompasses all changes, substitutions, changes, alterations and modifications of the example embodiments described or illustrated herein that would occur to one of ordinary skill in the art. The scope of the invention is not limited to the example embodiments described or illustrated herein. Furthermore, although the present disclosure has described and illustrated various embodiments herein as including particular components, elements, features, functions, operations or steps, any of such embodiments may include one of ordinary skill in the art. any combination or permutation of any of the components, elements, features, functions, operations or steps described or illustrated anywhere herein. Furthermore, references in the appended claims to an apparatus or system or a component of an apparatus or system adapted, configured, able, configured, enabled, operated, or operable to perform a particular function Covers that a device, system, component (regardless of its or that specific function) is activated, switched on, or unlock. Additionally, although particular embodiments are described or illustrated herein as providing particular advantages, particular embodiments may provide any, some, or all of such advantages.

100A: Artificial Reality Systems 100B: Example Augmented Reality System 102: user 104: Headwear group 106: Controller 108: Computing system 110:Head-mounted display 112: frame 114: Display 210: Application 211: Image 213:Movement vector and depth information 220: Operating system 221: Image 223: Image 311: UV depth map 312: Moving vector 313: View projection matrix 314: Three-dimensional position 321: Depth map 323:View projection matrix 324: Three-dimensional position 334: Estimated 3D position 335:View projection matrix 337:Distortion Mesh 600: method 610: Step 620: Step 630: step 640: step 650: step 700: Computer system 702: Processor 704: memory 706: Storage 708: Input/Output Interface 710: communication interface 712: busbar

[FIG. 1A] An example artificial reality system is illustrated.

[FIG. 1B] Illustrates an example augmented reality system.

[FIG. 2] An overview illustrating an example of frame extrapolation and reprojection using motion vector and depth information generated by an application.

[Fig. 3] Example data flow illustrating frame extrapolation.

[FIG. 4] Illustrates an example estimation of the future location of an object based on its past location and current location.

[FIG. 5] illustrates an example comparison of time budget per frame between a full frame rendering application and a half frame rendering application.

[FIG. 6] Illustrates an example method for outlining an inset based on motion vectors and depth maps generated by an application.

[FIG. 7] An example computer system is illustrated.

600: method

610: Step

620: Step

630: step

640: step

650: step

Claims (20)

A method comprising, by a computing system associated with a wearable device: receiving a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application; calculating a current three-dimensional position of an object corresponding to the current frame using the depth map for an object represented in the rendered image; calculating a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map; estimating a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object; and generating an extrapolated image corresponding to the future frame by reprojecting the object represented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object . The method of claim 1, wherein the motion vector data and the depth map are generated based on a three-dimensional object presented by the application program. The method according to claim 2, wherein the motion vector in the motion vector data is three-dimensional. The method of claim 1, further comprising processing the received motion vector data and the depth map such that a region corresponding to a foreground of the rendered image is expanded. The method of claim 1, wherein calculating the current three-dimensional position of the object includes back-projecting the depth map from a current viewpoint associated with the current frame onto a three-dimensional space. The method of claim 5, wherein the current viewpoint is associated with a position and an orientation of the wearable device at a time instant when the current frame is presented. The method of claim 1, wherein calculating the past three-dimensional position of the object includes: generating an estimated depth map corresponding to the past frame by subtracting the motion vector from the depth map; and The estimated depth map is backprojected onto a three-dimensional space from a past viewpoint associated with the past frame. The method of claim 1, wherein estimating the future three-dimensional position of the object is based on an assumption that the object moves at a constant speed from a time instant corresponding to the past frame to a time instant corresponding to the future frame Box one of the instants of time. The method of claim 8, further comprising generating a distortion mesh by projecting the estimated future three-dimensional position of the object onto the future viewpoint. The method of claim 9, wherein generating the extrapolated image corresponding to the future frame comprises applying the distortion grid to the rendered image. One or more computer-readable non-transitory storage media containing software operable when executed to: receiving a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application; calculating a current three-dimensional position of an object corresponding to the current frame using the depth map for an object represented in the rendered image; calculating a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map; estimating a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object; and generating an extrapolated image corresponding to the future frame by reprojecting the object represented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object . The medium of claim 11, wherein the motion vector data and the depth map are generated based on a three-dimensional object presented by the application. The medium according to claim 12, wherein the motion vector in the motion vector data is three-dimensional. The medium of claim 11, wherein the software, when executed, is further operable to process the received motion vector data and the depth map such that a region corresponding to a foreground of the rendered image is expanded. The medium of claim 11, wherein calculating the current three-dimensional position of the object includes back-projecting the depth map from a current viewpoint associated with the current frame onto a three-dimensional space. The medium of claim 15, wherein the current viewpoint is associated with a position and an orientation of the wearable device at an instant in time when the current frame is presented. The method of claim 11, wherein calculating the past three-dimensional position of the object comprises: generating an estimated depth map corresponding to the past frame by subtracting the motion vector from the depth map; and The estimated depth map is backprojected onto a three-dimensional space from a past viewpoint associated with the past frame. The medium of claim 11, wherein estimating the future three-dimensional position of the object is based on an assumption that the object moves at a constant speed from a time instant corresponding to the past frame to a time instant corresponding to the future frame Box one of the instants of time. The medium of claim 18, wherein the software, when executed, is further operable to: generate a distortion mesh by projecting the estimated future three-dimensional position of the object onto the future viewpoint. A system comprising: one or more processors; and a non-transitory memory coupled to the processors, the non-transitory memory containing instructions executable by the processors, the processors in When executing these instructions, you can operate to perform the following operations: receiving a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application; calculating a current three-dimensional position of an object corresponding to the current frame using the depth map for an object represented in the rendered image; calculating a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map; estimating a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object; and generating an extrapolated image corresponding to the future frame by reprojecting the object represented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object .

Images