CN117425869A - Dynamic over-rendering in post-distortion - Google Patents

Abstract

A method for adjusting an over-rendered area of a display in an AR device is described. The method comprises the following steps: identifying an angular velocity of the display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and adjusting the size of the dynamic over-rendered region based on a combination of the angular velocity, the most recent gesture, the previous warp gesture, and the previous over-rendered region.

Description

Dynamic over-rendering in post-distortion

RELATED APPLICATIONS

The present application claims priority to U.S. patent application Ser. No. 17/530,242, filed 11/18 of 2021, which claims priority to U.S. provisional patent application Ser. No. 63/189,928, filed 5/18 of 2021, each of which is incorporated herein by reference in its entirety.

Technical Field

The subject matter disclosed herein relates generally to display systems. In particular, the present disclosure proposes systems and methods for reducing motion-to-photon delay in Augmented Reality (AR) devices.

Background

Augmented Reality (AR) systems present virtual content to augment a user's real world environment. For example, virtual content overlaid over a physical object may be used to create an illusion that the physical object is moving, animating, and the like. The AR device worn by the user continually updates the presentation of the virtual content based on the user's movements to create the illusion that the virtual content is physically present in the user's real world environment. For example, as users move their heads, the AR device updates the presentation of the virtual content to create the illusion that the virtual content remains in the same geographic location in the user's real world environment. Thus, the user may move around the virtual object presented by the AR device in the same manner that the user would around the physical object.

In order to convincingly create the illusion that the virtual object is in the user's real world environment, the AR device must update the presentation of the virtual object almost instantaneously as the device moves. However, the virtual content may require a longer time to update because the AR device must process the environment data, render the virtual content, and then project the virtual content. The process creates a delay between the time that the physical object is tracked by the AR device and the time that the rendered virtual object is displayed in the display of the AR device. This delay is also referred to as the "motion-to-photon delay". Any perceived motion-to-photon delay can degrade the user experience.

Drawings

To readily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which that element is first introduced.

Fig. 1 is a block diagram illustrating an environment for operating an AR device according to an example embodiment.

Fig. 2 is a block diagram illustrating an AR/VR display device in accordance with an example embodiment.

FIG. 3 is a block diagram illustrating a visual tracking system according to an example embodiment.

Fig. 4 is a block diagram illustrating a display controller according to an example embodiment.

FIG. 5 is a block diagram illustrating a dynamic over-rendering module according to an example embodiment.

FIG. 6 is a block diagram illustrating a dynamic over-rendering process according to an example embodiment.

FIG. 7 is a flowchart illustrating a process for dynamic over-rendering according to an example embodiment.

FIG. 8 illustrates an example of a dynamic over-rendering region according to an example embodiment.

FIG. 9 illustrates an example of a dynamic over-rendering region according to an example embodiment.

Fig. 10 is a flowchart illustrating a method for rendering an image in a dynamic over-rendering area according to an example embodiment.

Fig. 11 is a flowchart illustrating a method for rendering an image in a dynamic over-rendering area according to an example embodiment.

Fig. 12 illustrates a network environment in which a head wearable device may be implemented, according to an example embodiment.

Fig. 13 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to an example embodiment.

FIG. 14 is a diagrammatic representation of machine in the form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed according to an example embodiment.

Detailed Description

The following description describes systems, methods, techniques, sequences of instructions, and computer program products that illustrate example implementations of the present subject matter. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the present subject matter. It will be apparent, however, to one skilled in the art that embodiments of the present subject matter may be practiced without some or other of these specific details. Examples merely typify possible variations. Structures (e.g., structural components, such as modules) are optional and may be combined or sub-divided, and operations (e.g., in a process, algorithm, or other function) may be varied in sequence or combined or sub-divided, unless explicitly stated otherwise.

The term "augmented reality" (AR) is used herein to refer to an interactive experience of a real-world environment in which physical objects residing in the real world are "augmented" or enhanced by computer-generated digital content (also referred to as virtual content or synthetic content). AR may also refer to a system that enables a combination of real world and virtual world, real-time interaction, and 3D registration of virtual objects and real objects. AR creates the illusion that virtual content is physically present in the user's real world environment and appears to attach or interact with the user's real world environment.

The term "virtual reality" (VR) is used herein to refer to a simulated experience of a virtual world environment that is completely different from a real world environment. Computer-generated digital content is displayed in a virtual world environment. VR also refers to a system that enables a user of a VR system to be fully immersed in a virtual world environment and interact with virtual objects that exist in the virtual world environment.

The term "AR application" is used herein to refer to a computer-operated application that implements an AR experience. The term "VR application" is used herein to refer to a computer-operated application that implements a VR experience. The term "AR/VR application" refers to a computer-operated application that implements an AR experience or a combination of VR experiences.

The term "AR display device" (also referred to as "AR device") is used herein to refer to a computing device that operates an AR application. The term "VR display device" (also referred to as "VR device") is used herein to refer to a computing device that operates a VR application. The term "AR/VR display device" (also referred to as "AR/VR device") is used herein to refer to a computing device that operates a combination of AR applications and VR applications.

The term "visual tracking system" (also referred to as a "visual tracking device") is used herein to refer to a computer-operated application that tracks visual features identified in images captured by one or more cameras of the visual tracking system. The vision tracking system builds a model of the real world environment based on the tracked vision features. Non-limiting examples of vision tracking systems include: a visual synchrony positioning and mapping system (VSLAM) and a Visual Inertial Odometer (VIO) system. The VSLAM may be used to construct a target from an environment or scene based on one or more cameras of the visual tracking system. VIO (also known as visual inertial tracking) determines the latest pose (e.g., position and orientation) of a device based on data acquired from a plurality of sensors (e.g., optical sensors, inertial sensors) of the device.

The term "inertial measurement unit" (IMU) is used herein to refer to a device that can report the inertial state of a moving body, including acceleration, speed, orientation, and positioning of the moving body. The IMU enables tracking of the motion of the subject by integrating the acceleration and angular velocity measured by the IMU. IMU may also refer to a combination of accelerometers and gyroscopes, which may determine and quantify linear acceleration and angular velocity, respectively. The values obtained from the IMU gyroscope may be processed to obtain the pitch, roll, and heading of the IMU, and thus the pitch, roll, and heading of the subject with which the IMU is associated. Signals from the accelerometer of the IMU may also be processed to obtain the velocity and displacement of the IMU.

The term "motion-to-photon delay" (M2P delay) is used herein to refer to the duration between a user moving a visual tracking device and its presentation of virtual content adapting to that particular motion. Motion-to-photon delay may also refer to the delay associated with presenting virtual content in an AR device. As the user moves the AR device, the view of the user's real world environment changes instantaneously. However, the virtual content requires a longer time to update because the AR device must process the environmental data with the IMU data, render the virtual content, and project the virtual content in front of the user's field of view. Motion-to-photon delay can cause the virtual content to appear jittery or lagging and degrade the user's AR experience.

The term "time-warping" (also referred to as "time-warping)", "late-warping") is used herein to refer to a re-projection technique that warps the rendered image before sending it to the display to correct for head movement that occurs after rendering. The process takes the rendered image, modifies it with the rotation data that was most recently collected from the IMU, and then displays the warped image on the screen.

One problem with implementing AR is the delay associated with presenting virtual content. As the user moves the AR device, the view of the user's real world environment changes instantaneously. The virtual content requires a longer time to change because the AR display device must process the environmental data with Inertial Measurement Unit (IMU) data, render the virtual content, and project the virtual content in front of the user's field of view. This delay is referred to herein as an M2P delay.

As mentioned previously, late warping compensates for M2P delays by warping the rendered image for pose differences. The warping process may result in lost information for some pixels that then typically remain black. "over-rendering" is a technique for rendering more pixels than actually displayed on a display to be able to fill all pixels on the screen during post-warping. However, the over-rendered area may still be large. A larger over-rendered area (more than necessary) results in greater memory and power consumption.

The present application describes a method for dynamically adjusting the size of the over-rendered area for which additional pixels are rendered in order to reduce the number of shadow pixels while keeping the number of black pixels after delay warping small. For example, the present system defines a dynamic over-rendering function for a particular rendering time based on: (a) the most recent pose and linear/angular velocity, (b) the previous warp pose, and (c) the previous over-rendered region. Prediction inaccuracy and reduced rendering rate, the required amount of over-rendering depend on the maximum prediction error or maximum pose difference between the two warped poses (both of which depend on the dynamics of the head motion of the AR device).

In one example embodiment, a method for adjusting an over-rendered region of a display in an AR device is described. The method comprises the following steps: identifying an angular velocity of the display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and adjusting the size of the dynamic over-rendered region based on a combination of the angular velocity, the most recent gesture, the previous warp gesture, and the previous over-rendered region.

As a result, one or more of the methods described herein help address the technical problem of power consumption savings by limiting the size of the over-rendered area based on the pose difference and angular/linear velocity of the AR device. The presently described methods provide improvements in the operation of the functionality of a computing device by reducing computing costs, power consumption, and thermal impact. Thus, one or more of the methods described herein may avoid the need for certain effort or computing resources. Examples of such computing resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

Fig. 1 is a network diagram illustrating an environment 100 suitable for operating an AR device 106, according to some example embodiments. The environment 100 includes a user 102, an AR device 106, and a physical object 104. The user 102 operates the AR device 106. The user 102 may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the AR device 106), or any suitable combination thereof (e.g., a human aided by a machine or a machine supervised by a human). The user 102 is associated with an AR device 106.

The AR device 106 may be a computing device with a display, such as a smart phone, a tablet computer, or a wearable computing device (e.g., a watch or glasses). The computing device may be handheld or may be removably mounted to the head of the user 102. In one example, the display includes a screen that displays an image captured with the camera of the AR device 106. In another example, the display of the device may be transparent, such as in the lenses of wearable computing eyewear. In other examples, the display may be opaque, partially transparent, partially opaque. In other examples, the display may be wearable by the user 102 to cover the field of view of the user 102.

The AR device 106 includes an AR application that generates virtual content based on images detected with the imaging apparatus of the AR device 106. For example, the user 102 may direct the camera of the AR device 106 to capture an image of the physical object 104. The AR application generates virtual content corresponding to the identified object (e.g., physical object 104) in the image and presents the virtual content in a display of AR device 106.

The AR device 106 includes a vision tracking system 108. The vision tracking system 108 tracks the pose (e.g., position and orientation) of the AR device 106 relative to the real world environment 110 using, for example, optical sensors (e.g., depth-enabled 3D camera, image camera), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (bluetooth, wi-Fi), GPS sensors, and audio sensors. In one example, the AR device 106 displays virtual content based on the pose of the AR device 106 with respect to the real world environment 110 and/or the physical object 104.

Any of the machines, databases, or devices illustrated in fig. 1 may be implemented in a general-purpose computer modified (e.g., configured or programmed) by software to be a special-purpose computer to perform one or more of the functions described herein for the machine, database, or device. For example, a computer system capable of implementing any one or more of the methods described herein is discussed below with reference to fig. 10-11. As used herein, a "database" is a data storage resource and may store data structured as text files, tables, spreadsheets, relational databases (e.g., object-relational databases), triad stores, hierarchical data stores, or any suitable combination thereof. Furthermore, any two or more of the machines, databases, or devices illustrated in fig. 1 may be combined into a single machine, and the functionality described herein with respect to any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

The AR device 106 may operate over a computer network. The computer network may be any network that enables communication between or among machines, databases, and devices. Thus, the computer network may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The computer network may include one or more portions that constitute a private network, a public network (e.g., the internet), or any suitable combination thereof.

Fig. 2 is a block diagram illustrating modules (e.g., components) of AR device 106 according to some example embodiments. The AR device 106 includes a sensor 202, a display 204, a processor 208, a graphics processing unit 216, a display controller 218, and a storage device 206. Examples of AR devices 106 include wearable computing devices (e.g., glasses), tablet computers, navigation devices, portable media devices, or smart phones.

The sensors 202 include, for example, optical sensors 212 (e.g., imaging devices such as color imaging devices, thermal imaging devices, depth sensors, and one or more gray scale, global shutter tracking imaging devices) and inertial sensors 214 (e.g., gyroscopes, accelerometers, magnetometers). Other examples of sensors 202 include proximity sensors or location sensors (e.g., near field communication, GPS, bluetooth, wifi), audio sensors (e.g., microphones), or any suitable combination thereof. Note that the sensor 202 described herein is for illustration purposes, and thus the sensor 202 is not limited to the above-described sensor.

The display 204 includes a screen or monitor configured to display images generated by the processor 208. In one example embodiment, the display 204 may be transparent or translucent so that the user 102 may view (in the AR use case) through the display 204. In another example, the display 204 (e.g., an LCOS display) presents each frame of virtual content in multiple presentations.

The processor 208 includes an AR application 210 and the visual tracking system 108. The AR application 210 uses computer vision to detect and identify the physical environment or physical object 104. The AR application 210 retrieves a virtual object (e.g., a 3D object model) based on the identified physical object 104 or physical environment. The AR application 210 presents the virtual object in the display 204. For AR applications, AR application 210 includes a local rendering engine that renders a 3D model of a virtual object overlaid (e.g., superimposed on or otherwise displayed with) on an image or view of physical object 104. The view of the virtual object may be manipulated by adjusting the positioning of the physical object 104 (e.g., its physical position, orientation, or both) relative to the optical sensor 212. Similarly, the view of the virtual object may be manipulated by adjusting the pose of the AR device 106 relative to 104. For VR applications, AR application 210 displays the virtual object in display 204 at a location (in display 204) determined based on the pose of AR device 106.

In one example implementation, the visual tracking system 108 estimates the pose of the AR device 106. For example, the vision tracking system 108 uses image data from the optical sensor 212 and the inertial sensor 214 and corresponding inertial data to track the position and pose of the AR device 106 relative to a frame of reference (e.g., the real-world environment 110). In one example, the visual tracking system 108 uses the sensor data to determine a three-dimensional pose of the AR device 106. The three-dimensional pose is a determined orientation and positioning of the AR device 106 relative to the user's real world environment 110. For example, the AR device 106 may use images of the user's real world environment 110 and other sensor data to identify the relative positioning and orientation of the AR device 106 and physical objects in the real world environment 110 surrounding the AR device 106. The vision tracking system 108 continuously collects and uses updated sensor data describing the movement of the AR device 106 to determine an updated three-dimensional pose of the AR device 106 that is indicative of changes in the relative positioning and orientation of the AR device 106 and physical objects in the real-world environment 110. The visual tracking system 108 provides the three-dimensional pose of the AR device 106 to the graphics processing unit 216.

Graphics processing unit 216 includes a rendering engine (not shown) configured to render frames of a 3D model of a virtual object based on virtual content provided by AR application 210 and gestures provided by visual tracking system 108. In other words, the graphics processing unit 216 uses the three-dimensional pose of the AR device 106 to generate frames of virtual content to be presented on the display 204. For example, the graphics processing unit 216 uses three-dimensional gestures to render frames of virtual content such that the virtual content is presented in the display 204 in the proper orientation and positioning to appropriately enhance the user's authenticity. As an example, the graphics processing unit 216 may render frames of virtual content using three-dimensional gesture data such that the virtual content overlaps physical objects in the user's real-world environment 110 when presented on the display 204. The graphics processing unit 216 generates an update frame of virtual content based on the updated three-dimensional pose of the AR device 106 that reflects changes in the position and orientation of the user relative to physical objects in the user's real-world environment 110. Graphics processing unit 216 communicates the rendered frames to display controller 218.

The display controller 218 is positioned as an intermediary mechanism between the graphics processing unit 216 and the display 204. The display controller 218 receives image data (e.g., rendered frames) from the graphics processing unit 216 and distorts the rendered frames by readjusting the position of the rendered virtual content in the time-warped frames. The display controller 218 performs a post-distortion transformation based on the latest pose of the AR device 106 and the latest tracking information (of the tracked physical object, the preset animation of the virtual object, the plurality of physical objects with different motions). The display controller 218 provides the time-warped frame to the display 204 for display.

The storage device 206 stores virtual object content 220. The virtual object content 220 includes, for example, a visual reference (e.g., image, QR code) and a database of corresponding virtual content (e.g., three-dimensional model of virtual object).

Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any of the modules described herein may configure a processor to perform the operations described herein for that module. Furthermore, any two or more of these modules may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

FIG. 3 is a block diagram illustrating a visual tracking system according to an example embodiment. The vision tracking system 108 includes, for example, an inertial sensor module 302, an optical sensor module 304, and a pose estimation module 306. The inertial sensor module 302 accesses inertial sensor data from the inertial sensor 214. The optical sensor module 304 accesses optical sensor data from the optical sensor 212.

The pose estimation module 306 determines a pose (e.g., position, location, orientation) of the AR device 106 relative to a frame of reference (e.g., real world environment 110). In one example implementation, pose estimation module 306 includes a VIO system that estimates the pose of AR device 106 based on a 3D map of feature points from an image captured with optical sensor 212 and inertial sensor data captured with inertial sensor 214.

For example, gesture estimation module 306 operates an application that tracks visual features (e.g., 3D feature points) identified in images captured by optical sensor 212 of AR device 106. The vision tracking system 108 builds a model of the real world environment based on the tracked visual features. The vision tracking system 108 includes, for example, a vision synchronized locating and mapping system (VSLAM) or a Vision Inertial Odometer (VIO) system. The VSLAM may be used to construct a target from an environment or scene based on one or more cameras of the visual tracking system 108. VIO (also known as visual inertial tracking) determines the latest pose (e.g., position and orientation) of a device based on data acquired from a plurality of sensors (e.g., optical sensors, inertial sensors) of the device.

Fig. 4 is a block diagram illustrating a display controller 218 according to an example embodiment. The display controller 218 includes a dynamic over-rendering engine 402 and a time warping engine 404.

The time warp engine 404 accesses the latest pose of the AR device 106 from the vision tracking system 108. The time warping engine 404 re-projects the rendered frame generated with the graphics processing unit 216 based on the pose of the AR device 106 (used to calculate the rendered frame) and the most recent pose of the AR device 106 (provided to the display controller 218 by the vision tracking system 108).

The dynamic over-rendering engine 402 dynamically adjusts the size of the over-rendered region of the rendered frame. In order for the time warp to operate properly, the size of the rendered frame should be larger than the size of the display 204. Otherwise, when the time warp engine 404 shifts the rendered image, the time warp engine 404 may eventually shift the empty pixels into the visible region. In one example embodiment, dynamic over-rendering engine 402 adjusts the size of the rendered frame based on a combination of: (a) the most recent pose of the vision tracking system 108 and the linear/angular velocity of the vision tracking system 108, (b) the previous warped pose of the vision tracking system 108, and (c) the previous over-rendered region from the dynamic over-rendering engine 402. The adjusted over-rendered frames are provided to the display 204 for display. Example components of dynamic over-rendering engine 402 are further described below with reference to fig. 5.

FIG. 5 is a block diagram illustrating a dynamic over-rendering engine 402 according to one example embodiment. Dynamic over-rendering engine 402 includes a recent pose module 508, an angular velocity module 510, a warped pose history module 512, an over-rendering region history module 514, and an over-rendering region estimator 502.

The recent gesture module 508 recognizes the recent gesture of the AR device 106. For example, the most recent pose module 508 accesses the most recent pose of the AR device 106 based on the most recent VIO data from the visual tracking system 108.

Angular velocity module 510 identifies the angular velocity of AR device 106. In one example, the angular velocity module 510 obtains an angular velocity from the IMU. In another example, the angular velocity module 510 estimates the angular velocity from VIO data (from the vision tracking system 108).

The warp gesture history module 512 identifies warp gestures previously generated by the time warp engine 404. For example, the distortion gesture history module 512 retrieves a distortion gesture corresponding to the last gesture identified by the visual tracking system 108. The distortion gesture history module 512 may store a preset number of last distortion gestures in the storage device 206.

The over-rendered region history module 514 identifies over-rendered regions previously generated by the dynamic over-rendering engine 402. For example, the over-rendered region history module 514 retrieves an over-rendered region corresponding to the last warped gesture or the last gesture. The over-rendered region history module 514 may store a preset number of last over-rendered regions in the storage device 206.

The over-rendered region estimator 502 accesses the most recent pose data from the most recent pose module 508, the angular velocity data from the angular velocity module 510, the warp pose history data from the warp pose history module 512, and the over-rendered region history data from the over-rendered region history module 514. The over-rendering region estimator 502 adjusts the size of the over-rendering region based on the latest pose data, the angular velocity data, the warp pose history data, and the over-rendering region history data.

In one example implementation, the over-rendered region estimator 502 includes a heuristic module 504 and a machine learning module 506. Heuristic module 504 performs a heuristic-based calculation method on the most recent gesture data, angular velocity data, warp gesture history data, and over-rendered region history data to estimate the desired region that needs to be rendered by time warp engine 404. In another example, the heuristic module 504 performs a statistical analysis on the received data to estimate the desired area.

The machine learning module 506 generates training data based on the most recent pose data, the angular velocity data, the warp pose history data, and the over-rendered region history data. The machine learning module 506 uses the training data to train a model that identifies the size of the over-rendered region based on different parameters. The trained model may then be used to identify the best render-through region. In another example, the over-rendered region estimator 502 uses a combination of a heuristic module 504 and a machine learning module 506 to estimate the optimal over-rendered region.

FIG. 6 is a block diagram illustrating an example process according to an example embodiment. The visual tracking system 108 receives sensor data from the sensors 202 to determine the pose (e.g., pose a) of the AR device 106. The vision tracking system 108 provides gestures to the graphics processing unit 216. Graphics processing unit 216 uses 3D rendering engine 602 to render frames (e.g., frame a) of virtual content (provided by AR application 210) at a first location (in display 204) based on a rotational gesture (e.g., gesture a) determined with vision tracking system 108. Graphics processing unit 216 provides the rendered frame (e.g., frame a) to display controller 218.

The display controller 218 receives the latest gesture (e.g., gesture b) from the vision tracking system 108. In other words, gesture b is a gesture of AR device 106 that is closer than gesture a. The dynamic over-rendering engine 402 dynamically adjusts the size of the over-rendered region of the rendered frame.

The time warping engine 404 re-projects the rendered frame by performing a time warping operation on the rendered frame (e.g., frame a) to generate a new frame (e.g., frame b). The display controller 218 communicates frame b to the display 204 for display.

FIG. 7 is a flowchart illustrating a process 700 for dynamic over-rendering according to an example embodiment. In block 702, the visual tracking system 108 identifies the angular velocity and the most recent pose of the AR device 106. In block 706, dynamic over-rendering engine 402 identifies a previous warp gesture. In block 708, dynamic over-rendering engine 402 identifies a previous over-rendered region. In block 704, dynamic over-rendering engine 402 dynamically adjusts the over-rendered region based on the most recent gesture, angular velocity, previous warped gesture, or previous over-rendered region.

It is noted that other embodiments may use different ordering, additional or fewer operations, and different naming or terminology to achieve similar functionality. In some implementations, various operations may be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein were chosen to illustrate some principles of operation in a simplified form.

FIG. 8 illustrates an example of a dynamic over-rendering region according to an example embodiment. For example, the size of dynamic over-rendering region 806 is smaller than the size of over-rendering region 802 and larger than screen size 804.

FIG. 9 illustrates an example of a dynamic over-rendering region according to an example embodiment. The AR device 106 is pointed to view 902 at tn to capture an image of the physical object 908 at time tn. The screen 910 of the AR device 106 displays an image of the rendered virtual object corresponding to the physical object 908 at a first location.

The AR device 106 is pointed to view 906 at tn+1 to capture an image of the physical object 908 at time tn+1. The screen 910 of the AR device 106 displays an image of the rendered virtual object corresponding to the physical object 908 at a second location.

The vision tracking system 108 determines the view 904 at the rendering pose and combines the over-rendered region 912 of tn (corresponding to the warp pose tn) with the over-rendered region 914 of tn+1 (corresponding to the warp pose tn+1).

Fig. 10 is a flowchart illustrating a method 1000 for rendering an image in a dynamic over-rendering region according to an example embodiment. The operations in method 1000 may be performed by AR device 106 using components (e.g., modules, engines) described above with reference to fig. 2. Thus, the method 1000 is described by way of example with reference to the AR device 106. However, it should be understood that at least some of the operations of method 1000 may be deployed on various other hardware configurations or performed by similar components residing elsewhere.

In block 1002, the AR device 106 renders an image in a dynamic over-rendering region. For example, the graphics processing unit 216 renders images. In block 1004, the AR device 106 applies a transformation to the rendered image based on the most recent pose of the AR device 106. For example, the display controller 218 determines the size of the over-rendered region and applies a time warping algorithm to the rendered image corresponding to the over-rendered region. In block 1006, the AR device 106 displays the transformed image. For example, the display 204 projects a distorted image.

It is noted that other embodiments may use different ordering, additional or fewer operations, and different naming or terminology to achieve similar functionality. In some implementations, various operations may be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein were chosen to illustrate some principles of operation in a simplified form.

Fig. 11 is a flowchart illustrating a method 1100 for rendering an image in a dynamic over-rendering region according to an example embodiment. The operations in method 1100 may be performed by AR device 106 using components (e.g., modules, engines) described above with reference to fig. 2. Thus, the method 1100 is described by way of example with reference to the AR device 106. However, it should be understood that at least some of the operations of method 1100 may be deployed on various other hardware configurations or performed by similar components residing elsewhere.

In block 1102, the AR device 106 determines a first pose. In one example implementation, the visual tracking system 108 identifies a first pose of the AR device 106. In block 1104, the AR device 106 accesses the virtual content. In one example implementation, the AR application 210 accesses virtual object content 220 from the storage device 206.

In block 1106, the AR device 106 identifies the size of the dynamic over-rendered region. In one example embodiment, dynamic over-rendering engine 402 determines the size of the over-rendered region. In block 1108, the AR device 106 uses the rendering engine of the graphics processing unit 216 to render a 3D model of the virtual content in the first frame in the dynamic over-rendering region. In block 1110, the time warp engine 404 applies a transformation to the rendered 3D model.

System with head wearable device

Fig. 12 illustrates a network environment 1200 in which a head wearable device 1202 may be implemented, according to an example embodiment. Fig. 12 is a high-level functional block diagram of an example head wearable apparatus 1202 that communicatively couples a mobile client device 1238 and a server system 1232 via various networks 1240.

The head wearable device 1202 includes an imaging device, such as at least one of a visible light imaging device 1212, an infrared emitter 1214, and an infrared imaging device 1216. The client device 1238 may be capable of connecting with the head wearable apparatus 1202 using both communication 1234 and communication 1236. Client device 1238 is connected to server system 1232 and network 1240. Network 1240 may include any combination of wired and wireless connections.

The head wearable device 1202 also includes two image displays of the image display 1204 of the optical assembly. The two image displays of the image display 1204 of the optical assembly include one image display associated with the left lateral side and one image display associated with the right lateral side of the head wearable device 1202. Head wearable device 1202 also includes image display driver 1208, image processor 1210, low power circuitry 1226, and high speed circuitry 1218. The image display 1204 of the optical assembly is used to present images and video to a user of the head wearable device 1202, including images that may include a graphical user interface.

The image display driver 1208 commands and controls the image display of the image display 1204 of the optical assembly. The image display driver 1208 may deliver image data directly to the image display of the image display 1204 of the optical assembly for presentation, or may have to convert the image data into a signal or data format suitable for delivery to an image display device. For example, the image data may be video data formatted according to a compression format such as h.264 (MPEG-4), HEVC, theora, dirac, realVideo RV40, VP8, VP9, etc., and the still image data may be formatted according to a compression format such as Portable Network Group (PNG), joint Photographic Experts Group (JPEG), tag Image File Format (TIFF), or exchangeable image file format (Exif), etc.

As described above, the head wearable device 1202 includes a frame and a handle (or temple) extending from a lateral side of the frame. Head wearable apparatus 1202 also includes a user input device 1206 (e.g., a touch sensor or button), user input device 1206 including an input surface on head wearable apparatus 1202. The user input device 1206 (e.g., a touch sensor or button) is a graphical user interface for receiving input selections from a user to manipulate the presented image.

The components shown in fig. 12 for the head wearable device 1202 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in a frame or temple. Alternatively or additionally, the depicted components may be located in a block, frame, hinge, or bridge of head wearable device 1202. The left and right sides may include digital camera elements such as Complementary Metal Oxide Semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible or light capturing element that may be used to capture data including images of a scene with unknown objects.

The head wearable device 1202 includes a memory 1222, the memory 1222 storing instructions to perform a subset or all of the functions described herein. Memory 1222 may also include a storage device.

As shown in fig. 12, high-speed circuitry 1218 includes a high-speed processor 1220, memory 1222, and high-speed wireless circuitry 1224. In an example, the image display driver 1208 is coupled to the high speed circuitry 1218 and operated by the high speed processor 1220 to drive the left and right image displays of the image display 1204 of the optical assembly. High-speed processor 1220 may be any processor capable of managing the operation and high-speed communication of any general computing system required by head wearable device 1202. The high speed processor 1220 includes processing resources required to manage high speed data transmission over communications 1236 to a Wireless Local Area Network (WLAN) using high speed wireless circuitry 1224. In some examples, the high-speed processor 1220 executes an operating system (such as the LINUX operating system or other such operating system of the head wearable device 1202) and the operating system is stored in the memory 1222 for execution. The high-speed processor 1220 executing the software architecture of the head wearable device 1202 is used to manage data transmission with the high-speed wireless circuitry 1224, among any other responsibilities. In some examples, the high-speed wireless circuitry 1224 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard (also referred to herein as Wi-Fi). In other examples, other high-speed communication standards may be implemented by high-speed wireless circuitry 1224.

The low power wireless circuitry 1230 and high speed wireless circuitry 1224 of the head wearable device 1202 may include a short range transceiver (bluetooth ^TM ) And a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). Details of the architecture of head wearable apparatus 1202 may be used to implement client device 1238, including transceivers that communicate via communications 1234 and 1236, as may other elements of network 1240.

The memory 1222 includes any storage devices capable of storing various data and applications including, among other things, camera data generated by the left and right infrared cameras 1216 and the image processor 1210, and images generated by the image display driver 1208 for display on the image display 1204 of the optical assembly. Although memory 1222 is shown as being integrated with high-speed circuitry 1218, in other examples, memory 1222 may be a separate, stand-alone element of head wearable device 1202. In some such examples, the circuit by wire may provide a connection from the image processor 1210 or the low power processor 1228 to the memory 1222 through a chip including the high speed processor 1220. In other examples, high-speed processor 1220 may manage addressing of memory 1222 such that low-power processor 1228 will boot high-speed processor 1220 at any time that a read or write operation involving memory 1222 is required.

As shown in fig. 12, a low power processor 1228 or a high speed processor 1220 of the head wearable apparatus 1202 may be coupled to the camera (visible light camera 1212; infrared emitter 1214 or infrared camera 1216), the image display driver 1208, the user input device 1206 (e.g., touch sensor or button), and the memory 1222.

The head wearable device 1202 is connected with a host computer. For example, head wearable apparatus 1202 pairs with client device 1238 via communication 1236 or connects to server system 1232 via network 1240. The server system 1232 may be one or more computing devices that are part of a service or network computing system, e.g., including a processor, memory, and a network communication interface that communicates with the client device 1238 and the head wearable 1202 over the network 1240.

Client device 1238 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication through the network 1240, the communication 1234 or the communication 1236. The client device 1238 may also store at least a portion of the instructions for generating the binaural audio content in a memory of the client device 1238 to implement the functionality described herein.

The output components of the head wearable device 1202 include visual components such as a display (such as a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), a Light Emitting Diode (LED) display, a projector, or a waveguide). The image display of the optical assembly is driven by an image display driver 1208. The output components of head wearable device 1202 also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, and the like. The input components of the head wearable apparatus 1202, client device 1238, and server system 1232 (such as user input device 1206) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optoelectronic keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., physical buttons, a touch screen providing position and force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), and the like.

Head wearable apparatus 1202 may optionally include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with head wearable device 1202. For example, a peripheral element may include any I/O component including an output component, a motion component, a positioning component, or any other such element described herein.

For example, biometric components include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biological signals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. The motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. The positioning component includes a position sensor component (e.g., a Global Positioning System (GPS) receiver component) that generates position coordinates, wiFi or bluetooth that generates positioning system coordinates ^TM Transceivers, altitude sensor components (e.g., altimeters or barometers that detect barometric pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and so forth. Such positioning system coordinates may also be received from client device 1238 via communication 1236 via low power wireless circuitry 1230 or high speed wireless circuitry 1224.

Where a phrase similar to "at least one of A, B or C", "at least one of A, B and C", "one or more A, B or C", or "one or more of A, B and C" is used, the phrase is intended to be construed to mean that a may be present in an embodiment alone, B may be present in an embodiment alone, C may be present in an embodiment alone, or any combination of elements A, B and C may be present in a single embodiment; for example, a and B, A and C, B and C, or a and B and C.

Changes and modifications may be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Fig. 13 is a block diagram 1300 illustrating a software architecture 1304 that may be installed on any one or more of the devices described herein. The software architecture 1304 is supported by hardware, such as a machine 1302, the machine 1302 including a processor 1320, memory 1326 and I/O components 1338. In this example, the software architecture 1304 may be conceptualized as a stack of layers, with each layer providing a particular function. The software architecture 1304 includes layers such as an operating system 1312, libraries 1310, frameworks 1308, and applications 1306. In operation, the application 1306 activates an API call 1350 through the software stack and receives a message 1352 in response to the API call 1350.

Operating system 1312 manages hardware resources and provides common services. Operating system 1312 includes, for example, kernel 1314, services 1316, and drivers 1322. The kernel 1314 serves as an abstraction layer between hardware and other software layers. For example, kernel 1314 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Service 1316 may provide other common services for other software layers. The driver 1322 is responsible for controlling or interfacing with the underlying hardware. For example, the driver 1322 may include a display driver, an imaging device driver, Or (b)Low power consumption drive, flash memory drive, serial communication drive (e.g., universal Serial Bus (USB) drive)>Drivers, audio drivers, power management drivers, etc.

Library 1310 provides a low level public infrastructure used by applications 1306. Library 1310 may include a system library 1318 (e.g., a C-standard library), which system library 1318 provides functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, libraries 1310 may include API libraries 1324, such as media libraries (e.g., libraries for supporting presentation and manipulation of various media formats, such as moving Picture experts group-4 (MPEG 4), advanced video coding (H.264 or AVC), moving Picture experts group layer-3 (MP 3), advanced Audio Coding (AAC), adaptive Multi-Rate (AMR) audio codec, joint Picture experts group (JPEG or JPG) or Portable Network Graphics (PNG)), graphics libraries (e.g., openGL framework for presentation in two-dimensional (2D) and three-dimensional (3D) in graphical content on a display), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., webKit providing web browsing functions), and the like. Library 1310 may also include various other libraries 1328 to provide many other APIs to applications 1306.

Framework 1308 provides a high-level public infrastructure used by applications 1306. For example, framework 1308 provides various Graphical User Interface (GUI) functions, high-level resource management, and high-level location services. Framework 1308 can provide a wide variety of other APIs that can be used by applications 1306, some of which can be specific to a particular operating system or platform.

In an example implementation, the applications 1306 may include a home application 1336, a contact application 1330, a browser application 1332, a book reader application 1334, a location application 1342, a media application 1344, a messaging application 1346, a gaming application 1348, and a variety of other applications such as a third party application 1340. The application 1306 is a program that performs the functions defined in the program. One or more of the applications 1306 that are variously structured may be created using a variety of programming languages, such as an object oriented programming language (e.g., objective-C, java or C++) or a procedural programming language (e.g., C-language or assembly language). In a particular example, third party application 1340 (e.g., using ANDROID by an entity other than the vendor of the particular platform) ^TM Or IOS ^TM Applications developed in Software Development Kits (SDKs) may be, for example, in IOS ^TM 、ANDROID ^TM 、Mobile software running on the Phone or Linux OS mobile operating system or other mobile operating systems. In this example, third party application 1340 may activate API call 1350 provided by operating system 1312 to facilitate the functions described herein.

Fig. 14 is a diagrammatic representation of machine 1400 within which machine 1400 may execute instructions 1408 (e.g., software, programs, applications, applets, apps, or other executable code) for causing machine 1400 to perform any one or more of the methods discussed herein. For example, instructions 1408 may cause machine 1400 to perform any one or more of the methods described herein. Instructions 1408 transform a generic, un-programmed machine 1400 into a specific machine 1400 that is programmed to perform the functions described and illustrated in the manner described. The machine 1400 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1400 may operate in the capacity of a server machine or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1400 may include, but is not limited to: a server computer, a client computer, a Personal Computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1408 that specify actions to be taken by machine 1400, sequentially or otherwise. Furthermore, while only a single machine 1400 is illustrated, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 1408 to perform any one or more of the methodologies discussed herein.

Machine 1400 may include a processor 1402, memory 1404, and I/O components 1442 that may be configured to communicate with each other via a bus 1444. In an example embodiment, the processor 1402 (e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an ASIC, a Radio Frequency Integrated Circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 1406 and a processor 1410 that execute instructions 1408. The term "processor" is intended to include a multi-core processor, which may include two or more separate processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although fig. 14 shows multiple processors 1402, machine 1400 may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

The memory 1404 includes a main memory 1412, a static memory 1414, and a storage unit 1416, all of which are accessible by the processor 1402 via a bus 1444. The main memory 1404, static memory 1414, and storage unit 1416 store instructions 1408 embodying any one or more of the methodologies or functions described herein. The instructions 1408 may also reside, completely or partially, within the main memory 1412, within the static memory 1414, within the machine-readable medium 1418 within the storage unit 1416, within at least one of the processors 1402 (e.g., within the cache memory of the processor), or within any suitable combination thereof, during execution thereof by the machine 1400.

I/O components 1442 may include a variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, and so forth. The particular I/O components 1442 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine would be unlikely to include such a touch input device. It should be appreciated that I/O component 1442 may include many other components not shown in FIG. 14. In various example embodiments, the I/O components 1442 may include an output component 1428 and an input component 1430. The output component 1428 can include visual components (e.g., a display such as a Plasma Display Panel (PDP), a Light Emitting Diode (LED) display, a Liquid Crystal Display (LCD), a projector, or a Cathode Ray Tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, and so forth. The input components 1430 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optoelectronic keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., physical buttons, a touch screen providing location and/or force of touch or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

In other example embodiments, the I/O components 1442 may include biometric components 1432, motion components 1434, environmental components 1436, or positioning components 1438, among various other components. For example, the biometric component 1432 includes components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biological signals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. The motion component 1434 includes an acceleration sensor component (e.g., accelerometer), a gravity sensor component, a rotation sensor component (e.g., gyroscope), and the like. The environmental component 1436 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting hazardous gas concentrations or measuring contaminants in the atmosphere for safety), or other components that may provide an indication, measurement, or signal corresponding to the surrounding physical environment. The positioning component 1438 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects barometric pressure from which altitude may be derived), an orientation sensor component (e.g., a magnetometer), and the like.

Communication may be accomplished using a variety of techniques. I/O component 1442 further comprises a communication component 1440, the communication component 1440 being operable via coupling 1424 and coupling, respectivelyA connection 1426 couples the machine 1400 to the network 1420 or to a device 1422. For example, communication component 1440 may include a network interface component or another suitable device to interface with network 1420. In the other instance of the present invention, the communication means 1440 may comprise wired communication means, wireless communication means cellular communication component, near Field Communication (NFC) component,Parts (e.g.)>Low energy consumption),>means and other communication means for providing communication via other modalities. Device 1422 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

In addition, communication component 1440 may detect an identifier or include a component operable to detect an identifier. For example, the communication component 1440 may include a Radio Frequency Identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, aztec codes, data matrices, data symbols (Dataglyph), maximum codes (MaxiCode), PDF417, ultra codes (Ultra Code), UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying marked audio signals). In addition, various information may be derived via the communication component 1440, e.g., via Internet Protocol (IP) geolocated locations, via The location of signal triangulation, the location of NFC beacon signals that may indicate a particular location via detection, etc.

Various memories (e.g., memory 1404, main memory 1412, static memory 1414, and/or memory of processor 1402) and/or storage unit 1416 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1408), when executed by the processor 1402, cause various operations to implement the disclosed embodiments.

The instructions 1408 may be transmitted or received over the network 1420 via a network interface device (e.g., a network interface component included in the communication component 1440) using a transmission medium and using any of a number of well-known transmission protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, instructions 1408 may be transmitted or received using a transmission medium via coupling 1426 (e.g., a peer-to-peer coupling) with device 1422.

As used herein, the terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. These terms refer to a single or multiple storage devices and/or media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the executable instructions and/or data. Accordingly, these terms should be considered to include, but are not limited to, solid-state memory as well as optical and magnetic media, including memory internal or external to the processor. Specific examples of machine, computer, and/or device storage media include: nonvolatile memory includes, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field Programmable Gate Array (FPGA), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disk; CD-ROM and DVD-ROM discs. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, and at least some of the carrier waves, modulated data signals, and other such media are covered by the term "signal medium" as discussed below.

The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be taken to include any intangible medium that is capable of storing, encoding or carrying instructions 1416 for execution by the machine 1400, and include digital or analog communications signals or other intangible medium to facilitate communication of such software. Accordingly, the terms "transmission medium" and "signal medium" should be construed to include any form of modulated data signal, carrier wave, or the like. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Accordingly, these terms include both storage devices/media and carrier wave/modulated data signals.

Although embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments shown are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. The detailed description is, therefore, not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

These embodiments of the inventive subject matter may be referred to, individually and/or collectively, herein by the term "invention" merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

The Abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. The abstract is submitted with such understanding: i.e., the abstract is not to be used to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.

Example

Example 1 is a method for adjusting an over-rendered region, comprising: identifying an angular velocity of the display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and adjusting the size of the dynamic over-rendered region based on a combination of the angular velocity, the most recent gesture, the previous warp gesture, and the previous over-rendered region.

Example 2 includes example 1, wherein adjusting the size of the dynamic over-rendering region further comprises: obtaining an angular velocity from an Inertial Motion Unit (IMU) at a display device; detecting that the angular velocity exceeds a threshold; and in response to detecting that the angular velocity exceeds the threshold, increasing the size of the dynamic over-rendering region.

Example 3 includes example 1, wherein adjusting the size of the dynamic over-rendering region further comprises: identifying a first twisting gesture; identifying a first over-rendered region corresponding to the first warp gesture; identifying a second twisting gesture subsequent to the first twisting gesture; identifying a second over-rendered region corresponding to the first warp gesture; and forming a dynamic over-rendering region based on the first over-rendering region and the second over-rendering region.

Example 4 includes example 3, wherein the dynamic over-rendering region includes a portion of the first over-rendering region and a portion of the second over-rendering region.

Example 5 includes example 1, wherein adjusting the size of the dynamic over-rendering region further comprises: based on angular velocity, recent pose, previous warped pose, or previous over-rendered region, a heuristic algorithm is applied to estimate the size of the dynamic over-rendered region.

Example 6 includes example 1, wherein adjusting the size of the dynamic over-rendering region further comprises: based on the angular velocity, the most recent pose, the previous warped pose, and the previous over-rendered region, a machine learning algorithm is applied to estimate the size of the dynamic over-rendered region.

Example 7 includes example 1, further comprising: generating virtual content based on the rendered gesture; identifying a size of the dynamic over-rendered region; rendering the virtual content in the dynamic over-rendering region; applying a transform to the rendered image based on the most recent gesture to generate a transformed image; and displaying the transformed image in a display of the display device.

Example 8 includes example 1, further comprising: accessing first sensor data from a plurality of sensors of a display device at a first time; determining, using a tracking system of the display device, a first pose of the display device based on the first sensor data; accessing second sensor data from the plurality of sensors of the display device at a second time subsequent to the first time; and determining a second pose of the display device based on the second sensor data using a tracking system of the display device.

Example 9 includes example 1, further comprising: the size of the dynamic over-rendering region is adjusted based on a rendering rate of a 3D rendering engine of the display device.

Example 10 includes example 1, further comprising: an AR application using a display device generates virtual content in a dynamic over-rendering region.

Example 11 is a computing device, comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations comprising: identifying an angular velocity of the display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and adjusting the size of the dynamic over-rendered region based on a combination of the angular velocity, the most recent gesture, the previous warp gesture, and the previous over-rendered region.

Example 12 includes example 11, wherein adjusting the size of the dynamic over-rendering region further comprises: obtaining an angular velocity from an Inertial Motion Unit (IMU) at a display device; detecting that the angular velocity exceeds a threshold; and in response to detecting that the angular velocity exceeds the threshold, increasing the size of the dynamic over-rendering region.

Example 13 includes example 11, wherein adjusting the size of the dynamic over-rendering region further comprises: identifying a first twisting gesture; identifying a first over-rendered region corresponding to the first warp gesture; identifying a second twisting gesture subsequent to the first twisting gesture; identifying a second over-rendered region corresponding to the first warp gesture; and forming a dynamic over-rendering region based on the first over-rendering region and the second over-rendering region.

Example 14 includes example 13, wherein the dynamic over-rendering region includes a portion of the first over-rendering region and a portion of the second over-rendering region.

Example 15 includes example 11, wherein adjusting the size of the dynamic over-rendering region further comprises: based on angular velocity, recent pose, previous warped pose, or previous over-rendered region, a heuristic algorithm is applied to estimate the size of the dynamic over-rendered region.

Example 16 includes example 11, wherein adjusting the size of the dynamic over-rendering region further comprises: based on the angular velocity, the most recent pose, the previous warped pose, and the previous over-rendered region, a machine learning algorithm is applied to estimate the size of the dynamic over-rendered region.

Example 17 includes example 11, wherein the instructions further configure the apparatus to: generating virtual content based on the rendered gesture; identifying a size of the dynamic over-rendered region; rendering the virtual content in the dynamic over-rendering region; applying a transform to the rendered image based on the most recent gesture to generate a transformed image; and displaying the transformed image in a display of the display device.

Example 18 includes example 11, wherein the instructions further configure the apparatus to: accessing first sensor data from a plurality of sensors of a display device at a first time; determining, using a tracking system of the display device, a first pose of the display device based on the first sensor data; accessing second sensor data from the plurality of sensors of the display device at a second time subsequent to the first time; and determining a second pose of the display device based on the second sensor data using a tracking system of the display device.

Example 19 includes example 11, wherein the instructions further configure the apparatus to: the size of the dynamic over-rendering region is adjusted based on a rendering rate of a 3D rendering engine of the display device.

Example 20 is a non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: identifying an angular velocity of the display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and adjusting the size of the dynamic over-rendered region based on a combination of the angular velocity, the most recent gesture, the previous warp gesture, and the previous over-rendered region.

Claims (20)

1. A method for adjusting an over-rendered region, comprising:

identifying an angular velocity of a display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and

the size of the dynamic over-rendered region is adjusted based on a combination of the angular velocity, the most recent gesture, the previous warped gesture, and the previous over-rendered region.

2. The method of claim 1, wherein adjusting the size of the dynamic over-rendering region further comprises:

Retrieving the angular velocity from an Inertial Motion Unit (IMU) at the display device;

detecting that the angular velocity exceeds a threshold; and

in response to detecting that the angular velocity exceeds the threshold, the size of the dynamic over-rendering region is increased.

3. The method of claim 1, wherein adjusting the size of the dynamic over-rendering region further comprises:

identifying a first twisting gesture;

identifying a first over-rendered region corresponding to the first warp gesture;

identifying a second twisting gesture subsequent to the first twisting gesture;

identifying a second over-rendered region corresponding to the first warp gesture; and

the dynamic over-rendering region is formed based on the first over-rendering region and the second over-rendering region.

4. The method of claim 3, wherein the dynamic over-rendering region comprises a portion of the first over-rendering region and a portion of the second over-rendering region.

5. The method of claim 1, wherein adjusting the size of the dynamic over-rendering region further comprises:

a heuristic algorithm is applied to estimate the size of the dynamic over-rendered region based on the angular velocity, the most recent pose, the previous warped pose, or the previous over-rendered region.

6. The method of claim 1, wherein adjusting the size of the dynamic over-rendering region further comprises:

a machine learning algorithm is applied to estimate a size of the dynamic over-rendered region based on the angular velocity, the most recent pose, the previous warped pose, and the previous over-rendered region.

7. The method of claim 1, further comprising:

generating virtual content based on the rendered gesture;

identifying a size of the dynamic over-rendered region;

rendering the virtual content in the dynamic over-rendering region;

applying a transformation to the rendered image based on the recent gesture to generate a transformed image; and

and displaying the converted image in a display of the display device.

8. The method of claim 1, further comprising:

accessing first sensor data from a plurality of sensors of the display device at a first time;

determining a first pose of the display device based on the first sensor data using a tracking system of the display device;

accessing second sensor data from a plurality of sensors of the display device at a second time subsequent to the first time; and

A second pose of the display device is determined based on the second sensor data using a tracking system of the display device.

9. The method of claim 1, further comprising:

the size of the dynamic over-rendering region is adjusted based on a rendering rate of a 3D rendering engine of the display device.

10. The method of claim 1, further comprising:

virtual content is generated in the dynamic over-rendering region using an AR application of the display device.

11. A computing device, comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations comprising:

identifying an angular velocity of a display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and

the size of the dynamic over-rendered region is adjusted based on a combination of the angular velocity, the most recent gesture, the previous warped gesture, and the previous over-rendered region.

12. The computing device of claim 11, wherein to adjust the size of the dynamic over-rendering region further comprises to:

retrieving the angular velocity from an Inertial Motion Unit (IMU) at the display device;

Detecting that the angular velocity exceeds a threshold; and

in response to detecting that the angular velocity exceeds the threshold, the size of the dynamic over-rendering region is increased.

13. The computing device of claim 11, wherein to adjust the size of the dynamic over-rendering region further comprises to:

identifying a first twisting gesture;

identifying a first over-rendered region corresponding to the first warp gesture;

identifying a second twisting gesture subsequent to the first twisting gesture;

identifying a second over-rendered region corresponding to the first warp gesture; and

the dynamic over-rendering region is formed based on the first over-rendering region and the second over-rendering region.

14. The computing device of claim 13, wherein the dynamic over-rendering region comprises a portion of the first over-rendering region and a portion of the second over-rendering region.

15. The computing device of claim 11, wherein to adjust the size of the dynamic over-rendering region further comprises to:

a heuristic algorithm is applied to estimate the size of the dynamic over-rendered region based on the angular velocity, the most recent pose, the previous warped pose, or the previous over-rendered region.

16. The computing device of claim 11, wherein to adjust the size of the dynamic over-rendering region further comprises to:

a machine learning algorithm is applied to estimate a size of the dynamic over-rendered region based on the angular velocity, the most recent pose, the previous warped pose, and the previous over-rendered region.

17. The computing device of claim 11, wherein the instructions further configure the device to:

generating virtual content based on the rendered gesture;

identifying a size of the dynamic over-rendered region;

rendering the virtual content in the dynamic over-rendering region;

applying a transformation to the rendered image based on the recent gesture to generate a transformed image; and

and displaying the converted image in a display of the display device.

18. The computing device of claim 11, wherein the instructions further configure the device to:

accessing first sensor data from a plurality of sensors of the display device at a first time;

determining a first pose of the display device based on the first sensor data using a tracking system of the display device;

Accessing second sensor data from a plurality of sensors of the display device at a second time subsequent to the first time; and

a second pose of the display device is determined based on the second sensor data using a tracking system of the display device.

19. The computing device of claim 11, wherein the instructions further configure the device to:

the size of the dynamic over-rendering region is adjusted based on a rendering rate of a 3D rendering engine of the display device.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising:

identifying an angular velocity of a display device, a most recent pose of the display device, a previous warped pose, and a previous over-rendered region; and

the size of the dynamic over-rendered region is adjusted based on a combination of the angular velocity, the most recent gesture, the previous warped gesture, and the previous over-rendered region.