US10957112B2 - Cross reality system - Google Patents

Cross reality system Download PDF

Info

Publication number
US10957112B2
US10957112B2 US16/538,759 US201916538759A US10957112B2 US 10957112 B2 US10957112 B2 US 10957112B2 US 201916538759 A US201916538759 A US 201916538759A US 10957112 B2 US10957112 B2 US 10957112B2
Authority
US
United States
Prior art keywords
map
coordinate frame
frame
head
world
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/538,759
Other languages
English (en)
Other versions
US20200090407A1 (en
Inventor
Jeremy Dwayne Miranda
Rafael Domingos Torres
Daniel Olshansky
Anush Mohan
Robert Blake Taylor
Samuel A. Miller
Jehangir Tajik
Ashwin Swaminathan
Lomesh Agarwal
Ali Shahrokni
Prateek Singhal
Joel David Holder
Xuan Zhao
Siddharth Choudhary
Helder Toshiro Suzuki
Hiral Honar Barot
Eran Guendelman
Michael Harold LIEBENOW
Christian Ivan Robert Moore
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Magic Leap Inc
Original Assignee
Magic Leap Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Magic Leap Inc filed Critical Magic Leap Inc
Priority to US16/538,759 priority Critical patent/US10957112B2/en
Priority to US16/593,745 priority patent/US11227435B2/en
Publication of US20200090407A1 publication Critical patent/US20200090407A1/en
Assigned to CITIBANK, N.A., AS COLLATERAL AGENT reassignment CITIBANK, N.A., AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MAGIC LEAP, INC., MENTOR ACQUISITION ONE, LLC, MOLECULAR IMPRINTS, INC.
Assigned to MAGIC LEAP, INC. reassignment MAGIC LEAP, INC. PROPRIETARY INFORMATION AND INVENTIONS AGREEMENT Assignors: SHAHROKNI, ALI
Assigned to MAGIC LEAP, INC. reassignment MAGIC LEAP, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIEBENOW, Michael Harold, GUENDELMAN, ERAN, SINGHAL, Prateek, SUZUKI, HELDER TOSHIRO, MILLER, SAMUEL A., TAYLOR, Robert Blake, MOORE, CHRISTIAN IVAN ROBERT, MOHAN, Anush, OLSHANSKY, Daniel, Holder, Joel David, Miranda, Jeremy Dwayne, TAJIK, JEHANGIR, AGARWAL, Lomesh, Barot, Hiral Honar, SWAMINATHAN, ASHWIN, Choudhary, Siddharth, Torres, Rafael Domingos, ZHAO, XUAN
Priority to US17/208,844 priority patent/US11386629B2/en
Application granted granted Critical
Publication of US10957112B2 publication Critical patent/US10957112B2/en
Priority to US17/542,306 priority patent/US11978159B2/en
Assigned to CITIBANK, N.A., AS COLLATERAL AGENT reassignment CITIBANK, N.A., AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MAGIC LEAP, INC., MENTOR ACQUISITION ONE, LLC, MOLECULAR IMPRINTS, INC.
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06K9/00671—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
    • G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems

Definitions

  • This application relates generally to a cross reality system.
  • Computers may control human user interfaces to create an X Reality (XR or cross reality) environment in which some or all of the XR environment, as perceived by the user, is generated by the computer.
  • XR environments may be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments, in which some or all of an XR environment may be generated by computers using, in part, data that describes the environment.
  • This data may describe, for example, virtual objects that may be rendered in a way that users' sense or perceive as a part of a physical world and can interact with the virtual objects.
  • the user may experience these virtual objects as a result of the data being rendered and presented through a user interface device, such as, for example, a head-mounted display device.
  • the data may be displayed to the user to see, or may control audio that is played for the user to hear, or may control a tactile (or haptic) interface, enabling the user to experience touch sensations that the user senses or perceives as feeling the virtual object.
  • XR systems may be useful for many applications, spanning the fields of scientific visualization, medical training, engineering design and prototyping, tele-manipulation and tele-presence, and personal entertainment.
  • AR and MR in contrast to VR, include one or more virtual objects in relation to real objects of the physical world.
  • the experience of virtual objects interacting with real objects greatly enhances the user's enjoyment in using the XR system, and also opens the door for a variety of applications that present realistic and readily understandable information about how the physical world might be altered.
  • An XR system may represent the physical world around a user of the system as a “mesh.”
  • a mesh may be represented by multiple, interconnected triangles. Each triangle has edges joining points on a surface of an object within the physical world, such that each triangle represents a portion of the surface. Information about the portion of the surface, such as color, texture or other properties may be stored in associate within the triangle.
  • an XR system may process image information to detect points that and surfaces so as to create or update the mesh.
  • aspects of the present application relate to methods and apparatus for providing X reality (cross reality or XR) scenes. Techniques as described herein may be used together, separately, or in any suitable combination.
  • Some embodiments relate to an XR system including a first XR device that includes a first processor, a first computer-readable medium connected to the first processor, a first origin coordinate frame stored on the first computer-readable medium, a first destination coordinate frame stored on the computer-readable medium, a first data channel to receive data representing local content, a first coordinate frame transformer executable by the first processor to transform a positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, and a first display system adapted to display the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.
  • Some embodiments relate to a viewing method including storing a first origin coordinate frame, storing a first destination coordinate frame, receiving data representing local content, transforming a positioning of local content from the first origin coordinate frame to the first destination coordinate frame, and displaying the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.
  • Some embodiments relate to an XR system including a map storing routine to store a first map, being a canonical map, having a plurality of anchors, each anchor of the first map having a set of coordinates, a real object detection device positioned to detect locations of real objects, an anchor identification system connected to the real object detection device to detect, based on the locations of the real objects, anchors of a second map, each anchor of the second map having a set of coordinates and a localization module connected to the canonical map and the second map and executable to localize the second map to the canonical map by matching a first anchor of the second map to a first anchor of the canonical map and matching a second anchor of the second map to a second anchor of the canonical map.
  • Some embodiments relate to a viewing method including storing a first map, being a canonical map, having a plurality of anchors, each anchor of the canonical map having a set of coordinates, detecting locations of real objects, detecting, based on the locations of the real objects, anchors of a second map, each anchor of the second map having a set of coordinates and localizing the second map to the canonical map by matching a first anchor of the second map to a first anchor of the first map and matching a second anchor of the second map to a second anchor of the canonical map.
  • Some embodiments relate to an XR system including a server that may have a processor, a computer-readable medium connected to the processor, a plurality of canonical maps on the computer-readable medium, a respective canonical map identifier on the computer-readable medium associated with each respective canonical map, the canonical map identifiers differing from one another to uniquely identify the canonical maps, a position detector on the computer-readable medium and executable by the processor to receive and store a position identifier from an XR device, a first filter on the computer-readable medium and executable by the processor to compare the position identifier with the canonical map identifiers to determine one or more canonical maps that form a first filtered selection, and a map transmitter on the computer-readable medium and executable by the processor to transmit one or more of the canonical maps to the XR device based on the first filtered selection.
  • Some embodiments relate to a viewing method including storing a plurality of canonical maps on a computer-readable medium, each canonical map having a respective canonical map identifier associated with the respective canonical map, the canonical map identifiers differing from one another to uniquely identify the canonical maps, receiving and storing, with a processor connected to the computer-readable medium, a position identifier from an XR device, comparing, with the processor, the position identifier with the canonical map identifiers to determine one or more canonical maps that form a first filtered selection, and transmitting, with the processor, a plurality of the canonical maps to the XR device based on the first filtered selection.
  • Some embodiments relate to an XR system including a processor, a computer readable medium connected to the processor, a multilayer perception unit, on the computer readable medium and, executable by the processor, that receives a plurality of features of an image and converts each feature to a respective string of numbers, and a max pool unit, on the computer-readable medium and executable by the processor, that combines a maximum value of each string of numbers into a global feature string representing the image.
  • Some embodiments relate to a viewing method, including receiving, with a processor, a plurality of features of an image, converting, with the processor, each feature to a respective string of numbers, and combining, with the processor, a maximum value of each string of numbers into a global feature string representing the image.
  • Some embodiments relate to a method of operating a computing system to identify one or more environment maps stored in a database to merge with a tracking map computed based on sensor data collected by a device worn by a user, wherein the device received signals of access points to computer networks while computing the tracking map, the method including determining at least one area attribute of the tracking map based on characteristics of communications with the access points, determining a geographic location of the tracking map based on the at least one area attribute, identifying a set of environment maps stored in the database corresponding to the determined geographic location, filtering the set of environment maps based on similarity of one or more identifiers of network access points associated with the tracking map and the environment maps of the set of environment maps, filtering the set of environment maps based on similarity of metrics representing contents of the tracking map and the environment maps of the set of environment maps, and filtering the set of environment maps based on degree of match between a portion of the tracking map and portions of the environment maps of the set of environment maps.
  • Some embodiments relate to a cloud computing environment for an augmented reality system configured for communication with a plurality of user devices comprising sensors, including a user database storing area identities indicating areas that the plurality of user devices were used in, the area identities comprising parameters of wireless networks detected by the user devices when in use, a map database storing a plurality of environment maps constructed from data supplied by the plurality of user devices and associated metadata, the associated metadata comprising area identities derived from area identities of the plurality of user devices that supplied data from which the maps were constructed, the area identities comprising parameters of wireless networks detected by the user devices that supplied data from which the maps were constructed, non-transitory computer storage media storing computer-executable instructions that, when executed by at least one processor in the cloud computing environment, receives messages from the plurality of user devices comprising parameters of wireless networks detected by the user devices, computes area identifiers for the user devices and updates the user database based on the received parameters and/or the computed area identifiers, and receives requests for environment maps from the plurality of user devices
  • Some embodiments relate to an XR system including a real object detection device to detect a plurality of surfaces of real-world objects, an anchor identification system connected to the real object detection device to generate a map based on the real-world objects, a persistent coordinate frame (PCF) generation system to generate a first PCF based on the map and associate the first PCF with the map, first and second storage mediums on first and second XR devices, respectively, and at least first and second processors of the first and second XR devices, to store the first PCF in first and second storage mediums of the first and second XR devices respectively.
  • a real object detection device to detect a plurality of surfaces of real-world objects
  • an anchor identification system connected to the real object detection device to generate a map based on the real-world objects
  • PCF persistent coordinate frame
  • Some embodiments relate to a viewing method including detecting, with at least one detection device a plurality of surfaces of real-world objects, generating, with at least one processor, a map based on the real-world objects, generating, with at least one processor, a first PCF based on the map, associating, with the at least one processor, the first PCF with the map, and storing, with at least first and second processors of first and second XR devices, the first PCF in first and second storage mediums of the first and second XR devices respectively.
  • Some embodiments relate to an XR system including a first XR device that may include a first XR device processor, a first XR device storage device connected to the first XR device processor, a set of instructions on the first XR device processor, including a download system, executable by the first XR device processor, to download persistent poses from a server, a PCF retriever, executable by the first XR device processor, to retrieve PCF's from the first storage device of the first XR device based on the persistent poses, and a coordinate frame calculator, executable by the first XR device processor, to calculate a coordinate frame based on the PCF's retrieved from the first storage device.
  • Some embodiments relate to a viewing method including downloading, with a first XR device processor of a first XR device, persistent poses from a server, retrieving, with the first XR device processor, PCF's from the first storage device of the first XR device based on the persistent poses, and calculating, with the first XR device processor, a coordinate frame based on the PCF's retrieved from the first storage device.
  • Some embodiments relate to a viewing method including a server that may include a server processor, a server storage device connected to the server processor, a map storing routine storing, executable with a server processor of the server, to store the first PCF in association with a map on the server storage device of the server, and a map transmitter, with the server processor, executable with a server processor, to transmit the map and the first PCF to a first XR device.
  • the invention also provides a viewing method including storing, with a server processor of the server, the first PCF in association with a map on a server storage device of the server, and transmitting, with the server processor of the server, the map and the first PCF to a first XR device.
  • Some embodiments relate to a viewing method including entering, by a processor of a XR device, tracking of head pose by capturing surfaces of an environment with a capture device on a head-mounted frame secured to a head of a user and determining an orientation of the head-mounted frame relative to the surfaces, determining, by the processor, whether head pose is lost due to an inability to determine the orientation of the head-mounted frame relative to the surfaces, and if head pose is lost, then, by the processor, entering pose recovery mode to establish the head pose by determining an orientation of the head-mounted frame relative to the surfaces.
  • FIG. 1 is a schematic diagram illustrating a cross reality (XR) system, according to some embodiments
  • FIG. 2 is a block diagram of a first XR device of the XR system of FIG. 1 , according to some embodiments;
  • FIG. 3 is a schematic diagram illustrating the transformation of origin coordinate frames into destination coordinate frames in order to, for example, correctly render local XR content, according to some embodiments;
  • FIG. 4 is a top plan view illustrating pupil-based coordinate frames, according to some embodiments.
  • FIG. 5 is a top plan view illustrating a camera coordinate frame that includes all pupil positions, according to some embodiments.
  • FIG. 6 is a block diagram of the XR system of FIG. 1 , including an additional second XR device, according to some embodiments;
  • FIG. 7 is a schematic diagram of the display system of FIG. 2 , according to some embodiments.
  • FIG. 8 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1 ), which may be generated by the first XR device of FIG. 2 , according to some embodiments;
  • FIG. 9 is a block diagram illustrating uploading Map 1 from the first XR device to the server of FIG. 6 , according to some embodiments.
  • FIG. 10 is a schematic diagram illustrating the XR system of FIG. 6 , showing the second user has initiated a second session using a second XR device of the XR system after the first user has terminated a first session, according to some embodiments;
  • FIG. 11A is a block diagram illustrating a new session for the second XR device of FIG. 10 , according to some embodiments.
  • FIG. 11B is a block diagram illustrating the creation of a tracking map for the second XR device of FIG. 10 , according to some embodiments;
  • FIG. 11C is a block diagram illustrating downloading a canonical map from the server to the second XR device of FIG. 10 , according to some embodiments;
  • FIG. 11D is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and XR content attached to the PCF, according to some embodiments;
  • PCF persistent coordinate frame
  • FIG. 12 is a schematic diagram illustrating a localization attempt to localize to a canonical map a second tracking map (Map 2 ), which may be generated by the second XR device of FIG. 10 , according to some embodiments;
  • FIG. 13 is a schematic diagram illustrating a localization attempt to localize to a canonical map the second tracking map (Map 2 ) of FIG. 12 , which may be further developed and with XR content associated with anchors of Map 2 , according to some embodiments;
  • FIGS. 14A-14B are a schematic diagram illustrating a successful localization of Map 2 of FIG. 13 to the canonical map, according to some embodiments;
  • FIG. 15 is a schematic diagram illustrating a canonical map generated by including an anchor or anchors from the canonical map of FIG. 14A into Map 2 of FIG. 13 , according to some embodiments;
  • FIG. 16 is a schematic diagram illustrating the canonical map of FIG. 15 with further expansion of Map 2 on the second XR device, according to some embodiments;
  • FIG. 17 is a block diagram illustrating uploading Map 2 from the second XR device to the server, according to some embodiments.
  • FIG. 18 is a block diagram illustrating merging Map 2 with the canonical map, according to some embodiments.
  • FIG. 19 is a block diagram illustrating transmission of a new canonical map from the server to the first and second XR devices, according to some embodiments.
  • FIG. 20 is block diagram illustrating a two-dimensional representation of Map 2 and a head coordinate frame of the second XR device that is referenced to Map 2 , according to some embodiments;
  • FIG. 21 is a block diagram illustrating, in two-dimensions, adjustment of the head coordinate frame which can occur in six degrees of freedom, according to some embodiments;
  • FIG. 22 a block diagram illustrating a canonical map on the second XR device wherein sound is localized relative to anchors of Map 2 , according to some embodiments;
  • FIGS. 23 and 24 are a perspective view and a block diagram illustrating use of the XR system when the first user has terminated a first session and the first user has initiated a second session using the XR system, according to some embodiments;
  • FIGS. 25 and 26 are a perspective view and a block diagram illustrating use of the XR system when three users are simultaneously using the XR system in the same session, according to some embodiments;
  • FIG. 27 is a block diagram illustrating a two-dimensional view representing latitude and longitude of canonical maps stored on the server, according to some embodiments.
  • FIG. 28 is a schematic diagram illustrating a series of filters for selecting canonical maps for transmission by the server to a XR device, according to some embodiments;
  • FIGS. 29A-29C are block diagrams illustrating a system for generating global feature strings of images and/or maps, according to some embodiments.
  • FIG. 29D is a block diagram illustrating a method of training a neural network, according to some embodiments.
  • FIG. 30 is a sketch illustrating an example of a simplified augmented reality (AR) scene, according to some embodiments.
  • AR augmented reality
  • FIG. 31 is a sketch of an exemplary simplified AR scene, showing exemplary World Reconstruction use cases including visual occlusion, physics-based interactions, and environment reasoning, according to some embodiments;
  • FIG. 32 is a schematic diagram illustrating data flow in an AR system configured to provide an experience of AR contents interacting with a physical world, according to some embodiments;
  • FIG. 33 is a schematic diagram illustrating an example of an AR display system, according to some embodiments.
  • FIG. 34A is a schematic diagram illustrating a user wearing an AR display system rendering AR content as the user moves through a physical world environment, according to some embodiments;
  • FIG. 34B is a schematic diagram illustrating a viewing optics assembly and attendant components, according to some embodiments.
  • FIG. 35A is a schematic diagram illustrating an AR system using a world reconstruction system, according to some embodiments.
  • FIG. 35B is a schematic diagram illustrating a passable world, according to some embodiments.
  • FIG. 36 is an exemplary environment map, according to some embodiments.
  • FIG. 37 is a schematic diagram illustrating an AR system configured to rank and merge a plurality of environment maps, according to some embodiments.
  • FIG. 38 is flow chart illustrating a method of selecting a plurality of ranked environment maps, according to some embodiments.
  • FIG. 39 is a schematic diagram illustrating an exemplary map rank portion of the AR system of FIG. 37 , according to some embodiments.
  • FIG. 40A is a schematic diagram illustrating an example of area attributes of a tracking map (TM) and environment maps in a database, according to some embodiments;
  • FIG. 40B is a schematic diagram illustrating an example of determining a geographic location of a tracking map (TM) for geolocation filtering of FIG. 38 , according to some embodiments;
  • FIG. 41 is a schematic diagram illustrating an example of geolocation filtering of FIG. 38 , according to some embodiments.
  • FIG. 42 is a schematic diagram illustrating an example of Wi-Fi BSSID filtering of FIG. 38 , according to some embodiments.
  • FIG. 43 is a schematic diagram illustrating an example of localization of FIG. 38 , according to some embodiments.
  • FIG. 44 is a schematic diagram illustrating a room and key frames that are established for various areas in the room, according to some embodiments.
  • FIG. 45 is a schematic diagram illustrating the establishment of persistent poses based on the key frames, according to some embodiments.
  • FIG. 46 is a schematic diagram illustrating the establishment of a persistent coordinate frame (PCF) based on the persistent poses, according to some embodiments.
  • PCF persistent coordinate frame
  • FIG. 47 is a flow chart illustrating a method of establishing and using a PCF, according to some embodiments.
  • FIGS. 48A to 48I are schematic diagrams illustrating an example of establishing and using a PCF of FIG. 47 ;
  • FIG. 49 is a flow chart illustrating a method of recovering and resetting a head pose, according to some embodiments.
  • FIG. 50 is a block diagram of a machine in the form of a computer that can find application in the present invention system, according to some embodiments.
  • FIGS. 51-53 are block diagrams illustrating examples of determining a map to be used based at least partly on deep keyframe (DKF) analysis, according to some embodiments.
  • DKF deep keyframe
  • Described herein are methods and apparatus for providing X reality (XR or cross reality) scenes.
  • XR X reality
  • an XR system must know the users' physical surroundings in order to correctly correlate locations of virtual objects in relation to real objects.
  • An XR system may build an environment map of a scene, which may be created from image and depth information collected with sensors that are part of XR devices worn by users of the XR system.
  • the inventors have realized and appreciated techniques for operating XR systems to provide XR scenes for a more immersive user experience by enabling persistent environment maps to be created, stored and retrieved by any of multiple users of an XR system.
  • Such persistent environment maps may enable one or more functions that enhance an XR experience.
  • the wearable device worn by a user after being turned on, may retrieve from persistent storage, such as from cloud storage, an appropriate environment map that was previously created and stored. That previously stored map may have been based on data about the environment collected with sensors on the user's wearable device a prior time or times that the user was present.
  • Retrieving a persistent map may enable use of the wearable device without a scan of the physical world with the sensors on the wearable device.
  • the user upon entering a new region of the physical world may similarly retrieve an appropriate environment map.
  • the persistent environment map accessed by one user may have been created and stored by another user and/or may have been constructed by aggregating data about the physical world collected by sensors on wearable devices worn by multiple users who were previously present in at least a portion of the physical world represented by the persistent environment map.
  • sharing data about the physical world among multiple users may enable shared user experiences of virtual content.
  • Two XR devices that have access to the same persistent environment map may both localize with respect to the persistent environment map.
  • a user device may render virtual content that has a location specified by reference to the persistent environment map by translating that location to a frame or reference maintained by the user device. The user device may use this local frame of reference to control the display of the user device to render the virtual content in the specified location.
  • the XR system may include components that, based on data about the physical world collected with sensors on user devices, develop, maintain, and use one or more persistent environment maps. These components may be distributed across the XR system, with some operating, for example, on a head mounted portion of a user device. Other components may operate on a computer, associated with the user coupled to the head mounted portion over a local or personal area network. Yet others may operate at a remote location, such as at one or more servers accessible over a wide area network.
  • components may include components that can identify from information about the physical world collected by one or more user devices information that is of sufficient quality to be stored as a persistent environment map.
  • An example of such a component, described in greater detail below is PCF integration unit 1300 in FIG. 6 .
  • these components may include components that may aid in determining an appropriate persistent environment map that may be retrieved and used by a user device.
  • An example of such a component, described in greater detail below is a map rank component.
  • Such a component may receive inputs from a user device and identify one or more persistent environment maps that are likely to represent the region of the physical world in which that device is operating.
  • a map rank component for example, may aid in selecting a persistent environment map to be used by that local device as it renders virtual content, gathers data about the environment, or performs other actions.
  • a map rank component alternatively or additionally, may aid in identifying persistent environment maps to be updated as additional information about the physical world is collected by one or more user devices.
  • API's application programming interfaces
  • a platform that builds and maintains persistent environment maps may provide one or more API's to enable separately created applications to obtain information about the physical world.
  • Yet other components may be transformations that transform information captured or described in relation to one reference frame to be translated into another reference frame.
  • sensors may be attached to a head mounted display such that the data read from that sensor indicates locations of objects in the physical world with respect to the head pose of the wearer.
  • One or more transformations may be applied to relate that location information to the coordinate frame associated with a persistent environment map.
  • data indicating where a virtual object is to be rendered when expressed in a coordinate frame of a persistent environment map may be put through one or more transformations to be in a frame of reference of the display on the user's head.
  • there may be multiple such transformations.
  • These transformations may be partitioned across the components of an XR system such that they may be efficiently updated and or applied in a distributed system.
  • map information may be represented in a way that may be readily shared among users and among the distributed components, including applications.
  • Information about the physical world may be represented as a coordinate frame with one or more anchors, representing features recognized in the physical world.
  • Such maps may be sparse, providing less than all of the available information about the physical world, such that they may be efficiently processed and transferred.
  • the persistent environment maps may be constructed from information collected by multiple user devices.
  • the XR devices may construct separate tracking maps (TM) with information collected by sensors of the XR devices worn by the user of the system at various locations and times.
  • Each TM may include points, each of which may be associated with a real object in the environment.
  • the TMs may be used to track users' motions in a scene, enabling an XR system to estimate respective users' head poses based on a TM.
  • the inventors have realized and appreciated techniques for operating XR systems to provide XR scenes for a more immersive user experience such as estimating head poses at a frequency of 1 kHz, with low usage of computational resources such as four video graphic array (VGA) cameras operating at 30 Hz, one inertial measurement unit (IMU) operating at 1 kHz, compute power of a single advanced RISC machine (ARM) core, memory less than 1 GB, and network bandwidth less than 100 Mbp.
  • VGA video graphic array
  • IMU inertial measurement unit
  • ARM advanced RISC machine
  • These techniques relate to reducing processing required to generate and maintain environment maps and estimate head poses as well as to providing and consuming data with low computational overhead.
  • These techniques may include hybrid tracking such that an XR system can leverage both (1) patch-based tracking of distinguishable points between successive images (frame-to-frame tracking) of the environment, and (2) matching of points of interest of a current image with a descriptor-based map of known real-world locations of corresponding points of interest (map-to-frame tracking).
  • frame-to-frame tracking the XR system can track particular points of interest (salient points), such as corners, between captured images of the real-world environment.
  • the display system can identify locations of visual points of interest in a current image, which were included in (e.g., located in) a previous image. This identification may be accomplished using, e.g., photometric error minimization processes.
  • the XR system can access map information indicating real-world locations of points of interest (e.g., three-dimensional coordinates), and match points of interest included in a current image to the points of interest indicated in the map information. Information regarding the points of interest is stored as descriptors in the map database. The XR system can then calculate its pose based on the matched visual features.
  • map information indicating real-world locations of points of interest (e.g., three-dimensional coordinates)
  • Information regarding the points of interest is stored as descriptors in the map database.
  • the XR system can then calculate its pose based on the matched visual features.
  • These techniques may include reducing the amount of data that is processed when constructing maps, such as by constructing sparse maps with a collection of mapped points and keyframes and/or dividing the maps into blocks to enable updates by blocks.
  • a mapped point may be associated with a three-dimensional (3D) world reconstruction of a real object in the environment.
  • a keyframe may include selected information from camera-captured data.
  • These techniques may include creating dynamic maps based on one or more coordinate systems in real space across one or more sessions, generating persistent coordinate frames (PCF) over the sparse maps, which may be exposed to XR applications via, for example, an application programming interface (API), ranking and mapping multiple maps created by one or more XR devices, using deep key frame analysis in some embodiments, and recovering and resetting head poses.
  • PCF persistent coordinate frames
  • API application programming interface
  • Techniques as described herein may be used together or separately with many types of devices and for many types of scenes, including wearable or portable devices with limited computational resources that provide an augmented reality scene.
  • Described herein are methods and apparatus for providing virtual contents using an XR system, independent of locations of eyes viewing the virtual content.
  • a virtual content is re-rendered upon any motion of the displaying system. For example, if a user wearing a display system views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, the 3D object should be re-rendered for each viewpoint such that the user has the perception that he or she is walking around an object that occupies real space.
  • the re-rendering consumes significant computational resources of a system and causes artifacts due to latency.
  • head pose e.g., the location and orientation of a user wearing an XR system
  • dynamic maps of a scene may be generated based on multiple coordinate frames in real space across one or more sessions such that virtual contents interacting with the dynamic maps may be rendered robustly, independent of eye rotations within the head of the user and/or independent of sensor deformations caused by, for example, heat generated during high-speed, computation-intensive operation.
  • the configuration of multiple coordinate frames may enable a first XR device worn by a first user and a second XR device worn by a second user to recognize a common location in a scene.
  • the configuration of multiple coordinate frames may enables the first and second users wearing the first and second XR devices to view a virtual content in a same location of a scene.
  • a tracking map may be built in a world coordinate frame, which may have a world origin.
  • the world origin may be the first pose of an XR device when the XR device is powered on.
  • the world origin may be aligned to gravity such that a developer of an XR application can get gravity alignment without extra work.
  • Different tracking maps may be built in different world coordinate frames because the tracking maps may be captured by a same XR device at different sessions and/or different XR devices worn by different users.
  • a session of an XR device may start from powering on to powering off the device.
  • an XR device may have a head coordinate frame, which may have a head origin.
  • the head origin may be the current pose of an XR device when an image is taken. The difference between head pose of a world coordinate frame and of a head coordinate frame may be used to estimate a tracking route.
  • an XR device may have a camera coordinate frame, which may have a camera origin.
  • the camera origin may be the current pose of one or more sensors of an XR device.
  • an XR device may have a head unit with a head-mountable frame that a user can secure to their head and may include two waveguides, one in front of each eye of the user.
  • the waveguides may be transparent so that ambient light from real-world objects can transmit through the waveguides and the user can see the real-world objects.
  • Each waveguide may transmit projected light from a projector to a respective eye of the user.
  • the projected light may form an image on the retina of the eye.
  • the retina of the eye thus receives the ambient light and the projected light.
  • the user may simultaneously see real-world objects and one or more virtual objects that are created by the projected light.
  • XR devices may have sensors that detect real-world objects around a user. These sensors may, for example, be cameras that capture images that may be processed to identify the locations of real-world objects.
  • an XR system may assign a local coordinate frame to a virtual content, as opposed to attaching the virtual content in a world coordinate frame. Such configuration enables a virtual content to be attached to a more persistent frame position such as a persistent coordinate frame (PCF).
  • a local coordinate frame when a head pose session ends, a local coordinate frame may be converted to a world coordinate frame.
  • a world coordinate frame may be the origin of a previous session's map that has been localized relative to the current local coordinate frame.
  • the XR device may detect the changes in the environment map and determine movement of the head unit worn by the user relative to real-world objects.
  • FIG. 1 illustrates an XR system 10 , according to some embodiments.
  • the XR system may include a first XR device 12 . 1 that is worn by a first user 14 . 1 , a real object in the form of a table 16 , a network 18 and a server 20 .
  • the first XR device 12 . 1 includes a head unit 22 , a belt pack 24 and a cable connection 26 .
  • the first user 14 . 1 secures the head unit 22 to their head and the belt pack 24 remotely from the head unit 22 on their waist.
  • the cable connection 26 connects the head unit 22 to the belt pack 24 .
  • the head unit 22 includes technologies that are used to display a virtual object or objects to the first user 14 . 1 while the first user 14 . 1 is permitted to see real objects such as the table 16 .
  • the belt pack 24 includes primarily processing and communications capabilities of the first XR device 12 . 1 . In some embodiments, the processing and communication capabilities may reside entirely or partially in the head unit 22 such that the belt pack 24 may be removed or may be located in another device such as a backpack.
  • the belt pack 24 is connected via a wireless connection to the network 18 .
  • the server 20 is connected to the network 18 and holds data representative of local content.
  • the belt pack 24 downloads the data representing the local content from the server 20 via the network 18 .
  • the belt pack 24 provides the data via the cable connection 26 to the head unit 22 .
  • the head unit 22 may include a display that has a light source, for example, a laser light source or a light emitting diode (LED) light source, and a waveguide that guides the light.
  • a light source for example, a laser light source or a light emitting diode (LED) light source
  • the first user 14 . 1 may mount the head unit 22 to their head and the belt pack 24 to their waist.
  • the belt pack 24 may download image data over the network 18 from the server 20 .
  • the first user 14 . 1 may see the table 16 through a display of the head unit 22 .
  • a projector forming part of the head unit 22 may receive the image data from the belt pack 24 and generate light based on the image data.
  • the light may travel through one or more of the waveguides forming part of the display of the head unit 22 .
  • the light may then leave the waveguide and propagates onto a retina of an eye of the first user 14 . 1 .
  • the projector may generate the light in a pattern that is replicated on a retina of the eye of the first user 14 . 1 .
  • the light that falls on the retina of the eye of the first user 14 . 1 may have a selected field of depth so that the first user 14 . 1 perceives an image at a preselected depth behind the waveguide.
  • both eyes of the first user 14 . 1 may receive slightly different images so that a brain of the first user 14 . 1 perceives a three-dimensional image or images at selected distances from the head unit 22 .
  • the first user 14 . 1 perceives a virtual content 28 above the table 16 .
  • the proportions of the virtual content 28 and its location and distance from the first user 14 . 1 are determined by the data representing the virtual content 28 and various coordinate frames that are used to display the virtual content 28 to the first user 14 . 1 .
  • the virtual content 28 is not visible from the perspective of the drawing and is visible to the first user 14 . 1 through using the first XR device 12 . 1 .
  • the virtual content 28 may initially resides as data structures within vision data and algorithms in the belt pack 24 .
  • the data structures may then manifest themselves as light when the projectors of the head unit 22 generate light based on the data structures.
  • the virtual content 28 has no existence in three-dimensional space in front of the first user 14 . 1
  • the virtual content 28 is still represented in FIG. 1 in three-dimensional space for illustration purpose.
  • the visualization of computer data in three-dimensional space may be used in this description to illustrate how the data structures that facilitate the renderings are perceived by one or more users relate to one another within the data structures in the belt pack 24 .
  • FIG. 2 illustrates components of the first XR device 12 . 1 , according to some embodiments.
  • the first XR device 12 . 1 may include the head unit 22 , and various components forming part of the vision data and algorithms including, for example, a rendering engine 30 , various coordinate systems 32 , various origin and destination coordinate frames 34 , and various origin to destination coordinate frame transformers 36 .
  • the head unit 22 may include a head-mountable frame 40 , a display system 42 , a real object detection camera 44 , a movement tracking camera 46 , and an inertial measurement unit 48 .
  • the head-mountable frame 40 may have a shape that is securable to the head of the first user 14 . 1 in FIG. 1 .
  • the display system 42 , real object detection camera 44 , movement tracking camera 46 , and inertial measurement unit 48 may be mounted to the head-mountable frame 40 and therefore move together with the head-mountable frame 40 .
  • the coordinate systems 32 may include a local data system 52 , a world frame system 54 , a head frame system 56 , and a camera frame system 58 .
  • the local data system 52 may include a data channel 62 , a local frame determining routine 64 and a local frame storing instruction 66 .
  • the data channel 62 may be an internal software routine, a hardware component such as an external cable or a radio frequency receiver, or a hybrid component such as a port that is opened up.
  • the data channel 62 may be configured to receive image data 68 representing a virtual content.
  • the local frame determining routine 64 may be connected to the data channel 62 .
  • the local frame determining routine 64 may be configured to determine a local coordinate frame 70 .
  • the local frame determining routine may determine the local coordinate frame based on real world objects or real world locations.
  • the local coordinate frame may be based on the origin at the initialization of the head pose session.
  • the local coordinate frame may be located where the device was booted up, or could be somewhere new if head pose was lost during the boot session.
  • the local coordinate frame may be the origin at the start of a head pose session.
  • the virtual content may also have an associated content coordinate frame.
  • the content coordinate frame may be based on a top edge relative to a bottom edge of a browser window, head or feet of a character, a node on an outer surface of a prism or bounding box that encloses the virtual content, or any other suitable location to place a coordinate frame that defines a facing direction of a virtual content and a location (e.g. a node, such as a placement node or anchor node) with which to place the virtual content, etc.
  • a location e.g. a node, such as a placement node or anchor node
  • the local frame storing instruction 66 may be connected to the local frame determining routine 64 .
  • the local frame storing instruction 66 may store the local coordinate frame 70 as a local coordinate frame 72 within the origin and destination coordinate frames 34 .
  • the origin and destination coordinate frames 34 may be one or more coordinate frames that may be manipulated or transformed in order for a virtual content to persist between sessions.
  • a session may be the period of time between a boot-up and shut-down of an XR device. Two sessions may be two start-up and shut-down periods for a single XR device, or may be a start-up and shut-down for two different XR devices.
  • the origin and destination coordinate frames 34 may be the coordinate frames involved in one or more transformations required in order for a first user's XR device and a second user's XR device to recognize a common location.
  • the destination coordinate frame may be the output of a series of computations and transformations applied to the target coordinate frame in order for a first and second user to view a virtual content in the same location.
  • the rendering engine 30 may be connected to the data channel 62 .
  • the rendering engine 30 may receive the image data 68 from the data channel 62 such that the rendering engine 30 may render virtual content based, at least in part, on the image data 68 .
  • the display system 42 may be connected to the rendering engine 30 .
  • the display system 42 may include components that transform the image data 68 into visible light.
  • the visible light may form two patterns, one for each eye.
  • the visible light may enter eyes of the first user 14 . 1 in FIG. 1 and may be detected on retinas of the eyes of the first user 14 . 1 .
  • the real object detection camera 44 may include one or more cameras that may capture images from different sides of the head-mountable frame 40 .
  • the movement tracking camera 46 may include one or more cameras that capture images on sides of the head-mountable frame 40 .
  • One set of one or more cameras may be used instead of the two sets of one or more cameras representing the real object detection camera(s) 44 and the movement tracking camera(s) 46 .
  • the cameras 44 , 46 may capture images.
  • images captured by the cameras 44 , 46 may be computed into one or more key frames (e.g., key frames 1 , 2 in FIG. 11D ).
  • a key frame may correspond to a camera pose.
  • a key frame may include one or more camera images captured at the camera pose.
  • an XR system may determine a portion of the camera images captured at the camera pose as not useful and thus not include the portion in a key frame. Therefore, using key frames to align new images with earlier knowledge of a scene reduces the use of computational resource of the XR system.
  • a key frame may be an image, or image data, at a location with a direction/angle.
  • a key frame may be a location and a direction from which one or more map point may be observed. In some embodiments, a key frame may be a coordinate frame with an ID.
  • U.S. patent application Ser. No. 15/877,359 describes key frames and is hereby incorporated by reference in its entirety.
  • the inertial measurement unit 48 may include a number of devices that are used to detect movement of the head unit 22 .
  • the inertial measurement unit 48 may include a gravitation sensor, one or more accelerometers and one or more gyroscopes.
  • the sensors of the inertial measurement unit 48 in combination, track movement of the head unit 22 in at least three orthogonal directions and about at least three orthogonal axes.
  • the world frame system 54 includes a world surface determining routine 78 , a world frame determining routine 80 , and a world frame storing instruction 82 .
  • the world surface determining routine 78 is connected to the real object detection camera 44 .
  • the world surface determining routine 78 receives images and/or key frames based on the images that are captured by the real object detection camera 44 and processes the images to identify surfaces in the images.
  • a depth sensor (not shown) may determine distances to the surfaces.
  • the surfaces are thus represented by data in three dimensions including their sizes, shapes, and distances from the real object detection camera.
  • world surface determining routine 78 may convert one or more key frames to a persistent pose (e.g., Persistent Pose 1 , 2 in FIG. 11D ).
  • a persistent pose may be the output of routine 78 .
  • a persistent pose may be a coordinate location and/or direction that has one or more associated key frames.
  • a persistent pose may be automatically created after the user has traveled a certain distance, e.g. three meters.
  • the persistent poses may act as reference points during localization.
  • the data determined by the world surface determining routine 78 may be stored in a passable world (e.g., a digital representation of the real objects in the physical world that can be stored and updated with changes to the real objects in the physical world).
  • a passable world may comprise or be created from one or more canonical maps, each of which may include a corresponding world coordinate frame.
  • a canonical map may be originated as a tracking map (TM) (e.g., TM 1102 in FIG. 40A ), which may be converted to a canonical map after the TM's session ended.
  • TM tracking map
  • a TM may be a headpose sparse map of an XR device.
  • XR devices may send one or more TMs to cloud for merging with additional TMs captured by the XR device at a different time or by other XR devices.
  • the persistent frame determining routine 80 may be connected to the world surface determining routine 78 and may determine a persistent coordinate frame (PCF) 84 .
  • the PCF may include multiple PCF anchors (e.g., PCF anchors 1 - 5 in FIG. 11D ). A new PCF anchor may be determined based on a pre-defined distance allowed between different PCF anchors.
  • the world surface determining routine 78 may store in the passable world the PCF anchors, which may constitute the PCF.
  • the world frame determining routine 80 may convert one or more persistent poses into a PCF anchor when the user travels a pre-determined distance, e.g. five meters.
  • PCF anchors may be associated with one or more world coordinate frames 86 , e.g., in the passable world.
  • the world coordinate frame 86 may be stored within the origin and destination coordinate frames 34 .
  • a virtual content's own content coordinate frame (i.e. a coordinate frame attached to the virtual content independent of its real world location) may be associated with one or more PCF anchors.
  • PCF anchors may be associated with associating virtual content with PCF anchors, as opposed to being measured directly to the world coordinate frame, may give the virtual content a more persistent frame position. For example, if a virtual lamp is placed on a table, there could be a plurality of data points on the table to provide placement input for relative positioning of the virtual lamp that does not substantially change over time.
  • the virtual lamp may continue to utilize the same local coordinate frame rather than adjust to a new world frame which may introduce jitter or positional shifts in the appearance of the lamp.
  • the persistent frame storing instruction 82 may be connected to the persistent frame determining routine 80 to receive the persistent coordinate frame 84 from the persistent frame determining routine 80 .
  • the persistent frame storing instruction 82 may store the persistent coordinate frame 84 in a database locally and/or in the cloud (e.g. may depend on security settings).
  • the head frame system 56 may include a head frame determining routine 90 and a head frame storing instruction 92 .
  • the head frame determining routine 90 may be connected to the movement tracking camera 46 and the inertial measurement unit 48 .
  • the head frame determining routine 90 may use data from the movement tracking camera 46 and the inertial measurement unit 48 to calculate a head coordinate frame 94 .
  • the inertial measurement unit 48 may have a gravitation sensor that determines the direction of gravitational force relative to the head unit 22 .
  • the movement tracking camera 46 may continually capture images that are used by the head frame determining routine 90 to refine the head coordinate frame 94 .
  • the head unit 22 moves when the first user 14 . 1 in FIG. 1 moves their head.
  • the movement tracking camera 46 and the inertial measurement unit 48 may continuously provide data to the head frame determining routine 90 so that the head frame determining routine 90 can update the head coordinate frame 94 .
  • the head frame storing instruction 92 may be connected to the head frame determining routine 90 to receive the head coordinate frame 94 from the head frame determining routine 90 .
  • the head frame storing instruction 92 may store the head coordinate frame 94 as a head coordinate frame 96 among the origin and destination coordinate frames 34 .
  • the head frame storing instruction 92 may repeatedly store the updated head coordinate frame 94 as the head coordinate frame 96 when the head frame determining routine 90 recalculates the head coordinate frame 94 .
  • the head coordinate frame may be the location of the wearable XR device 12 . 1 relative to the local coordinate frame 72 .
  • the camera frame system 58 may include camera intrinsics 98 .
  • the camera intrinsics 98 may include dimensions of the head unit 22 that are features of its design and manufacture.
  • the camera intrinsics 98 may be used to calculate a camera coordinate frame 100 that is stored within the origin and destination coordinate frames 34 .
  • the camera coordinate frame 100 may include all pupil positions of a left eye of the first user 14 . 1 in FIG. 1 . When the left eye moves from left to right or up and down, the pupil positions of the left eye are located within the camera coordinate frame 100 . In addition, the pupil positions of a right eye are located within a camera coordinate frame 100 for the right eye. In some embodiments, the camera coordinate frame 100 may include the location of the camera relative to the local coordinate frame when an image is taken.
  • the origin to destination coordinate frame transformers 36 may include a local-to-world coordinate transformer 104 , a world-to-head coordinate transformer 106 , and a head-to-camera coordinate transformer 108 .
  • the local-to-world coordinate transformer 104 may receive the local coordinate frame 72 and transform the local coordinate frame 72 to the world coordinate frame 86 .
  • the transformation of the local coordinate frame 72 to the world coordinate frame 86 may be represented as a local coordinate frame transformed to world coordinate frame 110 within the world coordinate frame 86 .
  • the world-to-head coordinate transformer 106 may transform from the world coordinate frame 86 to the head coordinate frame 96 .
  • the world-to-head coordinate transformer 106 may transform the local coordinate frame transformed to world coordinate frame 110 to the head coordinate frame 96 .
  • the transformation may be represented as a local coordinate frame transformed to head coordinate frame 112 within the head coordinate frame 96 .
  • the head-to-camera coordinate transformer 108 may transform from the head coordinate frame 96 to the camera coordinate frame 100 .
  • the head-to-camera coordinate transformer 108 may transform the local coordinate frame transformed to head coordinate frame 112 to a local coordinate frame transformed to camera coordinate frame 114 within the camera coordinate frame 100 .
  • the local coordinate frame transformed to camera coordinate frame 114 may be entered into the rendering engine 30 .
  • the rendering engine 30 may render the image data 68 representing the local content 28 based on the local coordinate frame transformed to camera coordinate frame 114 .
  • FIG. 3 is a spatial representation of the various origin and destination coordinate frames 34 .
  • the local coordinate frame 72 , world coordinate frame 86 , head coordinate frame 96 , and camera coordinate frame 100 are represented in the figure.
  • the local coordinate frame associated with the XR content 28 may define a position and rotation of the virtual content (e.g. may provide a node and facing direction), which may then be transformed to a local and/or world coordinate frame and/or PCF when the virtual content is placed in the real world so the virtual content may be viewed by the user.
  • Each camera may have its own camera coordinate frame 100 encompassing all pupil positions of one eye.
  • Reference numerals 104 A and 106 A represent the transformations that are made by the local-to-world coordinate transformer 104 , world-to-head coordinate transformer 106 , and head-to-camera coordinate transformer 108 in FIG. 2 , respectively.
  • FIG. 4 depicts a camera render protocol for transforming from a head coordinate frame to a camera coordinate frame, according to some embodiments.
  • a pupil for a single eye moves from position A to B.
  • a virtual object that is meant to appear stationary will project onto a depth plane at one of the two positions A or B depending on the position of the pupil (assuming that the camera is configured to use a pupil as its coordinate frame).
  • using a pupil coordinate frame transformed to a head coordinate frame will cause jitter in a stationary virtual object as the eye moves from position A to position B. This situation is referred to as view dependent display or projection.
  • a camera coordinate frame (e.g., CR) is positioned and encompasses all pupil positions and object projection will now be consistent regardless of pupil positions A and B.
  • the head coordinate frame transforms to the CR frame, which is referred to as view independent display or projection.
  • An image reprojection may be applied to the virtual content to account for a change in eye position, however, as the rendering is still in the same position, jitter is minimized.
  • a second XR device 12 . 2 which may be in the same scene as the first XR device 12 . 1 , may include a persistent coordinate frame (PCF) integration unit 1300 , an application 1302 that generates the image data 68 , and a frame embedding generator 308 .
  • a map download system 126 , anchor identification system 128 , Map 2 , localization module 130 , canonical map incorporator 132 , canonical map 133 , and map publisher 136 may be grouped into a passable world unit 1304 .
  • the PCF integration unit 1300 may be connected to the passable world unit 1304 and other components of the second XR device 12 .
  • a coordinate frame, constituted by PCF anchors may enable more persistence in a changing world, for example, a world where people move into and out of the scene and objects such as doors move relative to the scene, require less storage space and transmission rates, and enable the use of individual PCF anchors and their relationships relative to one another and/or integrated constellation of PCF anchors to map a scene.
  • the PCF integration unit 1300 may include PCF anchors 1306 that were previously stored in a data store on a storage unit of the second XR device 12 . 2 , a PCF tracker 1308 , a persistent pose acquirer 1310 , a PCF checker 1312 , a PCF generation system 1314 , a coordinate frame calculator 1316 , a persistent pose calculator 1318 , and three transformers, including a tracking map and persistent pose transformer 1320 , a persistent pose and PCF transformer 1322 and a PCF and image data transformer 1324 .
  • the PCF tracker 1308 may have an on-prompt and an off-prompt that are selectable by the application 1302 .
  • the application 1302 may be executable by a processor of the second XR device 12 . 2 to, for example, display a virtual content.
  • the application 1302 may have a call that switches the PCF tracker 1308 on via the on-prompt.
  • the PCF tracker 1308 may generate PCF anchors when the PCF tracker 1308 is switched on.
  • the application 1302 may have a subsequent call that can switch the PCF tracker 1308 off via the off-prompt.
  • the PCF tracker 1308 terminates PCF generation when the PCF tracker 1308 is switched off.
  • the server 20 may include a plurality of persistent poses 1332 and a plurality of PCF anchors 1330 that have previously been saved in association with a canonical map 120 .
  • the map transmitter 122 may transmit the canonical map 120 together with the persistent poses 1332 and/or the PCF anchors 1330 to the second XR device 12 . 2 .
  • the persistent poses 1332 and PCF anchors 1330 may be stored in association with the canonical map 133 on the second XR device 12 . 2 .
  • Map 2 localizes to the canonical map 133
  • the persistent poses 1332 and the PCF anchors 1330 may be stored in association with Map 2 .
  • the persistent pose acquirer 1310 may acquire the persistent poses for Map 2 .
  • the PCF checker 1312 may be connected to the persistent pose acquirer 1310 .
  • the PCF checker 1312 may retrieve PCF anchors from the PCF anchors 1306 based on the persistent poses retrieved by the persistent pose acquirer 1310 .
  • the PCF anchors retrieved by the PCF checker 1312 may form an initial group of PCF anchors that are used for image display based on PCF anchors.
  • the application 1302 may require additional PCF anchors to be generated. For example, if a user moves to an area that has not previously been mapped, the application 1302 may switch the PCF tracker 1308 on.
  • the PCF generation system 1314 may be connected to the PCF tracker 1308 and begin to generate PCF anchors based on Map 2 as Map 2 begins to expand.
  • the PCF anchors generated by the PCF generation system 1314 may form a second group of PCF anchors that may be used for PCF-based image display.
  • the coordinate frame calculator 1316 may be connected to the PCF checker 1312 . After the PCF checker 1312 retrieved PCF anchors, the coordinate frame calculator 1316 may invoke the head coordinate frame 96 to determine a head pose of the second XR device 12 . 2 . The coordinate frame calculator 1316 may also invoke the persistent pose calculator 1318 .
  • the persistent pose calculator 1318 may be directly or indirectly connected to the frame embedding generator 308 .
  • the frame embedding generator 308 may output a global feature string 316 which may be used as a data representation for an image (e.g. frame or key frame).
  • an image/frame may be designated a key frame after a threshold distance from the previous key frame, e.g. 3 meters, is traveled.
  • the persistent pose calculator 1318 may generate a persistent pose based on a plurality, for example three, key frames. In some embodiments, the persistent pose may be essentially an average of the coordinate frames of the plurality of key frames.
  • the tracking map and persistent pose transformer 1320 may be connected to Map 2 and the persistent pose calculator 1318 .
  • the tracking map and persistent pose transformer 1320 may transform Map 2 to the persistent pose to determine the persistent pose at an origin relative to Map 2 .
  • the persistent pose and PCF transformer 1322 may be connected to the tracking map and persistent pose transformer 1320 and further to the PCF checker 1312 and the PCF generation system 1314 .
  • the persistent pose and PCF transformer 1322 may transform the persistent pose (to which the tracking map has been transformed) to the PCF anchors from the PCF checker 1312 and the PCF generation system 1314 to determine the PCF's relative to the persistent pose.
  • the PCF and image data transformer 1324 may be connected to the persistent pose and PCF transformer 1322 and to the data channel 62 .
  • the PCF and image data transformer 1324 transforms the PCF's to the image data 68 .
  • the rendering engine 30 may be connected to the PCF and image data transformer 1324 to display the image data 68 to the user relative to the PCF anchors.
  • the PCF integration unit 1300 may store the additional PCF anchors that are generated with the PCF generation system 1314 within the PCF anchors 1306 .
  • the PCF anchors 1306 may be stored relative to persistent poses.
  • the map publisher 136 may retrieve the PCF anchors 1306 and the persistent poses associated with the PCF anchors 1306 when the map publisher 136 transmits Map 2 to the server 20 , the map publisher 136 also transmits the PCF's and persistent poses associated with Map 2 to the server 20 .
  • the map storing routine 118 of the server 20 stores Map 2
  • the map storing routine 118 may also store the persistent poses and PCF anchors generated by the second viewing device 12 . 2 .
  • the map merge algorithm 124 may create the canonical map 120 with the persistent poses and PCF anchors of Map 2 associated with the canonical map 120 and stored within the persistent poses 1332 and PCF anchors 1330 , respectively.
  • the first XR device 12 . 1 may include a PCF integration unit similar to the PCF integration unit 1300 of the second XR device 12 . 2 .
  • the map transmitter 122 may transmit the persistent poses 1332 and PCF's 1330 associated with the canonical map 120 and originating from the second XR device 12 . 2 .
  • the first XR device 12 . 1 may store the PCF anchors and the persistent poses within a data store on a storage device of the first XR device 12 . 1 .
  • the first XR device 12 . 1 may then make use of the persistent poses and the PCF anchors originating from the second XR device 12 .
  • the first XR device 12 . 1 may retrieve, generate, make use, upload, and download PCF anchors and persistent poses in a manner similar to the second XR device 12 . 2 as described above.
  • FIG. 44 illustrates key frames that are generated for a room.
  • a first key frame, KF 1 is generated for a door on a left wall of the room.
  • a second key frame, KF 2 is generated for an area in a corner where a floor, the left wall, and a right wall of the room meet.
  • a third key frame, KF 3 is generated for an area of a window on the right wall of the room.
  • a fourth key frame, KF 4 is generated for an area at a far end of a rug on a floor of the wall.
  • a fifth key frame, KF 5 is generated for an area of the rug closest to the user.
  • FIG. 45 illustrates persistent poses PP 1 and PP 2 in the environment of a user.
  • a new persistent pose is created when the device measures a threshold distance traveled, and/or when an application requests a new persistent pose.
  • the threshold distance may be 3 meters, 5 meters, 20 meters, or any other suitable distance. Selecting a smaller threshold distance (e.g., 1 m) may result in an increase in compute load since a larger number of PPs may be created and managed compared to larger threshold distances. Selecting a larger threshold distance (e.g.
  • 40 m may result in increased virtual content placement error since a smaller number of PPs would be created, which would result in fewer PCFs being created, which means the virtual content attached to the PCF anchor could be a relatively large distance (e.g. 30 m) away from the PCF anchor, and error increases with increasing distance from an anchor to the virtual content.
  • a PP may be created at the start of a new session. This initial PP may be thought of as zero, and can be visualized as the center of a circle that has a radius equal to the threshold distance. When the device reaches the perimeter of the circle, and, in some embodiments, an application requests a new PP, a new PP may be placed at the current location of the device (at the threshold distance). In some embodiments, a new PP will not be created at the threshold distance if the device is able to find an existing PP within the threshold distance from the device's new position. In some embodiments, when a new PP is created, the device attaches one or more of the closest key frames to the PP.
  • the location of the PP relative to the key frames may be based on the location of the device at the time a PP is created. In some embodiments, a PP will not be created when the device travels a threshold distance unless an application requests a PP.
  • an application may request PCF from the device when the application has virtual content to display to the user.
  • the PCF request from the application may trigger a PP request, and a new PP would be created after the device travels the threshold distance.
  • FIG. 45 illustrates a first persistent pose PP 1 which may have the closest key frames, (e.g. KF 1 , KF 2 , and KF 3 ) attached.
  • FIG. 45 also illustrates a second persistent pose PP 2 which may have the closest key frames (e.g. KF 4 and KF 5 ) attached.
  • FIG. 46 illustrates a PCF anchor that may include PP 1 and PP 2 .
  • the PCF may be used for displaying image data relative to the PCF.
  • each PCF may have coordinates in another coordinate frame (e.g., a world coordinate frame) and a descriptor.
  • various constellations of PCF anchors may be combined to represent the real world in a persistent manner and that requires less data and less transmission of data.
  • FIG. 47 illustrates a method 4700 of establishing and using a persistence coordinate frame, according to some embodiments.
  • the method 4700 may start from capturing (Act 4702 ) images about a scene using one or more sensors of an XR device. Multiple cameras may be used and one camera may generate multiple images.
  • the method 4700 may include extracting ( 4704 ) features from the captured images, generating (Act 4706 ) descriptors for the extracted features, generating (Act 4708 ) key frames based on the descriptors, and generating (Act 4710 ) persistent poses based on the key frames.
  • key frames may be generated based on each one of the images and the average of the key frames of all the images of all the cameras may be used to generate a persistent pose.
  • the method 4700 may include generating (Act 4712 ) anchors based on the persistent poses.
  • An anchor may include one or more persistent poses.
  • the method 4700 may include associating (Act 4714 ) image data of a virtual content to be displayed by the XR device to at least one of the anchors.
  • FIGS. 48A to 48I are schematic diagrams illustrating an example of establishing and using a persistent coordinate frame, which may enable data sharing between XR devices.
  • FIG. 48A shows two users 4802 A, 4802 B with respective local tracking maps 4804 A, 4804 B that have not localized to a canonical map.
  • the origins 4806 A, 4806 B for individual users are depicted by the coordinate system (e.g., a world coordinate system) in their respective areas.
  • the TM 4802 A may include a persistent pose (PP) 4808 A; the TM 4802 B may include a PP 4808 B.
  • FIG. 48B shows that the XR devices worn by respective users 4802 A, 4802 B may create local PCF anchors 4810 A, 4810 B based on the PP 4808 A, 4908 B.
  • FIG. 48C shows that a persistent content 4812 (e.g., a virtual content) can now attach to the PCF anchors 4810 A, 4810 B.
  • FIG. 48D shows that a canonical map 1814 , for example, from a cloud, is received by both devices.
  • FIG. 48E shows that both devices localize to the canonical map 4814 on the respective devices.
  • FIG. 48F shows that each device creates an attachment (e.g., attachments 4816 A, 4816 B) between a local PCF anchor (e.g., PCF anchors 4810 A, 4810 B) on the respective device to a respective persistent pose (e.g., PPs 4818 A, 4818 B) on the canonical map.
  • FIG. 48G shows a persistent pose snapshot from the canonical map to the local tracking maps. As can be seen, the local tracking maps are connected to one another via the persistent poses.
  • FIG. 48H shows that the PCF anchor 4810 A on the device worn by the user 4802 A is accessible in the device worn by the user 4802 B.
  • FIG. 48I shows that all of the maps may eventually merge, and the PCF may “unionize” in connected constellations.
  • FIG. 6 illustrates the first XR device 12 . 1 and vision data and algorithms of a second XR device 12 . 2 and the server 20 in more detail, according to some embodiments.
  • the first XR device 12 . 1 may be configured the same as the second XR device 12 . 2 .
  • the server 20 may have a map storing routine 118 , a canonical map 120 , a map transmitter 122 , and a map merge algorithm 124 .
  • the first XR device 12 . 1 generates a local tracking map (referred to hereinafter as “Map 1 ”) and the map storing routine 118 receives Map 1 from the first XR device 12 . 1 .
  • the map storing routine 118 then stores Map 1 on a storage device of the server 20 as the canonical map 120 .
  • the second XR device 12 . 2 includes a map download system 126 , an anchor identification system 128 , a localization module 130 , a canonical map incorporator 132 , a local content position system 134 , and a map publisher 136 .
  • the map transmitter 122 sends the canonical map 120 to the second XR device 12 . 2 and the map download system 126 downloads and stores the canonical map 120 as a canonical map 133 from the server 20
  • the anchor identification system 128 is connected to the world surface determining routine 78 .
  • the anchor identification system 128 identifies anchors based on objects detected by the world surface determining routine 78 .
  • the anchor identification system 128 generates a second map (Map 2 ) using the anchors.
  • Map 2 a second map
  • the anchor identification system 128 continues to identify anchors and continues to update Map 2 .
  • the locations of the anchors are recorded as three-dimensional data based on data provided by the world surface determining routing 78 .
  • the world surface determining routine 78 receives images from the real object detection camera 44 and depth data from depth sensors 135 to determine the locations of surfaces and their relative distance from the depth sensors 135
  • the localization module 130 is connected to the canonical map 133 and Map 2 .
  • the localization module 130 repeatedly attempts to localize Map 2 to the canonical map 133 .
  • the canonical map incorporator 132 is connected to the canonical map 133 and Map 2 .
  • the canonical map incorporator 132 incorporates the canonical map 133 into anchors of Map 2 . Map 2 is then updated with missing data that is included in the canonical map.
  • the local content position system 134 is connected to Map 2 .
  • the local content position system 134 may, for example, be a system wherein a user can locate local content in a particular location within a world coordinate frame. The local content then attaches itself to one anchor of Map 2 .
  • the local-to-world coordinate transformer 104 transforms the local coordinate frame to the world coordinate frame based on the settings of the local content position system 134 .
  • the functioning of the rendering engine 30 , display system 42 , and data channel 62 have been described with reference to FIG. 2 .
  • the map publisher 136 uploads Map 2 to the server 20 .
  • the map storing routine 118 of the server 20 then stores Map 2 within a storage medium of the server 20 .
  • the map merge algorithm 124 merges Map 2 with the canonical map 120 .
  • the map merge algorithm 124 merges all the maps into the canonical map 120 to render a new canonical map 120 .
  • the map transmitter 122 then transmits the new canonical map 120 to any and all devices 12 . 1 and 12 . 2 that are in an area represented by the new canonical map 120 .
  • the devices 12 . 1 and 12 . 2 localize their respective maps to the canonical map 120 , the canonical map 120 becomes the promoted map.
  • FIG. 7 illustrates the display system 42 in more detail.
  • the display system 42 includes a stereoscopic analyzer 144 that is connected to the rendering engine 30 and forms part of the vision data and algorithms.
  • the display system 42 further includes left and right projectors 166 A and 166 B and left and right waveguides 170 A and 170 B.
  • the left and right projectors 166 A and 166 B are connected to power supplies.
  • Each projector 166 A and 166 B has a respective input for image data to be provided to the respective projector 166 A or 166 B.
  • the respective projector 166 A or 166 B when powered, generates light in two-dimensional patterns and emanates the light therefrom.
  • the left and right waveguides 170 A and 170 B are positioned to receive light from the left and right projectors 166 A and 166 B, respectively.
  • the left and right waveguides 170 A and 170 B are transparent waveguides.
  • a user mounts the head mountable frame 40 to their head.
  • Components of the head mountable frame 40 may, for example, include a strap (not shown) that wraps around the back of the head of the user.
  • the left and right waveguides 170 A and 170 B are then located in front of left and right eyes 220 A and 220 B of the user.
  • the rendering engine 30 enters the image data that it receives into the stereoscopic analyzer 144 .
  • the image data is three-dimensional image data of the local content 28 in FIG. 1 .
  • the image data is projected onto a plurality of virtual planes.
  • the stereoscopic analyzer 144 analyzes the image data to determine left and right image data sets based on the image data for projection onto each depth plane.
  • the left and right image data sets are data sets that represent two-dimensional images that are projected in three-dimensions to give the user a perception of a depth.
  • the stereoscopic analyzer 144 enters the left and right image data sets into the left and right projectors 166 A and 166 B.
  • the left and right projectors 166 A and 166 B then create left and right light patterns.
  • the components of the display system 42 are shown in plan view, although it should be understood that the left and right patterns are two-dimensional patterns when shown in front elevation view.
  • Each light pattern includes a plurality of pixels. For purposes of illustration, light rays 224 A and 226 A from two of the pixels are shown leaving the left projector 166 A and entering the left waveguide 170 A. The light rays 224 A and 226 A reflect from sides of the left waveguide 170 A.
  • the light rays 224 A and 226 A propagate through internal reflection from left to right within the left waveguide 170 A, although it should be understood that the light rays 224 A and 226 A also propagate in a direction into the paper using refractory and reflective systems.
  • the light rays 224 A and 226 A exit the left light waveguide 170 A through a pupil 228 A and then enter a left eye 220 A through a pupil 230 A of the left eye 220 A.
  • the light rays 224 A and 226 A then fall on a retina 232 A of the left eye 220 A.
  • the left light pattern falls on the retina 232 A of the left eye 220 A.
  • the user is given the perception that the pixels that are formed on the retina 232 A are pixels 234 A and 236 A that the user perceives to be at some distance on a side of the left waveguide 170 A opposing the left eye 220 A. Depth perception is created by manipulating the focal length of the light.
  • the stereoscopic analyzer 144 enters the right image data set into the right projector 166 B.
  • the right projector 166 B transmits the right light pattern, which is represented by pixels in the form of light rays 224 B and 226 B.
  • the light rays 224 B and 226 B reflect within the right waveguide 170 B and exit through a pupil 228 B.
  • the light rays 224 B and 226 B then enter through a pupil 230 B of the right eye 220 B and fall on a retina 232 B of a right eye 220 B.
  • the pixels of the light rays 224 B and 226 B are perceived as pixels 134 B and 236 B behind the right waveguide 170 B.
  • the patterns that are created on the retinas 232 A and 232 B are individually perceived as left and right images.
  • the left and right images differ slightly from one another due to the functioning of the stereoscopic analyzer 144 .
  • the left and right images are perceived in a mind of the user as a three-dimensional rendering.
  • the left and right waveguides 170 A and 170 B are transparent. Light from a real-life object such as the table 16 on a side of the left and right waveguides 170 A and 170 B opposing the eyes 220 A and 220 B can project through the left and right waveguides 170 A and 170 B and fall on the retinas 232 A and 232 B.
  • FIG. 8 illustrates Map 1 and local content (Content 123 and Content 456 ) on the first XR device 12 . 1 .
  • Map 1 includes a number of anchors (Anchor a to Anchor d). From the perspective of the first XR device 12 . 1 , Anchor a, by way of example, has X, Y, and Z coordinates of (0,0,0) and Anchor b has X, Y, and Z coordinates ( ⁇ 1,0,0).
  • Content 123 associated with Anchor a In the present example, Content 123 has an X, Y, and Z relationship relative to Anchor a of (1,0,0).
  • Content 456 has a relationship relative to Anchor b. In the present example, Content 456 has an X, Y, and Z relationship of (1,0,0) relative to Anchor b.
  • Map 1 also has an origin (Origin 1 ).
  • the first XR device 12 . 1 uploads Map 1 to the server 20 .
  • the server 20 now has a canonical map based on Map 1 .
  • the first XR device 12 . 1 has a canonical map that is empty at this stage.
  • the server 20 for purposes of discussion, and in some embodiments, includes no other maps other than Map 1 . No maps are stored on the second XR device 12 . 2 .
  • the first XR device 12 . 1 also transmits its Wi-Fi signature data to the server 20 .
  • the server 20 may use the Wi-Fi signature data to determine a rough location of the first XR device 12 . 1 based on intelligence gathered from other devices that have, in the past, connected to the server 20 or other servers together with the GPS locations of such other devices that have been recorded.
  • the first XR device 12 . 1 may now end the first session (See FIG. 1 ) and may disconnect from the server 20 .
  • FIGS. 10 and 11A show the initiation of a second session by a second user 14 . 2 .
  • the first user 14 . 1 is shown in phantom lines because the first session by the first user 14 . 1 has ended.
  • the second XR device 12 . 2 begins to record objects.
  • Various systems with varying degrees of granulation may be used by the server 20 to determine that the second session by the second XR device 12 . 2 is in the same vicinity of the first session by the first XR device 12 . 1 .
  • Wi-Fi signature data, global positioning system (GPS) positioning data, GPS data based on Wi-Fi signature data, or any other data that indicates a location may be included in the first and second XR devices 12 . 1 and 12 . 2 to record their locations.
  • the anchors that are identified by the second XR device 12 . 2 may show a similarity to the anchors of Map 1 .
  • the second XR device boots up and begins to collect data, such as images 1110 from one or more cameras 44 , 46 .
  • a XR device e.g. the second XR device 12 . 2
  • Each feature may be converted to a descriptor 1130 .
  • the descriptors 1130 may be used to describe a key frame 1140 , which may have the position and direction of the associated image attached.
  • One or more key frames 1140 may correspond to a single persistent pose 1150 , which may be automatically generated after a threshold distance from the previous persistent pose 1150 , e.g. 3 meters.
  • One or more persistent poses 1150 may correspond to a single PCF 1160 , which may be automatically generated after a pre-determined distance, e.g. every 5 meters.
  • additional PCF anchors e.g. PCF anchors 4 , 5 and PCF anchor 3
  • An application or two 1180 may run on the XR device and provide virtual content 1170 to the XR device for presentation to the user.
  • the virtual content may have an associated content coordinate frame which may be placed relative to one or more PCF anchors.
  • the second XR device 12 . 2 creates three PCF anchors.
  • the second XR device 12 . 2 may try to localize into one or more canonical maps stored on the server 20 .
  • the second XR device 12 . 2 may download the canonical map 120 from the server 20 .
  • Map 1 on the second XR device 12 . 2 includes anchors a to d and Origin 1 .
  • the server 20 may have multiple canonical maps for various locations and may determine that the second XR device 12 . 2 is in the same vicinity as the vicinity of the first XR device 12 . 1 during the first session and sends the second XR device 12 . 2 the canonical map for that vicinity.
  • FIG. 12 shows the second XR device 12 . 2 beginning to identify anchors for purposes of generating Map 2 .
  • the second XR device 12 . 2 has only identified a single anchor, namely PCF 1 , 2 .
  • the X, Y, and Z coordinates of PCF 1 , 2 for the second XR device 12 . 2 may be (1,1,1).
  • Map 2 has its own origin (Origin 2 ), which may be based on the head pose of device 2 for the current head pose session.
  • the second XR device 12 . 2 may immediately attempt to localize Map 2 to the canonical map.
  • Map 2 may not be able to localize into Canonical Map (Map 1 ) (i.e.
  • the system may localize based on anchor/PCF comparison between the local and canonical maps. In some embodiments, the system may localize based on persistent pose comparison between the local and canonical maps. In some embodiments, the system may localize based on key frame comparison between the local and canonical maps.
  • FIG. 13 shows Map 2 after the second XR device 12 . 2 has identified further anchors (PCF 1 , 2 , PCF 3 , PCF 4 , 5 ) of Map 2 .
  • the second XR device 12 . 2 again attempts to localize Map 2 to the canonical map. Because Map 2 has expanded to overlap with at least a portion of the Canonical Map, the localization attempt will succeed.
  • the overlap between the local tracking map, Map 2 , and the Canonical Map may be represented by PCF/anchors, persistent poses, key frames, or any other suitable intermediate or derivative construct.
  • the second XR device 12 . 2 has associated Content 123 and Content 456 to PCFs 1 , 2 and PCF 3 of Map 2 .
  • Content 123 has X, Y, and Z coordinates relative to PCF 1 , 2 of (1,0,0).
  • FIGS. 14A and 14B illustrate a successful localization of Map 2 to the canonical map.
  • the overlapping area/volume/section of the maps 1410 represent the common parts to Map 1 and the canonical map. Since Map 2 created PCFs 3 and 4 , 5 before localizing, and the Canonical map created anchors a and c before Map 2 was created, different anchors/PCFs were created to represent the same volume in real space (e.g., in different maps).
  • the second XR device 12 . 2 expands Map 2 to include Anchors a-d from the Canonical Map.
  • Anchors a-d represents the localization of Map 2 to the Canonical Map.
  • the XR system may perform an optimization step to remove duplicate anchors/PCFs from overlapping areas, such as the PCFs in 1410 , PCF 3 and PCF 4 , 5 .
  • Map 2 localizes, the placement of virtual content, such as Content 456 and Content 123 will be relative to the closest updated anchors/PCFs in the updated Map 2 .
  • the virtual content appears in the same real-world location relative to the user, despite the changed PCF attachment for the content, and despite the updated anchors for Map 2 .
  • the second XR device 12 . 2 continues to expand Map 2 as further anchors (Anchors e, f, g, and h) are identified by the second XR device 12 . 2 , for example as the user walks around the real world. It can also be noted that Map 1 has not expanded in FIGS. 15 and 16 .
  • the second XR device 12 . 2 uploads Map 2 to the server 20 .
  • the server 20 stores Map 2 together with the canonical map.
  • Map 2 may upload to the server 20 when the session ends for the second XR device 12 . 2 .
  • the canonical map within the server 20 now includes anchor i which is not included in Map 1 on the first XR device 12 . 1 .
  • the canonical map on the server 20 may have expanded to include anchor i when a third XR device (not shown) uploaded a map to the server 20 and such a map included anchor i.
  • the server 20 merges Map 2 with the canonical map to form a new canonical map.
  • the server 20 determines that anchors a to d are common to the canonical map and Map 2 .
  • the server expands the canonical map to include anchors e to h and PCF 1 , 2 from Map 2 to form a new canonical map.
  • the canonical maps on the first and second XR devices 12 . 1 and 12 . 2 are based on Map 1 and are outdated.
  • the server 20 transmits the new canonical map to the first and second XR devices 12 . 1 and 12 . 2 . In some embodiments, this may occur when the first XR device 12 . 1 and second device 12 . 2 try to localize during a different or new or subsequent session.
  • the first and second XR devices 12 . 1 and 12 . 2 proceed as described above to localize their respective local maps (Map 1 and Map 2 respectively) to the new canonical map.
  • the head coordinate frame 96 or “head pose” is related to the anchors in Map 2 .
  • the origin of the map, Origin 2 is based off of the head pose of second XR device 12 . 2 at the start of the session.
  • the anchors are placed relative to the local coordinate frame, Origin 2 .
  • the anchors of Map 2 serve as a persistent coordinate frames relative to a world coordinate frame, where the world coordinate frame may be a previous session's local coordinate frame (e.g. Map 1 's Origin 1 in FIG. 8 ).
  • Map 1 's Origin 1 in FIG. 8 The transformation from the world coordinate frame to the head coordinate frame 96 has been previously discussed with reference to FIG. 2 .
  • the head coordinate frame 96 is in a three-dimensional location relative to the anchors of Map 2 and has three orthogonal axes within three-dimensional space.
  • the head coordinate frame 96 has moved relative to the anchors of Map 2 .
  • the head coordinate frame 96 has moved because the second user 14 . 2 has moved their head.
  • the user can move their head in six degrees of freedom (6 dof).
  • the head coordinate frame 96 can thus move in 6 dof, namely in three-dimensions from its previous location in FIG. 20 and about three orthogonal axes relative to the anchors of Map 2 .
  • the head coordinate frame 96 is adjusted when the real object detection camera 44 and inertial measurement unit 48 in FIG. 2 respectively detect real objects and motion of the head unit 22 . More information regarding head pose tracking is disclosed in U.S.
  • FIG. 22 shows that sound may be associated with one or more anchors.
  • a user may, for example, wear headphones or earphones with stereoscopic sound.
  • the location of sound through headphones can be simulated using conventional techniques.
  • the location of sound may be located in a stationary position so that, when the user rotates their head to the left, the location of sound rotates to the right so that the user perceives the sound coming from the same location in the real world.
  • location of sound is represented by Sound 123 and Sound 456 .
  • FIG. 22 is similar to FIG. 16 in its analysis. When the first and second users 14 . 1 and 14 . 2 are located in the same room at the same or different times, they perceive Sound 123 and Sound 456 coming from the same locations within the real world.
  • FIGS. 23 and 24 illustrate a further implementation of the technology described above.
  • the first user 14 . 1 has initiated a first session as described with reference to FIG. 1 .
  • the first user 14 . 1 has terminated the first session as indicated by the phantom lines.
  • the first XR device 12 . 1 uploaded Map 1 to the server 20 .
  • the first user 14 . 1 has now initiated a second session at a later time than the first session.
  • the first XR device 12 . 1 does not download Map 1 from the server 20 because Map 1 is already stored on the first XR device 12 . 1 . If Map 1 is lost, then the first XR device 12 . 1 downloads Map 1 from the server 20 .
  • the first XR device 12 . 1 then proceeds to build anchors for Map 2 , localizes to Map 1 and further develops a canonical map as described above. Map 2 is then used for relating local content, a head coordinate frame, local sound, etc. as described above.
  • the first user 14 . 1 and the second user 14 . 2 are joined by a third user 14 . 3 with a third XR device 12 . 3 .
  • Each XR device 12 . 1 , 12 . 2 and 12 . 3 begins to generate its own map, namely Map 1 , Map 2 and Map 3 , respectively.
  • Map 1 , Map 2 and Map 3 Maps 1 , 2 , and 3
  • the maps are incrementally uploaded to the server 20 .
  • the server 20 merges Maps 1 , 2 , and 3 to form a canonical map.
  • the canonical map is then transmitted from the server 20 to each one of the XR devices 12 . 1 , 12 . 2 and 12 . 3 .
  • FIG. 27 shows a plurality of canonical maps 120 that are stored in data storage on a computer-readable medium of the server 20 .
  • Each canonical map 120 has a plurality of canonical map identifiers associated therewith.
  • These canonical map identifiers include an area on the planet Earth represented by a range of longitudes and latitudes.
  • the canonical map identifiers also include global feature strings.
  • the canonical maps 120 are illustrated geographically in a two-dimensional pattern as they may exist on a surface of the earth.
  • the longitudes and latitudes differ from one another to uniquely identify the canonical maps.
  • the server 20 has a position detector that receives position data, including longitude and latitude, a Wi-Fi fingerprint, and features of an image that is taken by the respective viewing device 12 . 1 or 12 . 2 .
  • the position detector then stores the position identifiers.
  • Not all viewing devices provide the server 20 with longitude and latitude as position identifiers. However, viewing devices will normally include a Wi-Fi fingerprint and image features as position identifiers.
  • FIG. 28 illustrates the canonical maps 120 and a series of filters, including a neighboring areas filter 300 , a Wi-Fi fingerprint filter 302 , and a key frame filter 304 that the sever 20 executes to determine one or more maps that are sent to a viewing device, followed by localization 306 on the viewing device as described above.
  • the canonical maps 120 may have many canonical maps, such as thousands of canonical maps. In some embodiments, it may not be possible to send all the canonical maps to a viewing device due to bandwidth restrictions.
  • the neighboring areas filter 300 may select at least one matching canonical map 120 covering longitude and latitude that include that longitude and latitude of the position identifier received from the viewing device, as long as at least one map exists at that longitude and latitude. In some embodiments, the neighboring areas filter 300 further selects at least one neighboring canonical map covering longitude and latitude that are adjacent the matching canonical map. In some embodiments, the neighboring areas filter 300 may select a plurality of matching canonical maps and a plurality of neighboring canonical maps. The neighboring areas filter 300 may, for example, reduce the number of canonical maps from thousands to, for example, 130 to form a first filtered selection.
  • the neighboring areas filter 300 may be configured to reduce the number canonical maps 120 .
  • the number of canonical maps that enters the neighboring areas filter 300 may be greater than the number of canonical maps that exits the neighboring areas filter 300 .
  • the Wi-Fi fingerprint filter 302 forms a second filter.
  • the Wi-Fi fingerprint filter 302 determines a latitude and longitude based on the Wi-Fi fingerprint received as part of the position identifier from the viewing device.
  • the Wi-Fi fingerprint filter 302 compares the latitude and longitude from the Wi-Fi fingerprint filter 302 with latitude and longitude of the canonical maps 120 to determine one or more canonical maps that form a second filtered selection within the first filtered selection established by the neighboring areas filter 300 .
  • the second filtered selection may, for example, reduce the number of canonical maps to 50 canonical maps that form a second selection that does not include any canonical maps based on the first filtered selection outside of the second filtered selection.
  • the first filtered selection thus includes 130 canonical maps and the second filtered selection includes 50 of the 130 canonical maps and does not include the other 80 of the 130 canonical maps.
  • the key frame filter 304 reduces the number of canonical maps further to, for example, five canonical maps.
  • the key frame filter 304 may be a computer vision filter that compares data represented by an image captured by the viewing device with data representing the canonical maps 120 .
  • the key frame filter 304 thus provides, as an output, a third filtered selection that includes only five of the 50 canonical maps identified following the second filtered selection.
  • the map transmitter 122 then transmits the one or more canonical maps based on the third filtered selection to the viewing device.
  • the viewing device receives, in this example and in some embodiments, the five canonical maps and, at 306 , attempts to localize to all five canonical maps.
  • FIGS. 29A-29C illustrate a frame embedding generator 308 that, in some embodiments, may be used within the server 20 (or within one of the viewing devices 12 . 1 and 12 . 2 ) to generate a unique and reduced data representation of an image from an initial size (e.g. 76,800 Bytes) to a final size (e.g. 256 bytes/one bit).
  • the frame embedding generator may be configured to generate a data representation for an image which may be a key frame or a frame.
  • the frame embedding generator 308 may be configured to convert an image at a particular location and orientation into a unique 256 byte string of numbers.
  • An image 320 taken by a viewing device may be processed by the viewing device to detect features 324 in the image 320 (e.g. using a deep sparse feature method) and each feature 322 may be represented by a string of 32 numbers 326 .
  • the frame embedding generator 308 may include a multi-layer perceptron unit 312 and a maximum (max) pool unit 314 .
  • the multi-layer perceptron (MLP) unit 312 may comprise a multi-layer perceptron, which may comprise a neural network.
  • the MLP unit 312 may be configured to perform matrix multiplication.
  • the multi-layer perceptron unit 312 receives the plurality of features 322 of an image 320 and converts each feature to a respective string of 256 numbers. In the present example, there may be 100 features and each feature is represented by a byte size of 256 numbers.
  • a matrix in this example, may be created having 100 horizontal rows and 256 vertical columns. Each row has a series of 256 numbers that vary in magnitude with some being smaller and others being larger.
  • the output of the MLP may be an n ⁇ 256 matrix 328 , where n represents the number of features extracted from the image.
  • the MLP may comprise a neural network, which may have a training phase, e.g. to determine the model parameters, and a use phase.
  • the MLP may be trained as illustrated in FIG. 29D .
  • the input training data may comprise data in sets of three, the set of three comprising 1) a query image, 2) a positive sample, and 3) a negative sample.
  • the query image may be considered the reference image.
  • the positive sample may comprise an image that is similar to the query image. For example, in some embodiments, similar may be having the same object in both the query and positive sample image but viewed from a different angle. In some embodiments, similar may be having the same object in both the query and positive sample image but having the object shifted (e.g.
  • the negative sample may comprise an image that is dissimilar to the query image.
  • dissimilar may be having an object in the query image, but not in the sample image.
  • dissimilar may be having most of an object (e.g. >50%, or >75%) in the query image, but only a small portion of the same object in the negative sample (e.g. ⁇ 10%, 1%).
  • features e.g. sparse features
  • a deep sparse feature (DSF) process may be used to generate the descriptors (e.g., DSF descriptors, feature descriptors) as described in U.S. patent application Ser. No. 16/190,948.
  • DSF descriptors are n ⁇ 32 dimension.
  • the descriptors may then be passed through the model/MLP to create a 256 byte output.
  • the model/MLP may be the MLP described in FIGS. 29A-29C .
  • the 256 byte output from the MLP model may then be sent to a triplet margin loss module (and may only be used during the training phase, not during use phase of the MLP neural network).
  • the triplet margin loss module may be configured to minimize the difference between the 256 byte output from the query image and the 256 byte output from the positive sample, and to maximize the difference between the 256 byte output from the query image and the 256 byte output from the negative sample.
  • the training phase may comprise feeding a plurality of triplet input images into the learning process (e.g. MLP, model) to determine model parameters.
  • the max pool unit 314 analyzes each column to determine a maximum number in the respective column.
  • the max pool unit 314 combines the maximum value of each column of numbers into a global feature string 316 of 256 numbers.
  • the global feature string 316 is a relatively small number that takes up relatively little memory and is easily searchable. It is thus possible to search for images without analyzing each original frame from the camera and it is also cheaper to store 256 bytes instead of complete frames.
  • the key frame filter 304 in FIG. 28 may filter the canonical maps 120 based on the global feature strings 316 of the canonical maps 120 and the global feature string 316 that is based on an image that is captured by the viewing device (e.g. an image that may be part of the local tracking map for a user).
  • Each one of the canonical maps 120 in FIG. 27 thus has one or more global feature strings 316 associated therewith.
  • the global feature strings 316 may be acquired when a viewing device submits images or feature details to the server 20 and the server 20 processes the image or feature details to generate global feature strings 316 for the canonical maps 120 .
  • the server 20 may receive feature details of a live/new/current image captured by a viewing device, the server 20 may generate a global feature string 316 for the live image, and filter the canonical maps 120 based on the live global feature strings 316 .
  • the global feature string may be generated on the local viewing device.
  • the global feature string may be generated remotely, for example on the cloud or the server 20 .
  • the viewing device may take one or more images, execute the process illustrated in FIG. 29C , and attach the global feature string 316 to the local tracking map.
  • the global feature string 316 may be a data representation of a frame or a key frame, in a local tracking map or a canonical map on the local device or the cloud.
  • the server 20 may transmit the filtered canonical maps to the viewing device together with the global feature strings 316 associated with the filtered canonical maps. In some embodiments, when the viewing device localizes its tracking map to the canonical map, it may do so by matching the global feature strings 316 of the local tracking map with the global feature strings of the canonical map.
  • Described herein are methods and apparatus for ranking and merging a plurality of environment maps (e.g., canonical maps) in an X Reality (XR) system.
  • XR X Reality
  • the XR system must know the user's physical surroundings in order to correctly correlate locations of virtual objects in relation to real objects.
  • Information about a user's physical surroundings may be obtained from an environment map for the user's location.
  • An environment map may be created from image and depth information collected with sensors that are part of XR devices worn by users of the XR system.
  • the environment maps may be constructed with information collected by sensors worn by the user of the system at various locations and times.
  • Each environment map may include mapped points, each of which is associated with a three-dimensional (3D) world reconstruction of a real object in the environment.
  • an XR system could provide an enhanced XR experience to multiple users sharing a same world, comprising real and/or virtual content, by enabling efficient sharing of environment maps of the real/physical world collected by multiple users, whether those users are present in the world at the same or different times.
  • Such a system may store multiple maps generated by multiple users and/or the system may store multiple maps generated at different times. For operations that might be performed with a previously generated map, such as localization, for example as described above, substantial processing may be required to identify a relevant environment map of a same world (e.g. same real world location) from all the environment maps collected in an XR system. In some embodiments, such as those described in FIGS.
  • FIGS. 27-29C there may only be a small number of environment maps a device could access, for example for localization. In some embodiments, such as described in relation to FIGS. 27-29C above for example, there may be a large number of environment maps a device could access.
  • the inventors have recognized and appreciated techniques to quickly and accurately rank the relevance of environment maps out of all possible environment maps, such as the universe of all canonical maps 120 in FIG. 28 , for example.
  • a high ranking map may then be selected for further processing, such as to render virtual objects on a user display realistically interacting with the physical world around the user or merging map data collected by that user with stored maps to create larger or more accurate maps.
  • a stored map that is relevant to a task for a user at a location in the physical world may be identified by filtering stored maps based on multiple criteria. Those criteria may indicate comparisons of a tracking map, generated by the wearable device of the user in the location, to candidate environment maps stored in a database, such as described above in regards to FIG. 27-29C , for example.
  • the comparisons may be performed based on metadata associated with the maps, such as a Wi-Fi fingerprint detected by the device generating the map and/or set of BSSID's to which the device connected while forming the map.
  • the comparisons may also be performed based on compressed or uncompressed content of the map.
  • Comparisons based on a compressed representation may be performed, for example, by comparison of vectors computed from map content, such as described in FIGS. 29A-29C , for example. Comparisons based on un-compressed maps may be performed, for example, by localizing the tracking map within the stored map, or vice versa. Multiple comparisons may be performed in an order based on computation time needed to reduce the number of candidate maps for consideration, with comparisons involving less computation being performed earlier in the order than other comparisons requiring more computation.
  • Techniques as described herein may be used together or separately with many types of devices and for many types of scenes, including wearable or portable devices with limited computations resources that provide an XR scene.
  • the techniques may be implemented by one or more services that form a portion of an XR system.
  • FIGS. 30 and 31 illustrate such scenes.
  • an AR system is used as an example of an XR system.
  • FIGS. 32-35 illustrate an exemplary AR system, including one or more processors, memory, sensors and user interfaces that may operate according to the techniques described herein.
  • an outdoor AR scene 354 is depicted wherein a user of an AR technology sees a physical world park-like setting 356 , featuring people, trees, buildings in the background, and a concrete platform 358 .
  • the user of the AR technology also perceives that they “see” a robot statue 357 standing upon the physical world concrete platform 358 , and a cartoon-like avatar character 352 flying by which seems to be a personification of a bumble bee, even though these elements (e.g., the avatar character 352 , and the robot statue 357 ) do not exist in the physical world.
  • Due to the extreme complexity of the human visual perception and nervous system it is challenging to produce an AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or physical world imagery elements.
  • Such an AR scene may be achieved with a system that includes a world reconstruction component, which may build and update a digital representation of the physical world surfaces around the user. This representation may be used to occlude rendering, to place virtual objects, in physics based interactions, and for virtual character path planning and navigation, or for other operations in which information about the physical world is used.
  • FIG. 31 depicts another example of an indoor AR scene 400 , showing exemplary world reconstruction use cases, including visual occlusion 402 , physics-based interactions 404 , and environment reasoning 406 , according to some embodiments.
  • the exemplary scene 400 is a living room having walls, a bookshelf on one side of a wall, a floor lamp at a corner of the room, a floor, a sofa, and coffee table on the floor.
  • the user of the AR technology also perceives virtual objects such as images on the wall behind the sofa, birds flying through the door, a deer peeking out from the book shelf, and a decoration in the form of a windmill placed on the coffee table.
  • the AR technology requires information about not only surfaces of the wall but also objects and surfaces in the room such as lamp shape, which are occluding the images to render the virtual objects correctly.
  • the AR technology requires information about all the objects and surfaces around the room for rendering the birds with realistic physics to avoid the objects and surfaces or bounce off them if the birds collide.
  • the AR technology requires information about the surfaces such as the floor or coffee table to compute where to place the deer.
  • the system may identify that is an object separate from the table and may determine that it is movable, whereas corners of shelves or corners of the wall may be determined to be stationary. Such a distinction may be used in determinations as to which portions of the scene are used or updated in each of various operations.
  • a scene may be presented to the user via a system that includes multiple components, including a user interface that can stimulate one or more user senses, including sight, sound, and/or touch.
  • the system may include one or more sensors that may measure parameters of the physical portions of the scene, including position and/or motion of the user within the physical portions of the scene.
  • the system may include one or more computing devices, with associated computer hardware, such as memory. These components may be integrated into a single device or may be distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.
  • FIG. 32 depicts an AR system 502 configured to provide an experience of AR contents interacting with a physical world 506 , according to some embodiments.
  • the AR system 502 may include a display 508 .
  • the display 508 may be worn by the user as part of a headset such that a user may wear the display over their eyes like a pair of goggles or glasses.
  • At least a portion of the display may be transparent such that a user may observe a see-through reality 510 .
  • the see-through reality 510 may correspond to portions of the physical world 506 that are within a present viewpoint of the AR system 502 , which may correspond to the viewpoint of the user in the case that the user is wearing a headset incorporating both the display and sensors of the AR system to acquire information about the physical world.
  • AR contents may also be presented on the display 508 , overlaid on the see-through reality 510 .
  • the AR system 502 may include sensors 522 configured to capture information about the physical world 506 .
  • the sensors 522 may include one or more depth sensors that output depth maps 512 .
  • Each depth map 512 may have multiple pixels, each of which may represent a distance to a surface in the physical world 506 in a particular direction relative to the depth sensor.
  • Raw depth data may come from a depth sensor to create a depth map.
  • Such depth maps may be updated as fast as the depth sensor can form a new image, which may be hundreds or thousands of times per second. However, that data may be noisy and incomplete, and have holes shown as black pixels on the illustrated depth map.
  • the system may include other sensors, such as image sensors.
  • the image sensors may acquire monocular or stereoscopic information that may be processed to represent the physical world in other ways.
  • the images may be processed in world reconstruction component 516 to create a mesh, representing connected portions of objects in the physical world. Metadata about such objects, including for example, color and surface texture, may similarly be acquired with the sensors and stored as part of the world reconstruction.
  • the system may also acquire information about the headpose of the user with respect to the physical world.
  • sensors 522 may include inertial measurement units that may be used to compute and/or determine a headpose 514 .
  • a headpose 514 for a depth map may indicate a present viewpoint of a sensor capturing the depth map with six degrees of freedom (6 DoF), for example, but the headpose 514 may be used for other purposes, such as to relate image information to a particular portion of the physical world or to relate the position of the display worn on the user's head to the physical world.
  • the headpose information may be derived in other ways than from an IMU, such as from analyzing objects in an image.
  • the world reconstruction component 516 may receive the depth maps 512 and headposes 514 , and any other data from the sensors, and integrate that data into a reconstruction 518 .
  • the reconstruction 518 may be more complete and less noisy than the sensor data.
  • the world reconstruction component 516 may update the reconstruction 518 using spatial and temporal averaging of the sensor data from multiple viewpoints over time.
  • the reconstruction 518 may include representations of the physical world in one or more data formats including, for example, voxels, meshes, planes, etc.
  • the different formats may represent alternative representations of the same portions of the physical world or may represent different portions of the physical world.
  • portions of the physical world are presented as a global surface; on the right side of the reconstruction 518 , portions of the physical world are presented as meshes.
  • the reconstruction 518 may be used as or processed into a map providing information about the portion of the physical world represented by the reconstruction.
  • the map may indicate locations of key points and/or structures, such as edges and/or planes. Additionally, the map may include information about a pose of a user from which the points or structures are observable to a user, which in some embodiments may be called a key frame or a key rig.
  • the map represents the portion of the physical world in which a user of a single, wearable device is present.
  • the map may include metadata.
  • the metadata for example, may indicate time of capture of the sensor information used to form the map.
  • Metadata alternatively or additionally may indicate location of the sensors at the time of capture of information used to form the map. Location may be expressed directly, such as with information from a GPS chip, or indirectly, such as with a Wi-Fi signature indicating strength of signals received from one or more wireless access points while the sensor data was being collected and/or with the BSSID's of wireless access points to which the user device connected while the sensor data was collected.
  • the reconstruction 518 may be used for AR functions, such as producing a surface representation of the physical world for occlusion processing or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of the reconstruction 518 may be used, for example, by a component 520 that produces a changing global surface representation in world coordinates, which may be used by other components.
  • the AR content may be generated based on this information, such as by AR applications 504 .
  • An AR application 504 may be a game program, for example, that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interactions, and environment reasoning. It may perform these functions by querying data in different formats from the reconstruction 518 produced by the world reconstruction component 516 .
  • component 520 may be configured to output updates when a representation in a region of interest of the physical world changes. That region of interest, for example, may be set to approximate a portion of the physical world in the vicinity of the user of the system, such as the portion within the view field of the user, or is projected (predicted/determined) to come within the view field of the user.
  • the AR applications 504 may use this information to generate and update the AR contents.
  • the virtual portion of the AR contents may be presented on the display 508 in combination with the see-through reality 510 , creating a realistic user experience.
  • FIG. 33 illustrates an example of wearable display system 580 (hereinafter referred to as “system 580 ”).
  • the system 580 includes a head mounted display device 562 (hereinafter referred to as “display device 562 ”), and various mechanical and electronic modules and systems to support the functioning of the display device 562 .
  • the display device 562 may be coupled to a frame 564 , which is wearable by a display system user or viewer 560 (hereinafter referred to as “user 560 ”) and configured to position the display device 562 in front of the eyes of the user 560 .
  • the display device 562 may be a sequential display.
  • the display device 562 may be monocular or binocular.
  • the display device 562 may be an example of the display 508 in FIG. 32 .
  • a speaker 566 is coupled to the frame 564 and positioned proximate an ear canal of the user 560 .
  • another speaker not shown, is positioned adjacent another ear canal of the user 560 to provide for stereo/shapeable sound control.
  • the display device 562 is operatively coupled, such as by a wired lead or wireless connectivity 568 , to a local data processing module 570 which may be mounted in a variety of configurations, such as fixedly attached to the frame 564 , fixedly attached to a helmet or hat worn by the user 560 , embedded in headphones, or otherwise removably attached to the user 560 (e.g., in a backpack-style configuration, in a belt-coupling style configuration).
  • the local data processing module 570 may include a processor, as well as digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data.
  • the data include data a) captured from sensors (which may be, e.g., operatively coupled to the frame 564 ) or otherwise attached to the user 560 , such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros; and/or b) acquired and/or processed using remote processing module 572 and/or remote data repository 574 , possibly for passage to the display device 562 after such processing or retrieval.
  • the local data processing module 570 may be operatively coupled by communication links 576 , 578 , such as via a wired or wireless communication links, to the remote processing module 572 and remote data repository 574 , respectively, such that these remote modules 572 , 574 are operatively coupled to each other and available as resources to the local data processing module 570 .
  • the world reconstruction component 516 in FIG. 32 may be at least partially implemented in the local data processing module 570 .
  • the local data processing module 570 may be configured to execute computer executable instructions to generate the physical world representations based at least in part on at least a portion of the data.
  • processing may be distributed across local and remote processors.
  • local processing may be used to construct a map on a user device (e.g. tracking map) based on sensor data collected with sensors on that user's device. Such a map may be used by applications on that user's device.
  • previously created environment maps e.g. world maps and/or canonical maps
  • the environment map might be loaded on the user device to allow the user device to render virtual content without a delay associated with scanning a location and building a tracking map from sensor data acquired during the scan.
  • the tracking map may be merged with previously stored environment maps to extend or improve the quality of those environment maps.
  • the processing to determine whether a suitable previously created environment map is available and/or to merge a tracking map with one or more stored environment maps may be done in local data processing module 570 or remote processing module 572 .
  • the local data processing module 570 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process data and/or image information.
  • the local data processing module 570 may include a single processor (e.g., a single-core or multi-core ARM processor), which would limit the local data processing module 570 's compute budget but enable a more miniature device.
  • the world reconstruction component 516 may use a compute budget less than a single Advanced RISC Machine (ARM) core to generate physical world representations in real-time on a non-predefined space such that the remaining compute budget of the single ARM core can be accessed for other uses such as, for example, extracting meshes.
  • ARM Advanced RISC Machine
  • the remote data repository 574 may include a digital data storage facility, which may be available through the Internet or other networking configuration in a “cloud” resource configuration.
  • all data is stored and all computations are performed in the local data processing module 570 , allowing fully autonomous use from a remote module.
  • all data is stored and all or most computations are performed in the remote data repository 574 , allowing for a smaller device.
  • a world reconstruction for example, may be stored in whole or in part in this repository 574 .
  • data may be shared by multiple users of an augmented reality system.
  • user devices may upload their tracking maps which may be stored in a database of environment maps.
  • the tracking map upload occurs at the end of a session.
  • the tracking map uploads may occur continuously, semi-continuously, intermittently, at a pre-defined time, after a pre-defined period from the previous upload, or when triggered by an event.
  • a tracking map uploaded by any user device may be used to expand or improve a previously stored environment map, whether based on data from that user device or any other user device.
  • an environment map downloaded to a user device may be based on data from that user device or any other user device. In this way, high quality environment maps may be readily available to users to improve their experiences with the AR system.
  • the local data processing module 570 is operatively coupled to a battery 582 .
  • the battery 582 is a removable power source, such as over the counter batteries.
  • the battery 582 is a lithium-ion battery.
  • the battery 582 includes both an internal lithium-ion battery chargeable by the user 560 during non-operation times of the system 580 and removable batteries such that the user 560 may operate the system 580 for longer periods of time without having to be tethered to a power source to charge the lithium-ion battery or having to shut the system 580 off to replace batteries.
  • FIG. 34A illustrates a user 530 wearing an AR display system rendering AR content as the user 530 moves through a physical world environment 532 (hereinafter referred to as “environment 532 ”).
  • the user 530 positions the AR display system at positions 534 , and the AR display system records ambient information of a passable world (e.g., a digital representation of the real objects in the physical world that can be stored and updated with changes to the real objects in the physical world) relative to the positions 534 such as pose relation to mapped features or directional audio inputs.
  • the positions 534 are aggregated to data inputs 536 and processed at least by a passable world module 538 , which may be implemented, for example, by processing on a remote processing module 572 of FIG. 33 .
  • the passable world module 538 may include the world reconstruction component 516 .
  • FIG. 35B depicts component architecture of a PW, according to some embodiments.
  • the passable world module 538 determines, at least in part, where and how AR content 540 can be placed in the physical world as determined from the data inputs 536 .
  • the AR content is “placed” in the physical world by presenting via the user interface both a representation of the physical world and the AR content, with the AR content rendered as if it were interacting with objects in the physical world and the objects in the physical world presented as if the AR content were, when appropriate, obscuring the user's view of those objects.
  • the AR content may be placed by appropriately selecting portions of a fixed element 542 (e.g., a table) from a reconstruction (e.g., the reconstruction 518 ) to determine the shape and position of the AR content 540 .
  • the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table.
  • the AR content may be placed within structures in a field of view 544 , which may be a present field of view or an estimated future field of view.
  • the AR content may be persisted relative to a mapped (e.g. mesh) model 546 of the physical world.
  • the fixed element 542 serves as a proxy (e.g. digital copy) for any fixed element within the physical world which may be stored in the passable world module 538 so that the user 530 can perceive content on the fixed element 542 without the system having to map to the fixed element 542 each time the user 530 sees it.
  • the fixed element 542 may, therefore, be a mapped mesh model from a previous modeling session or determined from a separate user but nonetheless stored on the passable world module 538 for future reference by a plurality of users. Therefore, the passable world module 538 may recognize the environment 532 from a previously mapped environment and display AR content without a device of the user 530 mapping all or part of the environment 532 first, saving computation process and cycles and avoiding latency of any rendered AR content.
  • the mapped mesh model 546 of the physical world may be created by the AR display system and appropriate surfaces and metrics for interacting and displaying the AR content 540 can be mapped and stored in the passable world module 538 for future retrieval by the user 530 or other users without the need to completely or partially re-map or model.
  • the data inputs 536 are inputs such as geolocation, user identification, and current activity to indicate to the passable world module 538 which fixed element 542 of one or more fixed elements are available, which AR content 540 has last been placed on the fixed element 542 , and whether to display that same content (such AR content being “persistent” content regardless of user viewing a particular passable world model).
  • the passable world module 538 may be updated from time to time to account for the possibility of changes in the physical world.
  • the model of fixed objects may be updated with a very low frequency.
  • Other objects in the physical world may be moving or otherwise not regarded as fixed (e.g. kitchen chairs).
  • the AR system may update the position of these non-fixed objects with a much higher frequency than is used to update fixed objects.
  • an AR system may draw information from multiple sensors, including one or more image sensors.
  • FIG. 34B is a schematic illustration of a viewing optics assembly 548 and attendant components.
  • two eye tracking cameras 550 directed toward user eyes 549 , detect metrics of the user eyes 549 , such as eye shape, eyelid occlusion, pupil direction and glint on the user eyes 549 .
  • one of the sensors may be a depth sensor 551 , such as a time of flight sensor, emitting signals to the world and detecting reflections of those signals from nearby objects to determine distance to given objects.
  • a depth sensor may quickly determine whether objects have entered the field of view of the user, either as a result of motion of those objects or a change of pose of the user.
  • information about the position of objects in the field of view of the user may alternatively or additionally be collected with other sensors. Depth information, for example, may be obtained from stereoscopic visual image sensors or plenoptic sensors.
  • world cameras 552 record a greater-than-peripheral view to map the environment 532 and detect inputs that may affect AR content.
  • the world camera 552 and/or camera 553 may be grayscale and/or color image sensors, which may output grayscale and/or color image frames at fixed time intervals. Camera 553 may further capture physical world images within a field of view of the user at a specific time. Pixels of a frame-based image sensor may be sampled repetitively even if their values are unchanged.
  • Each of the world cameras 552 , the camera 553 and the depth sensor 551 have respective fields of view of 554 , 555 , and 556 to collect data from and record a physical world scene, such as the physical world environment 532 depicted in FIG. 34A .
  • Inertial measurement units 557 may determine movement and orientation of the viewing optics assembly 548 .
  • each component is operatively coupled to at least one other component.
  • the depth sensor 551 is operatively coupled to the eye tracking cameras 550 as a confirmation of measured accommodation against actual distance the user eyes 549 are looking at.
  • a viewing optics assembly 548 may include some of the components illustrated in FIG. 34B and may include components instead of or in addition to the components illustrated.
  • a viewing optics assembly 548 may include two world camera 552 instead of four. Alternatively or additionally, cameras 552 and 553 need not capture a visible light image of their full field of view.
  • a viewing optics assembly 548 may include other types of components.
  • a viewing optics assembly 548 may include one or more dynamic vision sensor (DVS), whose pixels may respond asynchronously to relative changes in light intensity exceeding a threshold.
  • DVDS dynamic vision sensor
  • a viewing optics assembly 548 may not include the depth sensor 551 based on time of flight information.
  • a viewing optics assembly 548 may include one or more plenoptic cameras, whose pixels may capture light intensity and an angle of the incoming light, from which depth information can be determined.
  • a plenoptic camera may include an image sensor overlaid with a transmissive diffraction mask (TDM).
  • TDM transmissive diffraction mask
  • a plenoptic camera may include an image sensor containing angle-sensitive pixels and/or phase-detection auto-focus pixels (PDAF) and/or micro-lens array (MLA). Such a sensor may serve as a source of depth information instead of or in addition to depth sensor 551 .
  • PDAF phase-detection auto-focus pixels
  • MSA micro-lens array
  • a viewing optics assembly 548 may include components with any suitable configuration, which may be set to provide the user with the largest field of view practical for a particular set of components. For example, if a viewing optics assembly 548 has one world camera 552 , the world camera may be placed in a center region of the viewing optics assembly instead of at a side.
  • Information from the sensors in viewing optics assembly 548 may be coupled to one or more of processors in the system.
  • the processors may generate data that may be rendered so as to cause the user to perceive virtual content interacting with objects in the physical world. That rendering may be implemented in any suitable way, including generating image data that depicts both physical and virtual objects.
  • physical and virtual content may be depicted in one scene by modulating the opacity of a display device that a user looks through at the physical world. The opacity may be controlled so as to create the appearance of the virtual object and also to block the user from seeing objects in the physical world that are occluded by the virtual objects.
  • the image data may only include virtual content that may be modified such that the virtual content is perceived by a user as realistically interacting with the physical world (e.g. clip content to account for occlusions), when viewed through the user interface.
  • a model of the physical world is required so that characteristics of the virtual objects, which can be impacted by physical objects, including the shape, position, motion and visibility of the virtual object, can be correctly computed.
  • the model may include the reconstruction of a physical world, for example, the reconstruction 518 .
  • That model may be created from data collected from sensors on a wearable device of the user. Though, in some embodiments, the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all of the users (and which may be “in the cloud”).
  • the model may be created, at least in part, by a world reconstruction system, for example, the world reconstruction component 516 of FIG. 32 depicted in more detail in FIG. 35A .
  • the world reconstruction component 516 may include a perception module 660 that may generate, update, and store representations for a portion of the physical world.
  • the perception module 660 may represent the portion of the physical world within a reconstruction range of the sensors as multiple voxels. Each voxel may correspond to a 3D cube of a predetermined volume in the physical world, and include surface information, indicating whether there is a surface in the volume represented by the voxel.
  • Voxels may be assigned values indicating whether their corresponding volumes have been determined to include surfaces of physical objects, determined to be empty or have not yet been measured with a sensor and so their value is unknown. It should be appreciated that values indicating that voxels that are determined to be empty or unknown need not be explicitly stored, as the values of voxels may be stored in computer memory in any suitable way, including storing no information for voxels that are determined to be empty or unknown.
  • the perception module 660 may identify and output indications of changes in a region around a user of an AR system. Indications of such changes may trigger updates to volumetric data stored as part of the persisted world, or trigger other functions, such as triggering components 604 that generate AR content to update the AR content.
  • the perception module 660 may identify changes based on a signed distance function (SDF) model.
  • the perception module 660 may be configured to receive sensor data such as, for example, depth maps 660 a and headposes 660 b , and then fuse the sensor data into a SDF model 660 c .
  • Depth maps 660 a may provide SDF information directly, and images may be processed to arrive at SDF information.
  • the SDF information represents distance from the sensors used to capture that information. As those sensors may be part of a wearable unit, the SDF information may represent the physical world from the perspective of wearable unit and therefore the perspective of the user.
  • the headposes 660 b may enable the SDF information to be related to a voxel in the physical world.
  • the perception module 660 may generate, update, and store representations for the portion of the physical world that is within a perception range.
  • the perception range may be determined based, at least in part, on a sensor's reconstruction range, which may be determined based, at least in part, on the limits of a sensor's observation range.
  • an active depth sensor that operates using active IR pulses may operate reliably over a range of distances, creating the observation range of the sensor, which may be from a few centimeters or tens of centimeters to a few meters.
  • the world reconstruction component 516 may include additional modules that may interact with the perception module 660 .
  • a persisted world module 662 may receive representations for the physical world based on data acquired by the perception module 660 .
  • the persisted world module 662 also may include various formats of representations of the physical world.
  • representations of the physical world may be images, features from images, descriptors of features, key frames/keyrigs, persistent poses, PCFs, and/or global feature strings.
  • volumetric metadata 662 b such as voxels may be stored as well as meshes 662 c and planes 662 d .
  • other information such as depth maps could be saved.
  • the perception module 660 may include modules that generate representations for the physical world in various formats including, for example, meshes 660 d , planes and semantics 660 e . These modules may generate representations based on data within the perception range of one or more sensors at the time the representation is generated as well as data captured at prior times and information in the persisted world module 662 . In some embodiments, these components may operate on depth information captured with a depth sensor. However, the AR system may include vision sensors and may generate such representations by analyzing monocular or binocular vision information.
  • these modules may operate on regions of the physical world. Those modules may be triggered to update a subregion of the physical world, when the perception module 660 detects a change in the physical world in that subregion. Such a change, for example, may be detected by detecting a new surface in the SDF model 660 c or other criteria, such as changing the value of a sufficient number of voxels representing the subregion.
  • the world reconstruction component 516 may include components 664 that may receive representations of the physical world from the perception module 660 . Information about the physical world may be pulled by these components according to, for example, a use request from an application. In some embodiments, information may be pushed to the use components, such as via an indication of a change in a pre-identified region or a change of the physical world representation within the perception range.
  • the components 664 may include, for example, game programs and other components that perform processing for visual occlusion, physics-based interactions, and environment reasoning.
  • the perception module 660 may send representations for the physical world in one or more formats. For example, when the component 664 indicates that the use is for visual occlusion or physics-based interactions, the perception module 660 may send a representation of surfaces. When the component 664 indicates that the use is for environmental reasoning, the perception module 660 may send meshes, planes and semantics of the physical world.
  • the perception module 660 may include components that format information to provide the component 664 .
  • An example of such a component may be raycasting component 660 f .
  • a use component e.g., component 664
  • Raycasting component 660 f may select from one or more representations of the physical world data within a field of view from that point of view.
  • the perception module 660 may process data to create 3D representations of portions of the physical world.
  • Data to be processed may be reduced by culling parts of a 3D reconstruction volume based at last in part on a camera frustum and/or depth image, extracting and persisting plane data, capturing, persisting and updating 3D reconstruction data in blocks that allow local update while maintaining neighbor consistency, providing occlusion data to applications generating such scenes, where the occlusion data is derived from a combination of one or more depth data sources, and/or performing a multi-stage mesh simplification.
  • a world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world.
  • the poses of the sensors e.g., position and orientation
  • the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
  • the reconstruction may be more complete and less noisy than the original sensor data by using spatial and temporal averaging (i.e. averaging data from multiple viewpoints over time), or any other suitable method.
  • the reconstruction may contain data of different levels of sophistication including, for example, raw data such as live depth data, fused volumetric data such as voxels, and computed data such as meshes.
  • a passable world may include environment maps created based, at least in part, on data reconstructed by world reconstruction systems of AR devices worn by multiple users.
  • individual AR devices such as wearable devices described above in connection with FIG. 33
  • TMs tracking maps
  • a tracking map may include a sparse map of headposes of the AR device scanning an environment as well as information about objects detected within that environment.
  • the AR device may send the tracking map to a cloud to, for example, merge with environment maps selected from environment maps stored in the cloud.
  • the selected environment maps may be sent from the cloud to the AR device for merging.
  • the environment maps may provide information about the portions of the physical world represented by the reconstruction data processed to create respective maps.
  • FIG. 36 depicts an exemplary environment map 700 , according to some embodiments.
  • the environment map 700 may include a floor plan 706 of reconstructed physical objects in a corresponding physical world, represented by points 702 .
  • the environment map 700 may also include tracking routes 710 , along which a device collecting data on points 702 traveled.
  • the tracking routes 710 may include keyrigs 704 , which may represent a location where an AR device uploaded reconstruction data and associated metadata, such as headposes and/or image data.
  • a keyrig may also be called a key frame.
  • the environment map includes two tracking routes 710 a and 710 b , which may indicate that the environment map 700 is created by merging two tracking maps: one is created along the tracking route 710 a ; the other is created along the tracking route 710 b .
  • the two tracking maps may be created by two different AR devices at the same time, two different AR devices at different times, or the same AR device at different times.
  • the two tracking maps may be created by one AR device and uploaded to the cloud through two network access points, each of which may be accessible within a designated area range.
  • the environment maps may have any of multiple formats depending on, for example, the storage locations of an environment map including, for example, local storage of AR devices and remote storage.
  • a map in remote storage may have higher resolution than a map in local storage.
  • the map may be down sampled or otherwise converted to an appropriate format.
  • a slice or portion of a high resolution map from remote storage is sent to local storage, where the slice or portion is not down sampled.
  • a database of the environment maps may be updated when a new tracking map is created.
  • updating may include efficiently selecting one or more environment maps stored in the database relevant to the new tracking map.
  • the selected one or more environment maps may be ranked by relevance and one or more of the highest ranking maps may be selected for processing to merge higher ranked selected environment maps with the new tracking map to create one or more updated environment maps.
  • FIG. 37 depicts an AR system 800 configured to rank and merge one or more environment maps, according to some embodiments.
  • the AR system may include a passable world model 802 of an AR device.
  • Information to populate the passable world model 802 may come from sensors on the AR device, which may include computer executable instructions stored in a processor 804 (e.g., a local data processing module 570 ), which may perform some or all of the processing to convert sensor data into a map.
  • a map may be a tracking map, as it can be built as sensor data is collected as the AR device operates in a region.
  • area attributes may be supplied so as to indicate area that the tracking map represents.
  • area attributes may be a geographic location identifier, such as coordinates presented as latitude and longitude or an ID used by the AR system to represent a location. Alternatively or additionally, the area attributes may be measured characteristics that have a high likelihood of being unique for that area. The area attributes, for example, may be derived from parameters of wireless networks detected in the area. In some embodiments, the area attribute may be associated with a unique address of an access-point the AR system is nearby and/or connected to. For example, the area attribute may be associated with a MAC address or basic service set identifiers (BSSIDs) of a 5G base station/router, a Wi-Fi router, and the like.
  • BSSIDs basic service set identifiers
  • a map rank portion 806 receives tracking maps from the device PW 802 and communicates with a map database 808 to select and rank environment maps selected from the map database 808 . Higher ranked, selected maps are sent to a map merge portion 810 .
  • the map merge portion 810 may perform merge processing on the maps sent from the map rank portion 806 .
  • Merge processing may entail merging the tracking map with some or all of the ranked maps and transmitting the new, merged maps to a passable world model 812 .
  • Passable world model 812 may be a cloud model, which may be shared by multiple AR devices. Passable world model 812 may store or otherwise have access to those environment maps in map database 808 .
  • the prior version of that map may be deleted so as to remove out of date maps from the database.
  • the prior version of that map may be archived enabling retrieving/viewing prior versions of an environment.
  • permissions may be set such that only AR systems having certain read/write access may trigger prior versions of maps being deleted/archived.
  • the map rank portion 806 also may be used in supplying environment maps to an AR device.
  • the AR device may send a message requesting an environment map for its current location, and map rank portion 806 may be used to select and rank environment maps relevant to the requesting device.
  • the AR system 800 may include a downsample portion 814 configured to receive the merged maps from the cloud PW 812 .
  • the received merged maps from the cloud PW 812 may be in a storage format for the cloud, which may include dense information.
  • the downsample portion 814 may be configured to downsample the cloud format maps to a format suitable for storage on AR devices.
  • the device format maps may be sparse to accommodate the limited local computing power and storage space of AR devices.
  • U.S. provisional patent application No. 62/812,935 entitled “Merging a Plurality of Individually Mapped Environments” describes exemplary methods and apparatus of merging environment maps.
  • a copy of the filed version of U.S. App. No. 62/812,935 is attached as an appendix and is hereby incorporated by reference in its entirety for all purposes. Techniques as described in that application may be used for merging environment maps.
  • FIG. 38 is flow chart illustrating a method 900 of selecting one or more ranked environment maps, according to some embodiments.
  • the ranking is performed for a user's AR device that is creating a tracking map. Accordingly, the tracking map is available for use in ranking environment maps. In embodiments in which the tracking map is not available, some or all of portions of the selection and ranking of environment maps that do not expressly rely on the tracking map may be used.
  • the method 900 may start at Act 902 , where a set of maps from a database of environment maps (sometimes referred to as canonical maps) that are in the neighborhood of the location where the tracking map was formed may be accessed and then filtered for ranking. Additionally, at Act 902 , at least one area attribute for the area in which the user's AR device is operating is determined. In scenarios in which the user's AR device is constructing a tracking map, the area attributes may correspond to the area over which the tracking map was created. As a specific example, the area attributes may be computed based on received signals from access points to computer networks while the AR device was computing the tracking map.
  • FIG. 39 depicts an exemplary map rank portion 806 of the AR system 800 , according to some embodiments.
  • the map rank portion 806 may be a cloud computing environment, as it may include portions of AR devices and portions of a remote computing system such as a cloud.
  • the map rank portion 806 may be configured to perform at least a portion of the method 900 .
  • FIG. 40A depicts an example of area attributes AA 1 -AA 8 of a tracking map (TM) 1102 and environment maps CM 1 -CM 4 in a database, according to some embodiments.
  • an environment map may be associated to multiple area attributes.
  • the area attributes AA 1 -AA 8 may include parameters of wireless networks detected by the AR device computing the tracking map 1102 , for example, basic service set identifiers (BSSIDs) of networks to which the AR device are connected and/or the strength of the received signals of the access points to the wireless networks through, for example, a network tower 1104 .
  • the parameters of the wireless networks may comply with protocols including Wi-Fi and 5G NR.
  • the area attributes are a fingerprint of the area in which the user AR device collected sensor data to form the tracking map.
  • FIG. 40B depicts an example of the determined geographic location 1106 of the tracking map 1102 , according to some embodiments.
  • the determined geographic location 1106 includes a centroid point 1110 and an area 1108 circling around the centroid point.
  • a determined geographic location may have any suitable formats including, for example, different area shapes.
  • the geographic location is determined from area attributes using a database relating area attributes to geographic locations. Databases are commercially available, for example, databases that relate Wi-Fi fingerprints to locations expressed as latitude and longitude and may be used for this operation.
  • a map database containing environment maps may also include location data for those maps, including latitude and longitude covered by the maps. Processing at Act 902 may entail selecting from that database a set of environment maps that covers the same latitude and longitude determined for the area attributes of the tracking map.
  • Act 904 is a first filtering of the set of environment maps accessed in Act 902 .
  • environment maps are retained in the set based on proximity to the geolocation of the tracking map. This filtering step may be performed by comparing the latitude and longitude associated with the tracking map and the environment maps in the set.
  • FIG. 41 depicts an example of Act 904 , according to some embodiments.
  • Each area attribute may have a corresponding geographic location 1202 .
  • the set of environment maps may include the environment maps with at least one area attribute that has a geographic location overlapping with the determined geographic location of the tracking map.
  • the set of identified environment maps includes environment maps CM 1 , CM 2 , and CM 4 , each of which has at least one area attribute that has a geographic location overlapping with the determined geographic location of the tracking map 1102 .
  • the environment map CM 3 associated with the area attribute AA 6 is not included in the set because it is outside the determined geographic location of the tracking map.
  • the method 900 may include filtering (Act 906 ) the set of environment maps based on similarity of one or more identifiers of network access points associated with the tracking map and the environment maps of the set of environment maps.
  • a device collecting sensor data to generate the map may be connected to a network through a network access point, such as through Wi-Fi or similar wireless communication protocol.
  • the access points may be identified by BSSID.
  • the user device may connect to multiple different access points as it moves through an area collecting data to form a map.
  • the devices may have connected through different access points, so there may be multiple access points used in forming the map for that reason too. Accordingly, there may be multiple access points associated with a map, and the set of access points may be an indication of location of the map.
  • filtering the set of environment maps based on similarity of the one or more identifiers of the network access points may include retaining in the set of environment maps environment maps with the highest Jaccard similarity to the at least one area attribute of the tracking map based on the one or more identifiers of network access points.
  • FIG. 42 depicts an example of Act 906 , according to some embodiments.
  • a network identifier associated with the area attribute AA 7 may be determined as the identifier for the tracking map 1102 .
  • the set of environment maps after Act 906 includes environment map CM 2 , which may have area attributes within higher Jaccard similarity to AA 7 , and environment map CM 4 , which also include the area attributes AA 7 .
  • the environment map CM 1 is not included in the set because it has the lowest Jaccard similarity to AA 7 .
  • Processing at Acts 902 - 906 may be performed based on metadata associated with maps and without actually accessing the content of the maps stored in a map database. Other processing may involve accessing the content of the maps.
  • Act 908 indicates accessing the environment maps remaining in the subset after filtering based on metadata. It should be appreciated that this act may be performed either earlier or later in the process, if subsequent operations can be performed with accessed content.
  • the method 900 may include filtering (Act 910 ) the set of environment maps based on similarity of metrics representing content of the tracking map and the environment maps of the set of environment maps.
  • the metrics representing content of the tracking map and the environment maps may include vectors of values computed from the contents of the maps.
  • the metrics may be computed from the maps retrieved at Act 908 or may be pre-computed and stored as metadata associated with those maps.
  • filtering the set of environment maps based on similarity of metrics representing content of the tracking map and the environment maps of the set of environment maps may include retaining in the set of environment maps environment maps with the smallest vector distance between a vector of characteristics of the tracking map and vectors representing environment maps in the set of environment maps.
  • the method 900 may include further filtering (Act 912 ) the set of environment maps based on degree of match between a portion of the tracking map and portions of the environment maps of the set of environment maps.
  • the degree of match may be determined as a part of a localization process.
  • localization may be performed by identifying critical points in the tracking map and the environment map that are sufficiently similar as they could represent the same portion of the physical world.
  • the critical points may be features, feature descriptors, key frames/key rigs, persistent poses, and/or PCFs.
  • the set of critical points in the tracking map might then be aligned to produce a best fit with the set of critical points in the environment map.
  • a mean square distance between the corresponding critical points might be computed and, if below a threshold for a particular region of the tracking map, used as an indication that the tracking map and the environment map represent the same region of the physical world.
  • filtering the set of environment maps based on degree of match between a portion of the tracking map and portions of the environment maps of the set of environment maps may include computing a volume of a physical world represented by the tracking map that is also represented in an environment map of the set of environment maps, and retaining in the set of environment maps environment maps with a larger computed volume than environment maps filtered out of the set.
  • FIG. 43 depicts an example of Act 912 , according to some embodiments.
  • the set of environment maps after Act 912 includes environment map CM 4 , which has an area 1402 matched with an area of the tracking map 1102 .
  • the environment map CM 1 is not included in the set because it has no area matched with an area of the tracking map 1102 .
  • the set of environment maps may be filtered in the order of Act 906 , Act 910 , and Act 912 . In some embodiments, the set of environment maps may be filtered based on Act 906 , Act 910 , and Act 912 , which may be performed in an order based on processing required to perform the filtering, from lowest to highest.
  • the method 900 may include loading (Act 914 ) the set of environment maps and data.
  • a user database stores area identities indicating areas that AR devices were used in.
  • the area identities may be area attributes, which may include parameters of wireless networks detected by the AR devices when in use.
  • a map database stores multiple environment maps constructed from data supplied by the AR devices and associated metadata.
  • the associated metadata may include area identities derived from the area identities of the AR devices that supplied data from which the environment maps were constructed.
  • An AR device may send a message to a PW module indicating new tracking map is created or being created. The PW module computes area identifiers for the AR device and updates the user database based on the received parameters and/or the computed area identifiers.
  • the PW module may also determine area identifiers associated with the AR device requesting the environment maps, identify sets of environment maps from the map database based on the area identifiers, filter the sets of environment maps, and transmit the filtered sets of environment maps to the AR devices.
  • the PW module may filter the sets of environment maps based on one or more criteria including, for example, a geographic location of the tracking map, similarity of one or more identifiers of network access points associated with the tracking map and the environment maps of the set of environment maps, similarity of metrics representing contents of the tracking map and the environment maps of the set of environment maps, and degree of match between a portion of the tracking map and portions of the environment maps of the set of environment maps.
  • embodiments are described in connection with an augmented (AR) environment. It should be appreciated that some or all of the techniques described herein may be applied in an MR environment or more generally in other XR environments, and in VR environments.
  • embodiments are described in connection with devices, such as wearable devices. It should be appreciated that some or all of the techniques described herein may be implemented via networks (such as cloud), discrete applications, and/or any suitable combinations of devices, networks, and discrete applications.
  • networks such as cloud
  • discrete applications and/or any suitable combinations of devices, networks, and discrete applications.
  • FIG. 38 provides examples of criteria that may be used to filter candidate maps to yield a set of high ranking maps. Other criteria may be used instead of or in addition to the described criteria. For example, if multiple candidate maps have similar values of a metric used for filtering out less desirable maps, characteristics of the candidate maps may be used to determine which maps are retained as candidate maps or filtered out. For example, larger or more dense candidate maps may be prioritized over smaller candidate maps.
  • FIGS. 27-28 may describe all or part of the systems and methods described in FIGS. 37-39 .
  • FIGS. 51-53 describe examples of determining a map to be used based at least partly on deep keyframe (DKF) analysis, according to some embodiments.
  • the DKF analysis may be part of the process to determine an environment map to be used by the XR device and/or to determine a current headpose for the user (e.g., the position and/or orientation of the user's head) based on a TM generated by the XR device.
  • DKF analysis employs suitable deep learning technique(s) to predict or otherwise determine a suitable environment map to be used, by providing keyframe(s) of the TM as input to a deep learning architecture, such as a neural network.
  • the DKF analysis is used as part of the process to rank environment maps, also referred to as canonical maps (CMs).
  • CMs can be ranked based on the degree of similarity of DKF descriptors between a TM and each CM being analyzed as a candidate.
  • the DKF analysis can be part of a hierarchical (e.g., multi-level) analysis to select a CM, in which geolocation filtering is performed (e.g., first) as a top level analysis, Wi-Fi BSSID filtering is performed as a next level, and DKF analysis is performed as a third (e.g., final) level to rank CMs.
  • a highest ranked CM, or set of CMs may be provided to the XR device to be used and/or further analyzed.
  • the ranking analysis may be performed in the cloud (e.g., using the SPIFF analysis module(s) as shown in the example of FIGS. 51-53 ).
  • the set of candidate CMs can be filtered (e.g., reduced) until a single map (or subset of maps) remains after the bottom-level analysis based on DKFs.
  • the MapRank module shown in FIG. 51 may perform the ranking, and pass the results to the MapMerge module which is described above.
  • FIG. 49 illustrates aspects of a viewing method to recover and/or reset head pose, according to some embodiments.
  • the viewing device is powered on.
  • a new session is initiated.
  • a new session may include establishing head pose.
  • One or more capture devices on a head-mounted frame secured to a head of a user capture surfaces of an environment by first capturing images of the environment and then determining the surfaces from the images. Surface data is also combined with a data from a gravitational sensor to establish head pose.
  • a processor of the viewing device enters a routine for tracking of head pose.
  • the capture devices continue to capture surfaces of the environment as the user moves their head to determine an orientation of the head-mounted frame relative to the surfaces.
  • the processor determines whether head pose has been lost.
  • Head pose may become lost due to “edge” cases, such as too many reflective surfaces, low light, blank walls, being outdoor, etc. that may result in low feature acquisition, or because of dynamic cases such as a crowd that moves and forms part of the map.
  • the routine at 1430 allows for a certain amount of time, for example 10 seconds, to pass to allow enough time to determine whether head pose has been lost. If head pose has not been lost, then the processor returns to 1420 and again enters tracking of head pose.
  • the processor enters a routine at 1440 to recover head pose. If head pose is lost due to low light, then a message such as the following message is displayed to the user through a display of the viewing device:
  • the system will continue to monitor whether there is sufficient light available and whether head pose can be recovered.
  • the system may alternatively determine that low texture of surfaces is causing head pose to be lost, in which case the user is given the following prompt in the display as a suggestion to improve capturing of surfaces:
  • the processor enters a routine to determine whether head pose recovery has failed. If head pose recovery has not failed (i.e. head pose recovery has succeeded), then the processor returns to Act 1420 by again entering tracking of head pose. If head pose recovery has failed, the processor returns to Act 1410 to establish a new session. As part of the new session, all cached data is invalidated, whereafter head pose is established anew.
  • FIG. 50 shows a diagrammatic representation of a machine in the exemplary form of a computer system 1900 within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed, according to some embodiments.
  • the machine operates as a standalone device or may be connected (e.g., networked) to other machines.
  • the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
  • the exemplary computer system 1900 includes a processor 1902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory 1904 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory 1906 (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus 1908 .
  • a processor 1902 e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both
  • main memory 1904 e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.
  • DRAM dynamic random access memory
  • SDRAM synchronous DRAM
  • RDRAM Rambus DRAM
  • static memory 1906 e.g., flash memory, static random access
  • the computer system 1900 may further include a disk drive unit 1916 , and a network interface device 1920 .
  • the disk drive unit 1916 includes a machine-readable medium 1922 on which is stored one or more sets of instructions 1924 (e.g., software) embodying any one or more of the methodologies or functions described herein.
  • the software may also reside, completely or at least partially, within the main memory 1904 and/or within the processor 1902 during execution thereof by the computer system 1900 , the main memory 1904 and the processor 1902 also constituting machine-readable media.
  • the software may further be transmitted or received over a network 18 via the network interface device 1920 .
  • the computer system 1900 includes a driver chip 1950 that is used to drive projectors to generate light.
  • the driver chip 1950 includes its own data store 1960 and its own processor 1962 .
  • machine-readable medium 1922 is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
  • the term “machine-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention.
  • the term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals.
  • embodiments are described in connection with an augmented (AR) environment. It should be appreciated that some or all of the techniques described herein may be applied in an MR environment or more generally in other XR environments, and in VR environments.
  • embodiments are described in connection with devices, such as wearable devices. It should be appreciated that some or all of the techniques described herein may be implemented via networks (such as cloud), discrete applications, and/or any suitable combinations of devices, networks, and discrete applications.
  • networks such as cloud
  • discrete applications and/or any suitable combinations of devices, networks, and discrete applications.
  • FIG. 38 provides examples of criteria that may be used to filter candidate maps to yield a set of high ranking maps. Other criteria may be used instead of or in addition to the described criteria. For example, if multiple candidate maps have similar values of a metric used for filtering out less desirable maps, characteristics of the candidate maps may be used to determine which maps are retained as candidate maps or filtered out. For example, larger or more dense candidate maps may be prioritized over smaller candidate maps.
  • the embodiments can be implemented in any of numerous ways.
  • the embodiments may be implemented using hardware, software or a combination thereof.
  • the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.
  • processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art by names such as CPU chips, GPU chips, microprocessor, microcontroller, or co-processor.
  • a processor may be implemented in custom circuitry, such as an ASIC, or semicustom circuitry resulting from configuring a programmable logic device.
  • a processor may be a portion of a larger circuit or semiconductor device, whether commercially available, semi-custom or custom.
  • some commercially available microprocessors have multiple cores such that one or a subset of those cores may constitute a processor.
  • a processor may be implemented using circuitry in any suitable format.
  • a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable or fixed electronic device.
  • PDA Personal Digital Assistant
  • a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format. In the embodiment illustrated, the input/output devices are illustrated as physically separate from the computing device. In some embodiments, however, the input and/or output devices may be physically integrated into the same unit as the processor or other elements of the computing device. For example, a keyboard might be implemented as a soft keyboard on a touch screen. In some embodiments, the input/output devices may be entirely disconnected from the computing device, and functionally integrated through a wireless connection.
  • Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet.
  • networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.
  • the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.
  • the disclosure may be embodied as a computer readable storage medium (or multiple computer readable media) (e.g., a computer memory, one or more floppy discs, compact discs (CD), optical discs, digital video disks (DVD), magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the disclosure discussed above.
  • a computer readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form.
  • Such a computer readable storage medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure as discussed above.
  • the term “computer-readable storage medium” encompasses only a computer-readable medium that can be considered to be a manufacture (i.e., article of manufacture) or a machine.
  • the disclosure may be embodied as a computer readable medium other than a computer-readable storage medium, such as a propagating signal.
  • program or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure as discussed above. Additionally, it should be appreciated that according to one aspect of this embodiment, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present disclosure.
  • Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices.
  • program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
  • functionality of the program modules may be combined or distributed as desired in various embodiments.
  • data structures may be stored in computer-readable media in any suitable form.
  • data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a computer-readable medium that conveys relationship between the fields.
  • any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements.
  • the disclosure may be embodied as a method, of which an example has been provided.
  • the acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.
US16/538,759 2018-08-13 2019-08-12 Cross reality system Active US10957112B2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US16/538,759 US10957112B2 (en) 2018-08-13 2019-08-12 Cross reality system
US16/593,745 US11227435B2 (en) 2018-08-13 2019-10-04 Cross reality system
US17/208,844 US11386629B2 (en) 2018-08-13 2021-03-22 Cross reality system
US17/542,306 US11978159B2 (en) 2018-08-13 2021-12-03 Cross reality system

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US201862718357P 2018-08-13 2018-08-13
US201862742237P 2018-10-05 2018-10-05
US201962812935P 2019-03-01 2019-03-01
US201962815955P 2019-03-08 2019-03-08
US201962868786P 2019-06-28 2019-06-28
US201962870954P 2019-07-05 2019-07-05
US201962884109P 2019-08-07 2019-08-07
US16/538,759 US10957112B2 (en) 2018-08-13 2019-08-12 Cross reality system

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US16/593,745 Continuation-In-Part US11227435B2 (en) 2018-08-13 2019-10-04 Cross reality system
US17/208,844 Continuation US11386629B2 (en) 2018-08-13 2021-03-22 Cross reality system

Publications (2)

Publication Number Publication Date
US20200090407A1 US20200090407A1 (en) 2020-03-19
US10957112B2 true US10957112B2 (en) 2021-03-23

Family

ID=69525776

Family Applications (2)

Application Number Title Priority Date Filing Date
US16/538,759 Active US10957112B2 (en) 2018-08-13 2019-08-12 Cross reality system
US17/208,844 Active US11386629B2 (en) 2018-08-13 2021-03-22 Cross reality system

Family Applications After (1)

Application Number Title Priority Date Filing Date
US17/208,844 Active US11386629B2 (en) 2018-08-13 2021-03-22 Cross reality system

Country Status (5)

Country Link
US (2) US10957112B2 (ja)
EP (1) EP3837674A4 (ja)
JP (2) JP7445642B2 (ja)
CN (1) CN112805750A (ja)
WO (1) WO2020036898A1 (ja)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11257294B2 (en) 2019-10-15 2022-02-22 Magic Leap, Inc. Cross reality system supporting multiple device types
US11386627B2 (en) 2019-11-12 2022-07-12 Magic Leap, Inc. Cross reality system with localization service and shared location-based content
US11386629B2 (en) * 2018-08-13 2022-07-12 Magic Leap, Inc. Cross reality system
US11410395B2 (en) 2020-02-13 2022-08-09 Magic Leap, Inc. Cross reality system with accurate shared maps
US11551430B2 (en) 2020-02-26 2023-01-10 Magic Leap, Inc. Cross reality system with fast localization
US11562542B2 (en) 2019-12-09 2023-01-24 Magic Leap, Inc. Cross reality system with simplified programming of virtual content
US11562525B2 (en) 2020-02-13 2023-01-24 Magic Leap, Inc. Cross reality system with map processing using multi-resolution frame descriptors
US11568605B2 (en) 2019-10-15 2023-01-31 Magic Leap, Inc. Cross reality system with localization service
US11632679B2 (en) 2019-10-15 2023-04-18 Magic Leap, Inc. Cross reality system with wireless fingerprints
US11789524B2 (en) 2018-10-05 2023-10-17 Magic Leap, Inc. Rendering location specific virtual content in any location
US11830149B2 (en) 2020-02-13 2023-11-28 Magic Leap, Inc. Cross reality system with prioritization of geolocation information for localization
US11900547B2 (en) 2020-04-29 2024-02-13 Magic Leap, Inc. Cross reality system for large scale environments
US11978159B2 (en) 2018-08-13 2024-05-07 Magic Leap, Inc. Cross reality system

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10140392B1 (en) 2017-06-29 2018-11-27 Best Apps, Llc Computer aided systems and methods for creating custom products
US10254941B2 (en) 2017-06-29 2019-04-09 Best Apps, Llc Computer aided systems and methods for creating custom products
US10922449B2 (en) 2018-11-21 2021-02-16 Best Apps, Llc Computer aided systems and methods for creating custom products
US11625806B2 (en) * 2019-01-23 2023-04-11 Qualcomm Incorporated Methods and apparatus for standardized APIs for split rendering
GB2582657B (en) * 2019-03-29 2021-09-15 Sony Interactive Entertainment Inc User tracking system using a camera mounted on the user and an external camera
US11748445B1 (en) * 2019-04-30 2023-09-05 Apple Inc. Techniques for managing feature maps
US11470017B2 (en) * 2019-07-30 2022-10-11 At&T Intellectual Property I, L.P. Immersive reality component management via a reduced competition core network component
US20220335693A1 (en) * 2019-07-31 2022-10-20 Nippon Telegraph And Telephone Corporation Mixed reality space sharing system, sharing information management server, mixed reality terminal, mixed reality space sharing method, and sharing information management program
US11816757B1 (en) * 2019-12-11 2023-11-14 Meta Platforms Technologies, Llc Device-side capture of data representative of an artificial reality environment
CN115053262A (zh) * 2020-02-05 2022-09-13 麦克赛尔株式会社 空间识别系统、空间识别方法、信息终端以及服务器装置
US11263371B2 (en) 2020-03-03 2022-03-01 Best Apps, Llc Computer aided systems and methods for creating custom products
US11688073B2 (en) * 2020-04-14 2023-06-27 Samsung Electronics Co., Ltd. Method and system for depth map reconstruction
US11514203B2 (en) * 2020-05-18 2022-11-29 Best Apps, Llc Computer aided systems and methods for creating custom products
CN111815779A (zh) * 2020-06-29 2020-10-23 浙江商汤科技开发有限公司 对象展示方法及装置、定位方法及装置以及电子设备
SE2030252A1 (en) 2020-08-13 2022-02-14 Nagoon AB Method performed by a computer system for creation of augmented reality experiences and connection of these to the real world
CN114190922B (zh) * 2020-09-18 2023-04-21 四川大学 Tms头动检测方法
WO2022074294A1 (en) * 2020-10-06 2022-04-14 Nokia Technologies Oy Network-based spatial computing for extended reality (xr) applications
EP4015996A1 (en) * 2020-12-17 2022-06-22 Siemens Aktiengesellschaft Method for generating a map for augmented reality devices in an industrial facility
US11615594B2 (en) 2021-01-21 2023-03-28 Samsung Electronics Co., Ltd. Systems and methods for reconstruction of dense depth maps
JPWO2023026700A1 (ja) * 2021-08-26 2023-03-02
EP4160521A1 (en) * 2021-09-30 2023-04-05 Siemens Aktiengesellschaft Processing a picture section with an augmented reality device
WO2023052485A1 (en) * 2021-09-30 2023-04-06 Siemens Aktiengesellschaft Processing a picture section with an augmented reality device
WO2023102552A1 (en) * 2021-12-03 2023-06-08 Hover Inc. System and methods for validating imagery pipelines
US11682180B1 (en) * 2021-12-09 2023-06-20 Qualcomm Incorporated Anchoring virtual content to physical surfaces
WO2024010220A1 (ko) * 2022-07-06 2024-01-11 삼성전자 주식회사 거리 센서 활성화 방법 및 전자 장치
EP4336458A1 (de) * 2022-09-08 2024-03-13 Viewlicity GmbH Verfahren und systeme zum betreiben einer anzeigeeinrichtung einer simulation

Citations (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070298866A1 (en) 2006-06-26 2007-12-27 Paolo Gaudiano Methods and systems for interactive customization of avatars and other animate or inanimate items in video games
US20080090659A1 (en) 2006-10-12 2008-04-17 Maximino Aguilar Virtual world event notification from a persistent world game server in a logically partitioned game console
US20080284889A1 (en) 2007-05-15 2008-11-20 Sony Corporation Image pickup apparatus and method of correcting captured image data
US20090256903A1 (en) 2004-09-23 2009-10-15 Conversion Works, Inc. System and method for processing video images
US20110254950A1 (en) * 2008-10-09 2011-10-20 Isis Innovation Limited Visual tracking of objects in images, and segmentation of images
US20120249741A1 (en) 2011-03-29 2012-10-04 Giuliano Maciocci Anchoring virtual images to real world surfaces in augmented reality systems
US20130141419A1 (en) * 2011-12-01 2013-06-06 Brian Mount Augmented reality with realistic occlusion
US20130257907A1 (en) * 2012-03-30 2013-10-03 Sony Mobile Communications Inc. Client device
US20140003762A1 (en) 2012-06-11 2014-01-02 Magic Leap, Inc. Multiple depth plane three-dimensional display using a wave guide reflector array projector
US20140282162A1 (en) 2013-03-15 2014-09-18 Elwha Llc Cross-reality select, drag, and drop for augmented reality systems
EP2808842A2 (en) * 2013-05-31 2014-12-03 Technische Universität München An apparatus and method for tracking and reconstructing three-dimensional objects
US20140372957A1 (en) 2013-06-18 2014-12-18 Brian E. Keane Multi-step virtual object selection
US20150016777A1 (en) 2012-06-11 2015-01-15 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US20150178939A1 (en) 2013-11-27 2015-06-25 Magic Leap, Inc. Virtual and augmented reality systems and methods
US20150186745A1 (en) 2012-07-30 2015-07-02 Sony Computer Entertainment Europe Limited Localisation and mapping
US20150302665A1 (en) 2014-04-18 2015-10-22 Magic Leap, Inc. Triangulation of points using known points in augmented or virtual reality systems
US20150321103A1 (en) 2014-05-08 2015-11-12 Sony Computer Entertainment Europe Limited Image capture method and apparatus
WO2015192117A1 (en) 2014-06-14 2015-12-17 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US20160026253A1 (en) 2014-03-11 2016-01-28 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US20160180593A1 (en) 2014-07-02 2016-06-23 Huizhou Tcl Mobile Communication Co., Ltd. Wearable device-based augmented reality method and system
US20160196692A1 (en) 2015-01-02 2016-07-07 Eon Reality, Inc. Virtual lasers for interacting with augmented reality environments
US20170031160A1 (en) * 2012-05-11 2017-02-02 Digilens, Inc. Apparatus for eye tracking
US20170061696A1 (en) * 2015-08-31 2017-03-02 Samsung Electronics Co., Ltd. Virtual reality display apparatus and display method thereof
US20170094227A1 (en) 2015-09-25 2017-03-30 Northrop Grumman Systems Corporation Three-dimensional spatial-awareness vision system
US20170091996A1 (en) 2015-09-25 2017-03-30 Magic Leap, Inc. Methods and Systems for Detecting and Combining Structural Features in 3D Reconstruction
US20170352192A1 (en) * 2014-11-16 2017-12-07 Eonite Perception Inc. Systems and methods for augmented reality preparation, processing, and application
US20180045963A1 (en) 2016-08-11 2018-02-15 Magic Leap, Inc. Automatic placement of a virtual object in a three-dimensional space
US20180053284A1 (en) 2016-08-22 2018-02-22 Magic Leap, Inc. Virtual, augmented, and mixed reality systems and methods
US20180213359A1 (en) 2017-01-23 2018-07-26 Magic Leap, Inc. Localization determination for mixed reality systems
US20180268611A1 (en) 2017-03-17 2018-09-20 Magic Leap, Inc. Technique for recording augmented reality data
US20180304153A1 (en) * 2015-10-21 2018-10-25 Sharp Kabushiki Kaisha Image generating device, method of controlling image generating device, display system, image generation control program, and computer-readable storage medium
US20190005725A1 (en) * 2017-06-29 2019-01-03 Fanuc Corporation Virtual object display system
US20190147341A1 (en) 2017-11-14 2019-05-16 Magic Leap, Inc. Fully convolutional interest point detection and description via homographic adaptation
US20190188474A1 (en) 2017-12-15 2019-06-20 Magic Leap, Inc. Enhanced pose determination for display device
US20200034624A1 (en) 2018-07-24 2020-01-30 Magic Leap, Inc. Methods and apparatuses for determining and/or evaluating localizing maps of image display devices
US20200051328A1 (en) 2018-08-13 2020-02-13 Magic Leap, Inc. Cross reality system
US20200111255A1 (en) 2018-10-05 2020-04-09 Magic Leap, Inc. Rendering location specific virtual content in any location

Family Cites Families (127)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100483806B1 (ko) 2002-07-18 2005-04-20 한국과학기술원 예제동작을 이용한 영상기반 동작복원방법
US20050228849A1 (en) 2004-03-24 2005-10-13 Tong Zhang Intelligent key-frame extraction from a video
US7583858B2 (en) 2004-10-12 2009-09-01 Eastman Kodak Company Image processing based on direction of gravity
US20080303787A1 (en) 2005-10-21 2008-12-11 Zheng Yu Brian Touch Screen Apparatus And Methods
US8781162B2 (en) 2011-01-05 2014-07-15 Ailive Inc. Method and system for head tracking and pose estimation
SG183690A1 (en) 2007-08-06 2012-09-27 Trx Systems Inc Locating, tracking, and/or monitoring personnel and/or assets both indoors and outdoors
US9812096B2 (en) 2008-01-23 2017-11-07 Spy Eye, Llc Eye mounted displays and systems using eye mounted displays
US20100208033A1 (en) 2009-02-13 2010-08-19 Microsoft Corporation Personal Media Landscapes in Mixed Reality
US8839121B2 (en) 2009-05-06 2014-09-16 Joseph Bertolami Systems and methods for unifying coordinate systems in augmented reality applications
KR20100138725A (ko) 2009-06-25 2010-12-31 삼성전자주식회사 가상 세계 처리 장치 및 방법
US9119027B2 (en) 2009-10-06 2015-08-25 Facebook, Inc. Sharing of location-based content item in social networking service
US8185596B2 (en) 2010-02-22 2012-05-22 Samsung Electronics Co., Ltd. Location-based communication method and system
JP2013141049A (ja) * 2010-03-24 2013-07-18 Hitachi Ltd 世界座標系データベースを利用したサーバ及び端末
US10580051B2 (en) 2010-05-18 2020-03-03 Google Llc Background element associated with an application installed in a browser application
KR101686171B1 (ko) 2010-06-08 2016-12-13 삼성전자주식회사 영상 및 거리 데이터를 이용한 위치 인식 장치 및 방법
WO2012006578A2 (en) 2010-07-08 2012-01-12 The Regents Of The University Of California End-to-end visual recognition system and methods
US8526368B2 (en) 2011-05-17 2013-09-03 Qualcomm Incorporated Wi-Fi access point characteristics database
US20120314031A1 (en) 2011-06-07 2012-12-13 Microsoft Corporation Invariant features for computer vision
US9082214B2 (en) 2011-07-01 2015-07-14 Disney Enterprises, Inc. 3D drawing system for providing a real time, personalized, and immersive artistic experience
US10019962B2 (en) 2011-08-17 2018-07-10 Microsoft Technology Licensing, Llc Context adaptive user interface for augmented reality display
CN103959308B (zh) 2011-08-31 2017-09-19 Metaio有限公司 以参考特征匹配图像特征的方法
US8243102B1 (en) 2011-10-12 2012-08-14 Google Inc. Derivative-based selection of zones for banded map display
WO2013101753A1 (en) 2011-12-30 2013-07-04 Mako Surgical Corp. Systems and methods for customizing interactive haptic boundaries
US9530221B2 (en) 2012-01-06 2016-12-27 Pelco, Inc. Context aware moving object detection
JP6208151B2 (ja) 2012-02-06 2017-10-04 ソニー インタラクティブ エンタテインメント ヨーロッパ リミテッド 拡張現実のためのブックオブジェクト
CN103297677B (zh) 2012-02-24 2016-07-06 卡西欧计算机株式会社 生成重构图像的图像生成装置以及图像生成方法
KR20130110907A (ko) 2012-03-30 2013-10-10 삼성전자주식회사 가상 현실과 증강 현실을 이용한 원격 제어 장치 및 방법
US9122321B2 (en) 2012-05-04 2015-09-01 Microsoft Technology Licensing, Llc Collaboration environment using see through displays
US9311750B2 (en) 2012-06-05 2016-04-12 Apple Inc. Rotation operations in a mapping application
US9111135B2 (en) 2012-06-25 2015-08-18 Aquifi, Inc. Systems and methods for tracking human hands using parts based template matching using corresponding pixels in bounded regions of a sequence of frames that are a specified distance interval from a reference camera
US8829409B2 (en) 2012-10-10 2014-09-09 Thermo Fisher Scientific Inc. Ultra-high speed imaging array with orthogonal readout architecture
KR102516124B1 (ko) 2013-03-11 2023-03-29 매직 립, 인코포레이티드 증강 및 가상 현실을 위한 시스템 및 방법
US9349072B2 (en) 2013-03-11 2016-05-24 Microsoft Technology Licensing, Llc Local feature based image compression
US20140267234A1 (en) * 2013-03-15 2014-09-18 Anselm Hook Generation and Sharing Coordinate System Between Users on Mobile
US9269022B2 (en) 2013-04-11 2016-02-23 Digimarc Corporation Methods for object recognition and related arrangements
US9154919B2 (en) 2013-04-22 2015-10-06 Alcatel Lucent Localization systems and methods
US9406137B2 (en) 2013-06-14 2016-08-02 Qualcomm Incorporated Robust tracking using point and line features
WO2014202258A1 (en) 2013-06-21 2014-12-24 National University Of Ireland, Maynooth A method for mapping an environment
US9779548B2 (en) 2013-06-25 2017-10-03 Jordan Kent Weisman Multiuser augmented reality system
US9264702B2 (en) * 2013-08-19 2016-02-16 Qualcomm Incorporated Automatic calibration of scene camera for optical see-through head mounted display
US9646384B2 (en) 2013-09-11 2017-05-09 Google Technology Holdings LLC 3D feature descriptors with camera pose information
AU2013237718A1 (en) * 2013-10-04 2015-04-23 Canon Kabushiki Kaisha Method, apparatus and system for selecting a frame
KR102493498B1 (ko) 2013-11-27 2023-01-27 매직 립, 인코포레이티드 가상 및 증강 현실 시스템들 및 방법들
US9877158B2 (en) 2013-12-20 2018-01-23 Intel Corporation Wi-Fi scan scheduling and power adaptation for low-power indoor location
US10586395B2 (en) 2013-12-30 2020-03-10 Daqri, Llc Remote object detection and local tracking using visual odometry
US9830679B2 (en) 2014-03-25 2017-11-28 Google Llc Shared virtual reality
US10380410B2 (en) 2014-04-07 2019-08-13 Eyeways Systems Ltd. Apparatus and method for image-based positioning, orientation and situational awareness
US9754167B1 (en) * 2014-04-17 2017-09-05 Leap Motion, Inc. Safety for wearable virtual reality devices via object detection and tracking
WO2015161307A1 (en) * 2014-04-18 2015-10-22 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US10068373B2 (en) 2014-07-01 2018-09-04 Samsung Electronics Co., Ltd. Electronic device for providing map information
US10198865B2 (en) 2014-07-10 2019-02-05 Seiko Epson Corporation HMD calibration with direct geometric modeling
US10509865B2 (en) 2014-09-18 2019-12-17 Google Llc Dress form for three-dimensional drawing inside virtual reality environment
US20200252233A1 (en) 2014-09-24 2020-08-06 James Thomas O'Keeffe System and method for user profile enabled smart building control
US10719727B2 (en) 2014-10-01 2020-07-21 Apple Inc. Method and system for determining at least one property related to at least part of a real environment
CN107209247B (zh) 2014-12-04 2021-06-18 赫尔环球有限公司 支持数据的协作收集
US10335677B2 (en) 2014-12-23 2019-07-02 Matthew Daniel Fuchs Augmented reality system with agent device for viewing persistent content and method of operation thereof
US9852546B2 (en) 2015-01-28 2017-12-26 CCP hf. Method and system for receiving gesture input via virtual control objects
US20160300389A1 (en) 2015-04-08 2016-10-13 Exactigo, Inc. Correlated immersive virtual simulation for indoor navigation
US9467718B1 (en) 2015-05-06 2016-10-11 Echostar Broadcasting Corporation Apparatus, systems and methods for a content commentary community
KR101725478B1 (ko) 2015-05-21 2017-04-11 주식회사 맥스트 3d 점군 인식 기반의 증강 현실 출력 방법과 이를 수행하기 위한 장치 및 시스템
US20160381118A1 (en) 2015-06-23 2016-12-29 Microsoft Technology Licensing, Llc Extracting and formatting content from web-resources
ES2920837T3 (es) 2015-09-10 2022-08-10 Oriient New Media Ltd Navegar, seguir y posicionar los dispositivos móviles en zonas sin GPS o con GPS inexacto con la generación automática de mapas
US10706465B2 (en) 2015-09-14 2020-07-07 The Toronto-Dominion Bank Connected device-based property evaluation
GB201517101D0 (en) 2015-09-28 2015-11-11 Univ Essex Entpr Ltd Mixed-reality system
US9706366B2 (en) 2015-11-06 2017-07-11 International Business Machines Corporation WiFi-fingerprint based indoor localization map
US10254845B2 (en) 2016-01-05 2019-04-09 Intel Corporation Hand gesture recognition for cursor control
US10523865B2 (en) 2016-01-06 2019-12-31 Texas Instruments Incorporated Three dimensional rendering for surround view using predetermined viewpoint lookup tables
EP3411779A4 (en) * 2016-02-05 2019-02-20 Magic Leap, Inc. SYSTEMS AND METHODS FOR INCREASED REALITY
WO2017143303A1 (en) 2016-02-17 2017-08-24 Meta Company Apparatuses, methods and systems for sharing virtual elements
US10373380B2 (en) 2016-02-18 2019-08-06 Intel Corporation 3-dimensional scene analysis for augmented reality operations
JP6776609B2 (ja) 2016-02-22 2020-10-28 デクセリアルズ株式会社 異方性導電フィルム
US20180122143A1 (en) 2016-03-15 2018-05-03 Sutherland Cook Ellwood, JR. Hybrid photonic vr/ar systems
US10115234B2 (en) 2016-03-21 2018-10-30 Accenture Global Solutions Limited Multiplatform based experience generation
US10802147B2 (en) 2016-05-18 2020-10-13 Google Llc System and method for concurrent odometry and mapping
AU2017203641B2 (en) 2016-05-31 2018-05-24 Accenture Global Solutions Limited Interactive virtual reality platforms
US10162362B2 (en) 2016-08-29 2018-12-25 PerceptIn, Inc. Fault tolerance to provide robust tracking for autonomous positional awareness
US10007868B2 (en) 2016-09-19 2018-06-26 Adobe Systems Incorporated Font replacement based on visual similarity
US10354129B2 (en) 2017-01-03 2019-07-16 Intel Corporation Hand gesture recognition for virtual reality and augmented reality devices
US11275163B2 (en) 2017-01-25 2022-03-15 Korea Institute Of Science And Technology Slam method and apparatus robust to wireless environment change
US10460489B2 (en) 2017-03-15 2019-10-29 Facebook, Inc. Visual editor for designing augmented-reality effects and configuring scaling parameters
US11017592B2 (en) 2017-03-30 2021-05-25 Magic Leap, Inc. Centralized rendering
US10600252B2 (en) 2017-03-30 2020-03-24 Microsoft Technology Licensing, Llc Coarse relocalization using signal fingerprints
US9754397B1 (en) 2017-04-07 2017-09-05 Mirage Worlds, Inc. Systems and methods for contextual augmented reality sharing and performance
CA3060209A1 (en) 2017-05-01 2018-11-08 Magic Leap, Inc. Matching content to a spatial 3d environment
US11417091B2 (en) 2017-05-30 2022-08-16 Ptc Inc. Use of coordinated local user devices during a shared augmented reality session
CN109145927A (zh) 2017-06-16 2019-01-04 杭州海康威视数字技术股份有限公司 一种对形变图像的目标识别方法及装置
US10503955B2 (en) 2017-08-29 2019-12-10 Synaptics Incorporated Device with improved circuit positioning
WO2019046774A1 (en) 2017-09-01 2019-03-07 Memorial Sloan Kettering Cancer Center SYSTEMS AND METHODS FOR GENERATING 3D MEDICAL IMAGES BY SCANNING A WHOLE TISSUE BLOCK
US10546387B2 (en) 2017-09-08 2020-01-28 Qualcomm Incorporated Pose determination with semantic segmentation
US10685456B2 (en) 2017-10-12 2020-06-16 Microsoft Technology Licensing, Llc Peer to peer remote localization for devices
US10612929B2 (en) 2017-10-17 2020-04-07 AI Incorporated Discovering and plotting the boundary of an enclosure
CN111684495A (zh) 2017-12-22 2020-09-18 奇跃公司 用于在混合现实系统中管理和显示虚拟内容的方法和系统
US11237004B2 (en) 2018-03-27 2022-02-01 Uatc, Llc Log trajectory estimation for globally consistent maps
US10838574B2 (en) 2018-04-09 2020-11-17 Spatial Systems Inc. Augmented reality computing environments—workspace save and load
US10803671B2 (en) 2018-05-04 2020-10-13 Microsoft Technology Licensing, Llc Authoring content in three-dimensional environment
US11202006B2 (en) 2018-05-18 2021-12-14 Samsung Electronics Co., Ltd. CMOS-assisted inside-out dynamic vision sensor tracking for low power mobile platforms
CN110515452B (zh) 2018-05-22 2022-02-22 腾讯科技（深圳）有限公司 图像处理方法、装置、存储介质和计算机设备
US10706629B2 (en) 2018-06-15 2020-07-07 Dell Products, L.P. Coordinate override in virtual, augmented, and mixed reality (xR) applications
JP7445642B2 (ja) 2018-08-13 2024-03-07 マジック リープ， インコーポレイテッド クロスリアリティシステム
WO2020049333A1 (en) 2018-09-04 2020-03-12 Aktiia Sa System for determining a blood pressure of one or a plurality of users
US10854007B2 (en) 2018-12-03 2020-12-01 Microsoft Technology Licensing, Llc Space models for mixed reality
US20200211290A1 (en) 2018-12-26 2020-07-02 Lg Electronics Inc. Xr device for providing ar mode and vr mode and method for controlling the same
WO2020192039A1 (en) 2019-03-27 2020-10-01 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Three-dimensional localization using light-depth images
US11151792B2 (en) 2019-04-26 2021-10-19 Google Llc System and method for creating persistent mappings in augmented reality
US10748302B1 (en) 2019-05-02 2020-08-18 Apple Inc. Multiple user simultaneous localization and mapping (SLAM)
US11010921B2 (en) 2019-05-16 2021-05-18 Qualcomm Incorporated Distributed pose estimation
US20200364937A1 (en) 2019-05-16 2020-11-19 Subvrsive, Inc. System-adaptive augmented reality
US11145083B2 (en) 2019-05-21 2021-10-12 Microsoft Technology Licensing, Llc Image-based localization
US10854012B1 (en) 2019-05-29 2020-12-01 Dell Products, L.P. Concealing loss of distributed simultaneous localization and mapping (SLAM) data in edge cloud architectures
US20200380263A1 (en) 2019-05-29 2020-12-03 Gyrfalcon Technology Inc. Detecting key frames in video compression in an artificial intelligence semiconductor solution
US10852828B1 (en) 2019-07-17 2020-12-01 Dell Products, L.P. Automatic peripheral pairing with hand assignments in virtual, augmented, and mixed reality (xR) applications
US10936874B1 (en) 2019-08-13 2021-03-02 Dell Products, L.P. Controller gestures in virtual, augmented, and mixed reality (xR) applications
KR20190104928A (ko) 2019-08-22 2019-09-11 엘지전자 주식회사 Xr 디바이스 및 xr 디바이스 제어 방법
US11270515B2 (en) 2019-09-04 2022-03-08 Qualcomm Incorporated Virtual keyboard
US11698674B2 (en) 2019-09-09 2023-07-11 Apple Inc. Multimodal inputs for computer-generated reality
CN114616534A (zh) 2019-10-15 2022-06-10 奇跃公司 具有无线指纹的交叉现实系统
CN114600064A (zh) 2019-10-15 2022-06-07 奇跃公司 具有定位服务的交叉现实系统
WO2021076757A1 (en) 2019-10-15 2021-04-22 Magic Leap, Inc. Cross reality system supporting multiple device types
EP4052086A4 (en) 2019-10-31 2023-11-15 Magic Leap, Inc. EXTENDED REALITY SYSTEM PROVIDING QUALITY INFORMATION ABOUT PERSISTENT COORDINATE FRAMES
JP2023504775A (ja) 2019-11-12 2023-02-07 マジック リープ， インコーポレイテッド 位置特定サービスおよび共有場所ベースのコンテンツを伴うクロスリアリティシステム
EP4073763A4 (en) 2019-12-09 2023-12-27 Magic Leap, Inc. CROSS-REALLY SYSTEM WITH SIMPLIFIED PROGRAMMING OF VIRTUAL CONTENT
US11410395B2 (en) 2020-02-13 2022-08-09 Magic Leap, Inc. Cross reality system with accurate shared maps
WO2021163289A1 (en) 2020-02-13 2021-08-19 Magic Leap, Inc. Cross reality system for large scale environments
WO2021163295A1 (en) 2020-02-13 2021-08-19 Magic Leap, Inc. Cross reality system with prioritization of geolocation information for localization
US11562525B2 (en) 2020-02-13 2023-01-24 Magic Leap, Inc. Cross reality system with map processing using multi-resolution frame descriptors
CN115461787A (zh) 2020-02-26 2022-12-09 奇跃公司 具有快速定位的交叉现实系统
WO2021222371A1 (en) 2020-04-29 2021-11-04 Magic Leap, Inc. Cross reality system for large scale environments

Patent Citations (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090256903A1 (en) 2004-09-23 2009-10-15 Conversion Works, Inc. System and method for processing video images
US20070298866A1 (en) 2006-06-26 2007-12-27 Paolo Gaudiano Methods and systems for interactive customization of avatars and other animate or inanimate items in video games
US20080090659A1 (en) 2006-10-12 2008-04-17 Maximino Aguilar Virtual world event notification from a persistent world game server in a logically partitioned game console
US20080284889A1 (en) 2007-05-15 2008-11-20 Sony Corporation Image pickup apparatus and method of correcting captured image data
US20110254950A1 (en) * 2008-10-09 2011-10-20 Isis Innovation Limited Visual tracking of objects in images, and segmentation of images
US20120249741A1 (en) 2011-03-29 2012-10-04 Giuliano Maciocci Anchoring virtual images to real world surfaces in augmented reality systems
US20130141419A1 (en) * 2011-12-01 2013-06-06 Brian Mount Augmented reality with realistic occlusion
US20130257907A1 (en) * 2012-03-30 2013-10-03 Sony Mobile Communications Inc. Client device
US20170031160A1 (en) * 2012-05-11 2017-02-02 Digilens, Inc. Apparatus for eye tracking
US20140003762A1 (en) 2012-06-11 2014-01-02 Magic Leap, Inc. Multiple depth plane three-dimensional display using a wave guide reflector array projector
US20150016777A1 (en) 2012-06-11 2015-01-15 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US20150186745A1 (en) 2012-07-30 2015-07-02 Sony Computer Entertainment Europe Limited Localisation and mapping
US20140282162A1 (en) 2013-03-15 2014-09-18 Elwha Llc Cross-reality select, drag, and drop for augmented reality systems
EP2808842A2 (en) * 2013-05-31 2014-12-03 Technische Universität München An apparatus and method for tracking and reconstructing three-dimensional objects
US20140372957A1 (en) 2013-06-18 2014-12-18 Brian E. Keane Multi-step virtual object selection
US20150178939A1 (en) 2013-11-27 2015-06-25 Magic Leap, Inc. Virtual and augmented reality systems and methods
US20160026253A1 (en) 2014-03-11 2016-01-28 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US20150302665A1 (en) 2014-04-18 2015-10-22 Magic Leap, Inc. Triangulation of points using known points in augmented or virtual reality systems
US20150302642A1 (en) 2014-04-18 2015-10-22 Magic Leap, Inc. Room based sensors in an augmented reality system
US20150321103A1 (en) 2014-05-08 2015-11-12 Sony Computer Entertainment Europe Limited Image capture method and apparatus
WO2015192117A1 (en) 2014-06-14 2015-12-17 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US20160180593A1 (en) 2014-07-02 2016-06-23 Huizhou Tcl Mobile Communication Co., Ltd. Wearable device-based augmented reality method and system
US20170352192A1 (en) * 2014-11-16 2017-12-07 Eonite Perception Inc. Systems and methods for augmented reality preparation, processing, and application
US20160196692A1 (en) 2015-01-02 2016-07-07 Eon Reality, Inc. Virtual lasers for interacting with augmented reality environments
US20170061696A1 (en) * 2015-08-31 2017-03-02 Samsung Electronics Co., Ltd. Virtual reality display apparatus and display method thereof
US20170094227A1 (en) 2015-09-25 2017-03-30 Northrop Grumman Systems Corporation Three-dimensional spatial-awareness vision system
US20170091996A1 (en) 2015-09-25 2017-03-30 Magic Leap, Inc. Methods and Systems for Detecting and Combining Structural Features in 3D Reconstruction
US20180304153A1 (en) * 2015-10-21 2018-10-25 Sharp Kabushiki Kaisha Image generating device, method of controlling image generating device, display system, image generation control program, and computer-readable storage medium
US20180045963A1 (en) 2016-08-11 2018-02-15 Magic Leap, Inc. Automatic placement of a virtual object in a three-dimensional space
US20180053284A1 (en) 2016-08-22 2018-02-22 Magic Leap, Inc. Virtual, augmented, and mixed reality systems and methods
US20180213359A1 (en) 2017-01-23 2018-07-26 Magic Leap, Inc. Localization determination for mixed reality systems
US20180268611A1 (en) 2017-03-17 2018-09-20 Magic Leap, Inc. Technique for recording augmented reality data
US20190005725A1 (en) * 2017-06-29 2019-01-03 Fanuc Corporation Virtual object display system
US20190147341A1 (en) 2017-11-14 2019-05-16 Magic Leap, Inc. Fully convolutional interest point detection and description via homographic adaptation
US20190188474A1 (en) 2017-12-15 2019-06-20 Magic Leap, Inc. Enhanced pose determination for display device
US20200034624A1 (en) 2018-07-24 2020-01-30 Magic Leap, Inc. Methods and apparatuses for determining and/or evaluating localizing maps of image display devices
US20200051328A1 (en) 2018-08-13 2020-02-13 Magic Leap, Inc. Cross reality system
US20200111255A1 (en) 2018-10-05 2020-04-09 Magic Leap, Inc. Rendering location specific virtual content in any location

Non-Patent Citations (19)

* Cited by examiner, † Cited by third party
Title
[No Author Listed], The difference between virtual reality, Augmented Reality and Mixed Reality. Forbes. Feb. 2, 2018. 5 pages. URL:https://www.forbes.com/sites/quora/2018/02/02/the-difference-between-virtual-reality-augmented-reality-and-mixed-reality/#634116762d07 [retrieved on Dec. 5, 2019].
Balntas et al., HPatches: A benchmark and evaluation of handcrafted and learned local descriptors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017. pp. 5173-5182.
Cadena et al., Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age. IEEE Transactions on Robotics. Dec. 2016;32(6):1309-1332.
Dang et al., Eigendecomposition-free training of deep networks with zero eigenvalue-based losses. arXiv:1803.08071. Mar. 26, 2018. 25 pages.
Gidaris et al, Unsupervised representation learning by predicting image rotations. arXiv:1803.07728. Mar. 21, 2018. 16 pages.
Henry et al., RGB-D mapping: Using Kinect-style depth cameras for dense 30 modeling of indoor environments. The International Journal of Robotics Research. Feb. 10, 2012. 26 pages. URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.480.160&rep=rep1 &type=pdf [retrieved on Jan. 26, 2020].
International Search Report and Written Opinion for International Application No. PCT/US2019/054819 dated Feb. 11, 2020.
International Search Report and Written Opinion for International Application No. PCT/US2019/054836 dated Dec. 31, 2019.
International search report and written opinion of the International Searching Authority for PCT/US2019/046240 dated Dec. 23, 2019, 13 pages. (Year: 2019). *
Invitation to Pay Additional Fees for International Application No. PCT/US2019/046240 dated Oct. 18, 2019.
Invitation to Pay Additional Fees for International Application No. PCT/US2019/054819 dated Dec. 4, 2019.
Qi et al., Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv:1612.00593. Apr. 10, 2017. 19 pages.
Shahrokni et al., Cross Reality System With Localization Service, U.S. Appl. No. 17/071,897, filed Oct. 15, 2020.
Shahrokni et al., Cross Reality System With Quality Information About Persistent Coordinate Frames, U.S. Appl. No. 17/084,174, filed Oct. 29, 2020.
Shveki et al., Cross Reality System With Wireless Fingerprints, U.S. Appl. No. 17/071,879, filed Oct. 15, 2020.
Snavely et al., Skeletal graphs for efficient structure from motion. IEEE Conference on Computer Vision and Pattern Recognition. Jun. 23, 2008. 11 pages. URL:http://www.cs.cornell.edu/˜snavely/projects/skeletalset.
Taira et al., InLoc: Indoor visual localization with dense matching and view synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. pp. 7199-7209.
Yi et al., Learning to find good correspondences. arXiv:1711.05971. May 21, 2018. 13 pages.
Zhao et al., Cross Reality System Supporting Multiple Device Types, U.S. Appl. No. 17/071,907, filed Oct. 15, 2020.

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11386629B2 (en) * 2018-08-13 2022-07-12 Magic Leap, Inc. Cross reality system
US11978159B2 (en) 2018-08-13 2024-05-07 Magic Leap, Inc. Cross reality system
US11789524B2 (en) 2018-10-05 2023-10-17 Magic Leap, Inc. Rendering location specific virtual content in any location
US11257294B2 (en) 2019-10-15 2022-02-22 Magic Leap, Inc. Cross reality system supporting multiple device types
US11568605B2 (en) 2019-10-15 2023-01-31 Magic Leap, Inc. Cross reality system with localization service
US11632679B2 (en) 2019-10-15 2023-04-18 Magic Leap, Inc. Cross reality system with wireless fingerprints
US11386627B2 (en) 2019-11-12 2022-07-12 Magic Leap, Inc. Cross reality system with localization service and shared location-based content
US11869158B2 (en) 2019-11-12 2024-01-09 Magic Leap, Inc. Cross reality system with localization service and shared location-based content
US11562542B2 (en) 2019-12-09 2023-01-24 Magic Leap, Inc. Cross reality system with simplified programming of virtual content
US11748963B2 (en) 2019-12-09 2023-09-05 Magic Leap, Inc. Cross reality system with simplified programming of virtual content
US11830149B2 (en) 2020-02-13 2023-11-28 Magic Leap, Inc. Cross reality system with prioritization of geolocation information for localization
US11790619B2 (en) 2020-02-13 2023-10-17 Magic Leap, Inc. Cross reality system with accurate shared maps
US11562525B2 (en) 2020-02-13 2023-01-24 Magic Leap, Inc. Cross reality system with map processing using multi-resolution frame descriptors
US11967020B2 (en) 2020-02-13 2024-04-23 Magic Leap, Inc. Cross reality system with map processing using multi-resolution frame descriptors
US11410395B2 (en) 2020-02-13 2022-08-09 Magic Leap, Inc. Cross reality system with accurate shared maps
US11551430B2 (en) 2020-02-26 2023-01-10 Magic Leap, Inc. Cross reality system with fast localization
US11900547B2 (en) 2020-04-29 2024-02-13 Magic Leap, Inc. Cross reality system for large scale environments

Also Published As

Publication number Publication date
WO2020036898A1 (en) 2020-02-20
CN112805750A (zh) 2021-05-14
EP3837674A4 (en) 2022-05-18
US20200090407A1 (en) 2020-03-19
JP2021534491A (ja) 2021-12-09
JP2024009071A (ja) 2024-01-19
US11386629B2 (en) 2022-07-12
JP7445642B2 (ja) 2024-03-07
US20210209859A1 (en) 2021-07-08
EP3837674A1 (en) 2021-06-23

Similar Documents

Publication Publication Date Title
US11386629B2 (en) Cross reality system
US11978159B2 (en) Cross reality system
US11568605B2 (en) Cross reality system with localization service
US11790619B2 (en) Cross reality system with accurate shared maps
US11257294B2 (en) Cross reality system supporting multiple device types
US11632679B2 (en) Cross reality system with wireless fingerprints
US11551430B2 (en) Cross reality system with fast localization
US20210256766A1 (en) Cross reality system for large scale environments
US11967020B2 (en) Cross reality system with map processing using multi-resolution frame descriptors
US11694394B2 (en) Cross reality system for large scale environment reconstruction
EP3861533A1 (en) A cross reality system

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: CITIBANK, N.A., AS COLLATERAL AGENT, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNORS:MAGIC LEAP, INC.;MOLECULAR IMPRINTS, INC.;MENTOR ACQUISITION ONE, LLC;REEL/FRAME:052729/0791

Effective date: 20200521

AS Assignment

Owner name: MAGIC LEAP, INC., FLORIDA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MIRANDA, JEREMY DWAYNE;TORRES, RAFAEL DOMINGOS;OLSHANSKY, DANIEL;AND OTHERS;SIGNING DATES FROM 20191031 TO 20210209;REEL/FRAME:055249/0372

Owner name: MAGIC LEAP, INC., FLORIDA

Free format text: PROPRIETARY INFORMATION AND INVENTIONS AGREEMENT;ASSIGNOR:SHAHROKNI, ALI;REEL/FRAME:055292/0614

Effective date: 20171224

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: CITIBANK, N.A., AS COLLATERAL AGENT, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNORS:MOLECULAR IMPRINTS, INC.;MENTOR ACQUISITION ONE, LLC;MAGIC LEAP, INC.;REEL/FRAME:060338/0665

Effective date: 20220504