JP2022509731A - Cross reality system - Google Patents

Abstract

About the physical world, where one or more user devices may have access to determine their location within the physical world, and applications may have access to define the location of virtual objects within the physical world. A cross-reality system that provides an immersive user experience by storing persistent spatial information. Persistent spatial information allows users to have a shared virtual and physical experience when interacting with a cross-reality system. In addition, persistent spatial information is used within the map of the physical world, where one or more devices access, locate, and use the cross-reality system within a previously stored map. Before doing so, it may be possible to reduce the need to map the physical space.

Description

(Mutual reference of related applications)
This patent application is filed October 5, 2018, which is incorporated herein by reference in its entirety, and is entitled "COORDINATE FRAME PROCESSING AUGMENTED REALITY", US Provisional Patent Application No. 62/742. Claim the priority and interests of No. 237. This patent application is also incorporated herein by reference in its entirety, filed March 1, 2019, entitled "MERGING A PLURALITY OF INDIVIDUALLY MAPPED ENVIRONMENTS", US Provisional Patent Application No. 62. Claim the priority and interests of / 812,935. This patent application is also filed March 8, 2019, which is incorporated herein by reference in its entirety, entitled "VIEWING DEVICE OR VIEWING DEVICES HAVING ONE OR MORE COORDINATE FRAME TRANSFORMERS", USA. Claim the priority and interests of Provisional Patent Application No. 62 / 815,955. This patent application is also filed June 28, 2019, which is incorporated herein by reference in its entirety, and is entitled "RANKING AND MERGING A PLURALITY OF ENVIRONMENT MAPS". Claim the priority and interests of No. 62 / 868,786. This patent application is also incorporated herein by reference in its entirety, filed July 5, 2019, entitled "RANKING AND MERGING A PLURALITY OF ENVIRONMENT MAPS". Claim the priority and interests of No. 62 / 870,954. This patent application is also incorporated herein by reference in its entirety, filed August 7, 2019, entitled "A VIEWING SYSTEM", US Provisional Patent Application No. 62 / 884,109. Claim the priority and interests of the issue.

This application generally relates to a cross-reality system.

The computer may create an X-reality (XR or cross-reality) environment in which some or all of the XR environment is generated by the computer as it controls the human user interface and is perceived by the user. These XR environments are virtual reality (VR), augmented reality (AR), and mixed reality, where some or all of the XR environment can be computer-generated using data that describes the environment in part. It can be a (MR) environment. The data may describe, for example, a virtual object that the user can sense or perceive and render to interact with the virtual object as part of the physical world. Users may experience these virtual objects as a result of data rendered and presented through a user interface device, such as a head-mounted display device. The data may be visible to the user, or played audibly to the user, may control the audio, or control the tactile (or tactile) interface, as the user perceives the virtual object. It may allow the user to experience a touch sensation, perceived or perceived.

XR systems can be useful for many applications ranging from scientific visualization, medical training, engineering design, and prototyping, remote control and telepresence, and personal entertainment. AR and MR, in contrast to VR, include one or more objects in relation to real objects in the physical world. The virtual object experience, which interacts with real objects, generally enhances user enjoyment when using an XR system and provides realistic and easily understandable information about how the physical world can be altered. To present, expand the possibilities for various uses.

To realistically render virtual content, the XR system may construct a representation of the physical world around the user of the system. The representation may be constructed, for example, by processed images obtained using sensors on wearable devices that form part of the XR system. In such a system, the user looks around the room or other physical environment in which the user intends to use the XR system until the system has sufficient information to construct a representation of that environment. Thereby, the initialization routine can be carried out. As the system operates and users move around or to other environments, sensors on wearable devices can obtain additional information and extend or update the representation of the physical world.

Aspects of the present application relate to methods and devices for providing X-reality (cross-reality or XR) scenes. Both techniques described herein may be used separately or in any suitable combination.

Some embodiments relate to electronic systems that include one or more sensors configured to capture information about a three-dimensional (3D) environment. The captured information includes multiple images. The electronic system includes at least one processor configured to execute computer executable instructions and generate a map of at least a portion of the 3D environment based on multiple images. Computer-executable instructions further include multiple keyframes from among multiple images, based on the steps of identifying multiple features within multiple images and, at least in part, multiple features of the selected keyframe. One or more coordinate frames associated with the steps to select and, at least in part, the steps to generate one or more coordinate frames based on the identified characteristics of the selected keyframe and the map of the 3D environment. Includes instructions for and with the step of storing as one or more persistent coordinate frames.

In some embodiments, the one or more sensors comprises a plurality of pixel circuits in which each image of the plurality of images is arranged in a two-dimensional array such that it comprises a plurality of pixels. Each feature corresponds to multiple pixels.

In some embodiments, the step of identifying a plurality of features in a plurality of images is based on a measurement of similarity to a group of pixels depicting a portion of a persistent object as the identified feature. Includes the step of selecting a number less than a given maximum value for the group of.

In some embodiments, the step of storing one or more coordinate frames represents at least a subset of the features in the selected keyframe from which the coordinate frames were generated, for each one or more coordinate frames. , Includes a step to memorize the descriptor.

In some embodiments, the step of storing one or more coordinate frames stores at least a subset of the features in the selected keyframe from which the coordinate frames were generated, for each one or more coordinate frames. Includes steps to do.

In some embodiments, the step of storing one or more coordinate frames is, for each one or more coordinate frames, the transformation between the coordinate frames of the map in the 3D environment and the persistent coordinate frames, and the coordinates from there. Includes a step to store geographic information that indicates the location of the selected keyframe in the 3D environment from which the frame was generated.

In some embodiments, the geographic information comprises a WiFi finger print of the location.

In some embodiments, the computer executable instruction comprises an instruction for calculating a feature descriptor for an individual feature using an artificial neural network.

In some embodiments, the first artificial neural network is the first artificial neural network. Computer executable instructions are configured to calculate a frame descriptor to represent a key frame, at least in part, based on the calculated feature descriptor for the identified feature within the key frame. It includes instructions for implementing a second artificial neural network.

In some embodiments, computer executable instructions are further configured to provide information running on a portable electronic system that characterizes a persistent coordinate frame of one or more persistent coordinate frames. Of the persistent coordinate frames, based on the application programming interface and the second plurality of images, instructions for refining the map of the 3D environment, and at least in part, the second plurality of images. It comprises instructions for adjusting one or more of the following and instructions for providing notification of a sustained coordinate frame adjusted through an application programming interface.

In some embodiments, the step of adjusting one or more persistent coordinate frames comprises adjusting the translation and rotation of one or more persistent coordinate frames with respect to the origin of the map in the 3D environment.

In some embodiments, the electronic system comprises a wearable device and one or more sensors are mounted on the wearable device. The map is a tracking map calculated on the wearable device. The origin of the map is determined based on where the device was powered on.

In some embodiments, the electronic system comprises a wearable device and one or more sensors are mounted on the wearable device. Computer-executable instructions also include instructions and / / instructions for generating one or more coordinate frames based on the tracked motion, indicating the steps to track the motion of the portable device and the motion of the wearable device above the threshold distance. Alternatively, it comprises a step for controlling the timing of execution of an instruction for storing one or more persistent coordinate frames, the threshold distance being 2 to 20 meters, and an instruction for the step.

Some embodiments relate to how to operate an electronic system and render virtual content in a 3D environment comprising a portable device. The method uses one or more processors to maintain a coordinate frame local to the portable device on the portable device, based on the output of one or more sensors on the portable device, and the stored coordinates. The step of retrieving the frame from the stored spatial information about the 3D environment, the step of calculating the transformation between the coordinate frame local to the portable device and the acquired stored coordinate frame, and the coordinates local to the virtual object. Based on the step of receiving the specification of the virtual object with the frame and the location of the virtual object with respect to the selected stored coordinate frame, and at least in part, the calculated transformation and the received location of the virtual object. Includes the step of rendering a virtual object onto the display of a portable device at a determined location.

In some embodiments, the step of retrieving the stored coordinate frame comprises the step of retrieving the coordinate frame through an application programming interface (API).

In some embodiments, the portable device comprises a first portable device comprising a first processor of one or more processors. The system further comprises a second portable device comprising a second processor of one or more processors. The processors on each of the first and second devices get the same stored coordinate frame and calculate the conversion between the coordinate frame local to the individual device and the acquired identical stored coordinate frame. It receives the specifications of the virtual object and renders the virtual object on a separate display.

In some embodiments, the first and second devices are each configured to output a plurality of camera images, a camera, and a plurality of camera images to be converted into a plurality of keyframes. , A keyframe generator, and a persistent pose calculator configured to generate a sustained pose by averaging multiple keyframes, and a tracking map that transforms the tracking map into a persistent pose and relative to the origin of the tracking map. A tracking map and a persistent pose converter configured to determine a sustained pose, and a sustained pose and continuous coordinate frame (PCF) converter configured to convert a sustained pose to a PCF. , A map issuer configured to transmit spatial information including PCF to the server.

In some embodiments, the method further comprises running an application to generate a virtual object specification and a virtual object location relative to a selected stored coordinate frame.

In some embodiments, the step of maintaining a coordinate frame local to the portable device on the portable device is for each of the first and second portable devices, from one or more sensors of the portable device, for a 3D environment. A step of capturing multiple images, a step of at least partially calculating one or more persistent postures based on multiple images, and at least a partial calculation of one or more persistent postures. Based on, it includes a step of generating spatial information about the 3D environment. The method further comprises a step of transmitting the generated spatial information to a remote server for each of the first and second portable devices, the step of acquiring the stored coordinate frame remotely. Includes steps to receive from the server.

In some embodiments, at least in part, the step of calculating one or more persistent postures based on a plurality of images is a step of extracting one or more features from each of the plurality of images and one. A step to generate a descriptor for each of one or more features, a step to generate a keyframe based on the descriptor, at least partially for each image, and at least partially one or more keys. Includes steps to generate one or more sustained postures based on the frame.

In some embodiments, the step of generating one or more sustained postures is a step of selectively generating a sustained posture based on a portable device that travels a predetermined distance from the location of the other persistent posture. including.

In some embodiments, the first and second devices each include a download system configured to download stored coordinate frames from a server.

Some embodiments relate to electronic systems for maintaining persistent spatial information about a 3D environment in order to render virtual content on each of a plurality of portable devices. Electronic systems include networked computing devices. Networked computing devices can be executed using at least one processor, at least one storage device connected to the processor, and at least one processor, from portable devices of multiple portable devices. A map storage routine that receives a plurality of maps and stores map information on at least one storage device, wherein each of the received maps has at least one coordinate frame, and at least a map storage routine. It can be executed using one processor, receives location information from portable devices of multiple portable devices, selects one or more maps from the stored maps, and is portable of multiple portable devices. Information from one or more selected maps is transmitted to the device, and the transmitted information includes a map transmitter for comprising a coordinate frame of the map of one or more selected maps.

In some embodiments, the coordinate frame comprises a computer data structure. Computer data structures include coordinate frames with information that characterize multiple features of objects in a 3D environment.

In some embodiments, the information features a descriptor that characterizes a region of the 3D environment.

In some embodiments, each coordinate frame of at least one coordinate frame comprises a persistent point characterized by features detected in sensor data representing a 3D environment.

In some embodiments, each coordinate frame of at least one coordinate frame comprises a sustained posture.

In some embodiments, each coordinate frame of at least one coordinate frame comprises a persistent coordinate frame.

The above description is provided as an illustration and is not intended to be limiting.

The attached drawings are not intended to be drawn to scale. In the drawings, each same or nearly identical component illustrated in the various figures is represented by similar numbers. For clarity purposes, not all components are labeled in all drawings.

FIG. 1 is a sketch illustrating an example of a simplified augmented reality (AR) scene with some embodiments.

FIG. 2 is a sketch of an exemplary simplified AR scene showing an exemplary use case of an XR system, according to some embodiments.

FIG. 3 illustrates a data flow for a single user in an AR system configured to provide the user with an experience of AR content that interacts with the physical world, according to some embodiments. It is a figure.

FIG. 4 is a schematic diagram illustrating an exemplary AR display system that displays virtual content for a single user, according to some embodiments.

FIG. 5A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.

FIG. 5B is a schematic diagram illustrating a visual optics assembly and ancillary components according to some embodiments.

FIG. 6A is a schematic diagram illustrating an AR system using a world reconstruction system according to some embodiments.

FIG. 6B is a schematic diagram illustrating the components of an AR system that maintain a passable world model, according to some embodiments.

FIG. 7 is a schematic of a tracking map formed by a device traversing a path through the physical world.

FIG. 8 is a schematic diagram illustrating a user of a cross-reality (XR) system that perceives virtual content, according to some embodiments.

FIG. 9 is a block diagram of the components of the first XR device of the XR system of FIG. 8 that transforms between coordinate systems, according to some embodiments.

FIG. 10 is a schematic diagram illustrating an exemplary conversion of an origin coordinate frame to a destination coordinate frame for correctly rendering local XR content according to some embodiments.

FIG. 11 is an upper plan view illustrating a pupil-based coordinate frame according to some embodiments.

FIG. 12 is an upper plan view illustrating a camera coordinate frame, including all pupil positions, according to some embodiments.

FIG. 13 is a schematic diagram of the display system of FIG. 9 according to some embodiments.

FIG. 14 is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and the binding of XR content to a PCF according to some embodiments.

FIG. 15 is a flow chart illustrating how to establish and use a PCF according to some embodiments.

FIG. 16 is a block diagram of the XR system of FIG. 8 comprising a second XR device, according to some embodiments.

FIG. 17 is a schematic diagram illustrating a room and key frames established for various areas within the room, according to some embodiments.

FIG. 18 is a schematic diagram illustrating the establishment of a sustained posture based on key frames by some embodiments.

FIG. 19 is a schematic diagram illustrating the establishment of a sustained coordinate frame (PCF) based on a sustained posture according to some embodiments.

20A-20C are schematic views illustrating examples of creating a PCF according to some embodiments.

FIG. 21 is a block diagram illustrating a system for generating global descriptors for individual images and / or maps, according to some embodiments.

FIG. 22 is a flowchart illustrating a method of calculating an image descriptor according to some embodiments.

FIG. 23 is a flowchart illustrating a method of locating using an image descriptor according to some embodiments.

FIG. 24 is a flow chart illustrating a method of training a neural network according to some embodiments.

FIG. 25 is a block diagram illustrating a method of training a neural network according to some embodiments.

FIG. 26 is a schematic diagram illustrating an AR system configured to rank and merge multiple environmental maps, according to some embodiments.

FIG. 27 is a simplified block diagram illustrating a plurality of reference maps stored on a remote storage medium, according to some embodiments.

FIG. 28 illustrates how, according to some embodiments, a canonical map is selected, eg, a new tracking map is located within one or more canonical maps, and / or a PCF is obtained from the canonical map. It is a schematic diagram.

FIG. 29 is a flowchart illustrating how to select a plurality of ranked environment maps according to some embodiments.

FIG. 30 is a schematic diagram illustrating an exemplary map ranking portion of the AR system of FIG. 26, according to some embodiments.

FIG. 31A is a schematic diagram illustrating examples of area attributes of a tracking map (TM) and an environment map in a database according to some embodiments.

FIG. 31B is a schematic diagram illustrating an example of determining the geographic location of the tracking map (TM) for geographic location filtering of FIG. 29, according to some embodiments.

FIG. 32 is a schematic diagram illustrating an embodiment of the geographic location filtering of FIG. 29 according to some embodiments.

FIG. 33 is a schematic diagram illustrating an example of Wi-Fi BSSID filtering of FIG. 29 according to some embodiments.

FIG. 34 is a schematic diagram illustrating an embodiment of the positioning of FIG. 29 according to some embodiments.

35 and 36 are block diagrams of an XR system configured to rank and merge multiple environmental maps, according to some embodiments. 35 and 36 are block diagrams of an XR system configured to rank and merge multiple environmental maps, according to some embodiments.

FIG. 37 is a block diagram illustrating a method of creating an environmental map of the physical world in a canonical form, according to some embodiments.

38A and 38B are schematics illustrating an environmental map created in a canonical form by updating the tracking map of FIG. 7 with a new tracking map, according to some embodiments. 38A and 38B are schematics illustrating an environmental map created in a canonical form by updating the tracking map of FIG. 7 with a new tracking map, according to some embodiments.

39A-39F are schematic diagrams illustrating examples of merging maps according to some embodiments. 39A-39F are schematic diagrams illustrating examples of merging maps according to some embodiments. 39A-39F are schematic diagrams illustrating examples of merging maps according to some embodiments. 39A-39F are schematic diagrams illustrating examples of merging maps according to some embodiments. 39A-39F are schematic diagrams illustrating examples of merging maps according to some embodiments. 39A-39F are schematic diagrams illustrating examples of merging maps according to some embodiments.

FIG. 40 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1) according to some embodiments, which can be generated by the first XR device of FIG.

FIG. 41 is a block diagram illustrating a step of uploading a map 1 from a first XR device to the server of FIG. 9 according to some embodiments.

FIG. 42 is a schematic diagram illustrating the XR system of FIG. 16 according to some embodiments, wherein the second user is the second user of the XR system after the first user has completed the first session. Indicates that the second session has been started using the XR device of 2.

FIG. 43A is a block diagram illustrating a new session for the second XR device of FIG. 42, according to some embodiments.

FIG. 43B is a block diagram illustrating the creation of a tracking map for the second XR device of FIG. 42, according to some embodiments.

FIG. 43C is a block diagram illustrating a step of downloading a canonical map from a server to the second XR device of FIG. 42, according to some embodiments.

FIG. 44 attempts to locate a second tracking map (Map 2), which may be generated by the second XR device of FIG. 42, for a reference map, according to some embodiments. It is a schematic diagram to be illustrated.

FIG. 45 locates a second tracking map (Map 2) of FIG. 44 with XR content that can be further expanded and associated with the PCF of Map 2 with respect to the reference map according to some embodiments. It is a schematic diagram which illustrates the position identification.

46A-46B are schematic views illustrating the success of mapping Map 2 in FIG. 45 relative to the reference map, according to some embodiments. 46A-46B are schematic views illustrating the success of mapping Map 2 in FIG. 45 relative to the reference map, according to some embodiments.

FIG. 47 is a schematic diagram illustrating a canonical map generated by including one or more PCFs from the canonical map of FIG. 46A within map 2 of FIG. 45, according to some embodiments.

FIG. 48 is a schematic diagram illustrating a canonical map of FIG. 47 with further extensions of map 2 on a second XR device, according to some embodiments.

FIG. 49 is a block diagram illustrating a step of uploading a map 2 from a second XR device to a server, according to some embodiments.

FIG. 50 is a block diagram illustrating a step of merging map 2 and a canonical map according to some embodiments.

FIG. 51 is a block diagram illustrating the transmission of new canonical maps from the server to the first and second XR devices, according to some embodiments.

FIG. 52 is a block diagram illustrating a two-dimensional representation of map 2 and a head coordinate frame of a second XR device referenced in map 2 according to some embodiments.

FIG. 53 is a block diagram illustrating the adjustment of the head coordinate frame, which can occur in 6 degrees of freedom in 2 dimensions, according to some embodiments.

FIG. 54 is a block diagram illustrating a canonical map on a second XR device in which sound is located relative to the PCF of map 2, according to some embodiments.

55 and 56 show, according to some embodiments, of the XR system when the first user ends the first session and the first user starts the second session using the XR system. It is a perspective view and a block diagram illustrating the use. 55 and 56 show, according to some embodiments, of the XR system when the first user ends the first session and the first user starts the second session using the XR system. It is a perspective view and a block diagram illustrating the use.

57 and 58 are perspective views and block diagrams illustrating the use of the XR system when three users simultaneously use the XR system in the same session, according to some embodiments. 57 and 58 are perspective views and block diagrams illustrating the use of the XR system when three users simultaneously use the XR system in the same session, according to some embodiments.

FIG. 59 is a flow chart illustrating a method of restoring and resetting the head posture according to some embodiments.

FIG. 60 is a block diagram of a machine in the form of a computer from which applications can be found within the system of the invention, according to some embodiments.

Described herein are methods and devices for providing an X-reality (XR or cross-reality) scene. In order to provide a realistic XR experience to multiple users, the XR system must know the user's physical surroundings in order to properly correlate the location of the virtual object in relation to the real object. The XR system may construct an environmental map of the scene, which may be created from images and / or depth information collected using sensors that are part of the XR device worn by the user of the XR system.

We have an XR system in which each XR device develops a local map of its physical environment by integrating information from one or more images collected during a point-in-time scan. Realized that it could be beneficial and acknowledged its true value. In some embodiments, the coordinate system of the map is tied to the orientation of the device when the scan is initiated. The orientation allows the user to interact with the XR system regardless of whether different moments are associated with different users, their own wearable device with sensors that each scan the environment, or the same user who uses the same device at different times. It can change from moment to moment as it interacts. We collect for different orientations, where each user device is different for different user instances (eg, time-based snapshots) or system sessions (eg, time between on and off). We have realized a technique for operating the XR system based on sustainable spatial information, which relies only on spatial information, overcomes the limitations of the XR system, and acknowledged its true value. The technique is calculated more for a single or multiple users, for example by allowing persistent spatial information to be created, stored, and read by any of the multiple users of the XR system. It can provide XR scenes for an efficient and immersive experience.

Persistent spatial information may be represented by a persistent map that can enable one or more functions that enhance the XR experience. The persistent map may be stored in a remote storage medium (eg, cloud). For example, a wearable device worn by a user may, after being turned on, read an appropriate stored map previously created and stored from a persistent storage device such as a cloud storage device. Previously stored maps may be based on data about the environment collected using sensors on the user's wearable device during the previous session. Retrieving the stored map may enable the use of the wearable device without scanning the physical world with the sensors on the wearable device. Alternatively, or in addition, the system / device may similarly read out the appropriate stored map as it enters a new realm of the physical world.

The stored map may be represented in a canonical form in which each XR device may be associated with its local reference frame. In a multi-device XR system, the stored map accessed by one device may be created and stored by another device and / or within at least part of the physical world represented by the stored map. It may be constructed by aggregating data about the physical world collected by sensors on multiple wearable devices that previously existed in.

In addition, sharing data about the physical world across multiple devices can enable a shared user experience for virtual content. Both two XR devices with access to the same stored map may be located, for example, relative to the stored map. Once located, the user device renders the virtual content, which has a location defined by the criteria, into a stored map by translating that location to a frame or reference maintained by the user device. You may. The user device may use this local reference frame to control the display of the user device and render the virtual content within a defined location.

To support these and other features, the XR system is a persistent space containing one or more stored maps based on data about the physical world collected using sensors on user devices. It may include components that deploy, maintain, and use information. These components may be distributed across the XR system, some of which operate, for example, on a head-mounted portion of the user device. Other components may run on the computer associated with the user, which is coupled to the head-mounted portion via a local or personal area network. Others may operate at remote locations, such as one or more servers accessible via a wide area network.

These components identify, for example, information about the physical world collected by one or more user devices that is of sufficient quality to be stored as a persistent map or within a persistent map. Obtain, may include components. An example of such a component, described in more detail below, is the map merge component. Such components may, for example, receive input from the user device and determine the aptitude of the portion of the input used to update the persistent map. The map merge component, for example, splits a local map created by a user device into parts, determines the mergeability of a persistent map with one or more of the parts, and sets a certified mergeability criterion. You may merge the fills into a persistent map. The map merge component may also be promoted, for example, so that the persistent map and the unmerged portion are separate persistent maps.

As another embodiment, these components may include components that can be read and used by the user device to assist in determining the appropriate persistent map. An example of such a component, described in more detail below, is a map ranking component. Such components may, for example, receive input from a user device and identify one or more persistent maps that are likely to represent areas of the physical world in which the device is operating. good. The map ranking component may help select a persistent map to be used by its local device, for example, when rendering virtual content, collecting data about the environment, or performing other actions. The map ranking component may, as an alternative, or in addition, help identify a persistent map to be updated as additional information about the physical world is collected by one or more user devices.

Yet another component may decide to transform the information captured or described in relation to one reference frame into another reference frame. For example, the sensor may be attached to a head-mounted display so that the data read from the sensor indicates the location of the object in the physical world with respect to the wearer's head posture. One or more transformations may be applied to relate the location information to the coordinate frame associated with the sustained environment map. Similarly, when represented within a coordinate frame of a persistent environment map, the data indicates where the virtual object should be rendered, and the data is one or more transformations to be within the reference frame of the display on the user's head. You may receive. There may be multiple such transformations, as described in more detail below. These transformations may be partitioned across the components of the XR system so that they can be updated and / or applied efficiently within the distributed system.

In some embodiments, the persistent map may be constructed from information collected by multiple user devices. The XR device may use the information collected by the respective sensors of the XR device at various locations and times to capture local spatial information and build a separate tracking map. Each tracking map may contain points that may contain multiple features and may be associated with features of a real object. Potentially, in addition to the steps of supplying input and creating and maintaining a persistent map, tracking maps are used to track the user's movements in the scene, and the XR system is individualized based on the tracking map. It may be possible to estimate the head posture of the user.

This co-dependence between mapping and estimating head posture constitutes a significant task. Substantial processing may be required at the same time to create a map and estimate head posture. Waiting times make the XR experience less realistic for the user, so the process moves the object in the scene (eg, moving the cup on the table) and the user in the scene. As it goes, it must be carried out quickly. On the other hand, the weight of the XR device should be light for the user to wear comfortably, so the XR device may provide limited computational resources. The lack of computational resources cannot be compensated for with more sensors due to the addition of sensors, which would undesirably also add weight. In addition, either more sensors or more computational resources lead to heat, which can cause deformation of the XR device.

We have estimated head posture at a frequency of 1 kHz, eg, at 1 kHz, a low usage of computational resources associated with XR devices, which may be configured with four videographic array (VGA) cameras operating at 30 Hz. XR for a more immersive user experience, such as one inertial measurement unit (IMU) operating, calculated power of a single advanced RISC machine (ARM) core, less than 1GB of memory, and less than 100Mbp of network bandwidth. He realized the technique for operating the system and providing the XR scene, and recognized its true value. These techniques relate to the steps of generating and maintaining a map, reducing the processing required to estimate head posture, and providing and consuming data with low computational overhead.

These techniques allow the XR system to (1) patch-based tracking of distinguishable points between continuous images in an environment (eg, frame / frame tracking) and (2) focus on the current image. Hybrid tracking may be included to take advantage of both descriptor-based map matching of known real-world locations of points (eg, map / frame tracking). In frame / frame tracking, the XR system may track specific points of interest (eg, prominent points) such as angles between captured images in the real world environment. For example, the display system may identify the location of a visual point of interest in the current image that was included (eg, located in) in the previous image. The identification may be performed, for example, using a photometric error minimization process. In map / frame tracking, the XR system may access map information that indicates the real-world location of the point of interest and match the point of interest contained in the current image with the point of interest shown in the map information. Information about the point of interest may be stored in the map database as a descriptor. The XR system may calculate its orientation based on the matched visual features. U.S. Patent Application No. 16 / 221,065 is incorporated herein by reference in its entirety by describing and referencing hybrid tracking.

These techniques are used when building a map using a set of mapped points and keyframes, and / or by dividing the map into blocks and enabling block updates, etc. It may include steps to reduce the amount of data processed. The mapped points may be associated with points of interest in the environment. The key frame may contain information selected from the camera captured data. U.S. Patent Application No. 16 / 520,582 is incorporated herein by reference in its entirety by describing and referring to the steps of determining and / or evaluating a location map.

In some embodiments, persistent spatial information may be represented in a way that can be easily shared between users and among distributed components, including applications. Information about the physical world may be represented, for example, as a persistent coordinate frame (PCF). The PCF may be defined on the basis of one or more points representing features recognized within the physical world. The features may be selected so that they are likely to be the same for each user session of the XR system. PCFs may be sparsely present and provide less than all of the available information about the physical world so that they can be processed and transferred efficiently. Techniques for processing persistent spatial information include the steps of creating a dynamic map based on one or more coordinate systems in real space across one or more sessions, and, for example, an application programming interface (application programming interface). It may include a step of generating a sustained coordinate frame (PCF) over a sparse map that can be exposed to an XR application via API). These capabilities may be supported by techniques for ranking and merging multiple maps created by one or more XR devices. Persistent spatial information may also allow for rapid restoration and reset of head posture on each of one or more XR devices in a computationally efficient manner.

In addition, the technique can enable efficient comparison of spatial information. In some embodiments, the image frame may be represented by a numeric descriptor. The descriptor may be calculated via a transformation that maps the set of features identified in the image to the descriptor. The transformation may be performed within a trained neural network. In some embodiments, the set of features, supplied as input to the neural network, was extracted from the image using a technique that preferentially selects features that are likely to be persistent, for example. , May be a filtered set of features.

Representation of an image frame as a descriptor enables, for example, efficient matching of new image information with stored image information. The XR system may store the descriptors of one or more frames in the lower layer of the persistent map together with the persistent map. Local image frames obtained by the user device may be converted to such descriptors as well. By selecting a stored map with a descriptor similar to that of a local image frame, one or more persistent maps that are likely to represent the same physical space as the user device will be processed in a relatively small amount. Can be selected with. In some embodiments, descriptors are calculated for keyframes in local and persistent maps, which may further reduce processing when comparing maps. Such an efficient comparison is, for example, finding a persistent map to find a persistent map to load or update within the local device, based on image information obtained using the local device. May be used to simplify.

The techniques described herein are with and for many types of devices, including wearable or portable devices, with limited computational resources that provide extended or mixed reality situations. , Together or separately. In some embodiments, the technique may be implemented by one or more services that form part of the XR system.

AR system overview

FIGS. 1 and 2 illustrate scenes with virtual content that are displayed in conjunction with a portion of the physical world. For purposes of illustration, an AR system is used as an embodiment of an XR system. FIG. 3-6B illustrates an exemplary AR system that includes one or more processors, a memory, a sensor, and a user interface that can operate according to the techniques described herein.

Referring to FIG. 1, an outdoor AR scene 354 is depicted, and users of AR technology see a physical world park-like setting 356 featuring people, trees, buildings in the background, and a concrete platform 358. .. In addition to these items, users of AR technology also have a robot image 357 standing on the physical world concrete platform 358 and a flying cartoon-like avatar character 352, thereby looking like anthropomorphic bumblebees. Although they perceive "visible", these elements (eg, avatar character 352 and robot image 357) do not exist in the physical world. AR technology that promotes the comfortable, natural and rich presentation of virtual image elements among other virtual or physical world image elements due to the extreme complexity of human visual perception and nervous system. Is difficult to produce.

Such an AR scene allows you to place the AR content in the physical world, determines the location in the map of the physical world where the AR content is installed, and the installed AR content is, for example. Based on tracking information, it saves AR scenes and allows multiple users to share their AR experience, so that they can be reloaded for viewing within the physical world during different AR experience sessions. It can be achieved using a system that builds a map of the physical world. The system may build and update digital representations of the physical world surface around the user. This representation is used to place virtual objects, in physics-based interactions, and for virtual character path planning and navigation, or for other actions in which information about the physical world is used. , Completely or partially, may be used to render the virtual content so that it appears to be included by the physical object between the user and the rendered location of the virtual content.

FIG. 2 illustrates another embodiment of the indoor AR scene 400 according to some embodiments and shows an exemplary use case of an XR system. An exemplary scene 400 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp at the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, users of AR technology also have images on the wall behind the sofa, birds flying through the door, deer looking out from the bookshelf, and a windmill installed on a coffee table. Perceive virtual objects such as ornaments in the form of.

For images on walls, AR technology requires information not only about the surface of the wall, but also about objects and surfaces in the room, such as lamp shapes, that occlude the image in order to render the virtual object correctly. For flying birds, AR technology uses realistic physics to render the bird, avoiding objects and surfaces, or bouncing when the bird collides, with all objects around the room. And request information about the surface. With respect to deer, AR technology requires information about surfaces such as floors or coffee tables to calculate where deer should be placed. With respect to the wind turbine, the system can identify that it is an object separate from the table and can determine that it is mobile, while the corners of the shelves or the corners of the walls can be determined to be stationary. Such specificity may be used in determining the part of the scene to be used or updated in each of the various actions.

The virtual object may be installed in the previous AR experience session. When a new AR experience session begins in the living room, AR technology requires that virtual objects be accurately displayed at previously installed locations and realistically visible from different perspectives. For example, the windmill should appear to be standing on the book rather than drifting above the table, even in different places without the book. Such drifting can occur if the user's location for a new AR experience session is not accurately located in the living room. As another embodiment, if the user is viewing the wind turbine from a different perspective than when the wind turbine was installed, AR technology requires the corresponding side of the displayed wind turbine.

The scene may be presented to the user via a system that includes a plurality of components, including a user interface, which may stimulate one or more user perceptions such as visual, auditory, and / or tactile sensations. In addition, the system may include one or more sensors capable of measuring parameters of the physical part of the scene, including the position and / or movement of the user within the physical part of the scene. Further, the system may include one or more computing devices with associated computer hardware such as memory. These components may be integrated within a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into the wearable device.

FIG. 3 depicts an AR system 502 configured to provide an experience of AR content that interacts with the physical world 506, according to some embodiments. The AR system 502 may include a display 508. In the illustrated embodiment, the display 508 may be worn by the user as part of a headset so that the user can wear the display across his eyes, such as a pair of goggles or eyeglasses. At least a portion of the display may be transparent so that the user can observe the see-through reality 510. The see-through reality 510 may correspond to a portion of the physical world 506 within the current viewpoint of the AR system 502, which allows the user to wear a headset and physically incorporate both the display and the sensor of the AR system. It can correspond to the user's point of view when getting information about the world.

AR content may also be presented on display 508, which is overlaid on see-through reality 510. To provide the exact interaction between AR content and see-through reality 510 on display 508, AR system 502 also includes sensor 522, which is configured to capture information about the physical world 506. good.

The sensor 522 may include one or more depth sensors that output the depth map 512. Each depth map 512 may have a plurality of pixels that may represent the distance to a surface within the physical world 506 in a particular direction with respect to the depth sensor. Raw depth data can be generated from the depth sensor to create a depth map. Such depth maps can be updated fast enough for the depth sensor to form new images, which can be hundreds or thousands of times / second. However, the data may have holes, which are noisy and incomplete and are shown as black pixels on the illustrated depth map.

The system may include other sensors such as image sensors. The image sensor may obtain monocular or stereoscopic information that can be otherwise processed to represent the physical world. For example, the image may be processed within the World Reconstruction Component 516 to create a mesh that represents the connected parts of the object in the physical world. For example, metadata about such objects, including colors and surface textures, may also be obtained using sensors and stored as part of the world reconstruction.

The system may also obtain information about the user's head posture (or "posture") with respect to the physical world. In some embodiments, the head posture tracking component of the system may be used to calculate head posture in real time. Head posture tracking components include, for example, translation in three vertical axes (eg, front / back, up / down, left / right) and rotation around the three vertical axes (eg, pitch, yaw, and roll). ), With 6 degrees of freedom, may represent the user's head orientation within the coordinate frame. In some embodiments, the sensor 522 may include an inertial measurement unit that can be used to calculate and / or determine the head posture 514. The head posture 514 for a depth map may indicate, for example, the current viewpoint of a sensor that captures the depth map, with 6 degrees of freedom, while the head posture 514 makes the image information a specific part of the physical world. It may be used for other purposes such as relating to or relating the position of the display mounted on the user's head to the physical world.

In some embodiments, the head posture information may be derived by methods other than IMU, such as from analysis of objects in the image. For example, the head posture tracking component may calculate the relative position and orientation of the AR device with respect to the physical object based on the visual information captured by the camera and the inertial information captured by the IMU. The head posture tracking component may then calculate the head posture of the AR device, for example, by comparing the calculated relative position and orientation of the AR device to the physical object with the characteristics of the physical object. .. In some embodiments, the comparison is stable over time so that changes in the position of these features within the captured image over time can be associated with changes in the user's head posture, sensor 522. It may be done by identifying features in the captured image using one or more of them.

In some embodiments, the AR device builds a map from feature points that are recognized within a series of images captured as the user travels with the AR device throughout the physical world. May be good. Each image frame can be obtained from different poses as the user moves, but the system aligns the features of each continuous image frame by matching the features of the continuous image frame with the previously captured image frame. It may be adjusted to match the orientation of the initial image frame. Translation of continuous image frames aligns each continuous image frame and was previously processed so that points representing identical features would match the corresponding feature points from previously collected image frames. It can be used to match the orientation of the image frame. The resulting frames in the map may have the common orientation established when the first image frame is added to the map. The map, along with a set of feature points within a common reference frame, may be used to determine the posture of the user within the physical world by matching features from the current image frame to the map. In some embodiments, the map may be referred to as a tracking map.

In addition to enabling tracking of the user's attitude within the environment, this map may allow other components of the system, such as the World Reconstruction Component 516, to determine the location of physical objects for the user. The world reconstruction component 516 may receive depth map 512 and head posture 514 and any other data from the sensor and integrate that data into the reconstruction 518. Reconstruction 518 may be more complete and less noisy than the sensor data. The world reconstruction component 516 may update the reconstruction 518 using the spatial and temporal averages of the sensor data from multiple perspectives over time.

Reconstruction 518 may include representations of the physical world in one or more data formats, including, for example, voxels, meshes, planes, and the like. Different formats can represent alternative representations of the same part of the physical world, or can represent different parts of the physical world. In the illustrated embodiment, on the left side of the reconstructed structure 518, a part of the physical world is presented as a global surface, and on the right side of the reconstructed structure 518, a part of the physical world is presented as a mesh. ..

In some embodiments, the map maintained by the head posture component 514 may be isolated from other maps that may be maintained in the physical world. Rather than providing information about locations and possibly other properties of the surface, sparse maps may indicate points of interest and / or structural locations such as corners or edges. In some embodiments, the map may include image frames such as those captured by sensor 522. These frames can be reduced to features that can represent points of interest and / or structure. Along with each frame, information about the posture of the user from which the frame was obtained may also be stored as part of the map. In some embodiments, all images obtained by the sensor may or may not be stored. In some embodiments, the system may process the images as they are collected by the sensor and select a subset of image frames for further calculation. The choice may be based on one or more criteria that limit the addition of information, but ensure that the map contains useful information. The system is based on image frames, for example, based on overlapping with previous image frames already added to the map, or containing a sufficient number of features determined to represent stationary objects. You may add a new image frame to the map. In some embodiments, the selected image frame or the set of features from the selected image frame can serve as a keyframe for the map, which is used to provide spatial information. To.

The AR system 502 may integrate sensor data from multiple viewpoints in the physical world over time. The posture of the sensor (eg, position and orientation) may be tracked as the device containing the sensor is moved. As we understand the frame orientation of the sensor and how it relates to other attitudes, each of these multiple perspectives of the physical world is within a single combined reconstruction of the physical world. May be fused together, which can serve as an abstraction layer for the map and provide spatial information. The reconstruction can be more complete and less noisy than the original sensor data by using spatial and temporal averaging (ie, averaging of data from multiple perspectives over time) or any other suitable method.

In the embodiment illustrated in FIG. 3, the map represents a part of the physical world in which a user of a single wearable device resides. In that scenario, the head posture associated with the frame in the map may be represented as a local head posture that indicates the orientation with respect to the initial orientation for a single device at the start of the session. For example, the head posture is tracked against the initial head posture when the device is turned on or otherwise operated to scan the environment and construct a representation of that environment. May be good.

In combination with the content that characterizes that part of the physical world, the map may contain metadata. It may indicate the capture time of the metadata, eg, the sensor information used to form the map. The metadata may, as an alternative, or in addition, indicate the location of the sensor in the capture time of the information used to form the map. The location indicates the strength of the signal received from one or more wireless access points while the sensor data was being collected, directly using information such as from a GPS chip, or indirectly, the Wi-Fi signature. Etc. and / or using the BSSID of the wireless access point to which the user device is connected to the sensor data while it was collected.

Reconstruction 518 may be used for AR functions such as the production of surface representations of the physical world for occlusion processing or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of the reconstruction 518 may be used, for example, by component 520, which produces a variable global surface representation in world coordinates that can be used by other components.

The AR content may be generated by the AR application 504 or the like based on this information. The AR application 504 may be a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interactions, and environmental inference. It may perform these functions by querying data in different formats from the rebuild 518 produced by the world rebuild component 516. In some embodiments, component 520 may be configured to output updates as the representation within the region of interest in the physical world changes. The region of interest is set to approximate (predict) a portion of the physical world within the user's vicinity of the system, such as a portion of the user's field of view, or projected into the user's field of view. / Decided).

The AR application 504 may use this information to generate and update AR content. The virtual portion of the AR content may be presented on the display 508 in combination with the see-through reality 510 to create a realistic user experience.

In some embodiments, the AR experience may include remote processing and / or remote data storage, a wearable display device that may be part of the system, and / or, in some embodiments, other users. It may be provided to the user through an XR device, which may be another wearable display device worn by. FIG. 4 illustrates an example of a system 580 (hereinafter referred to as "system 580"), including a single wearable device, for convenience of illustration. The system 580 includes a head-mounted display device 562 (hereinafter referred to as "display device 562") and various mechanical and electronic modules and systems that support the functions of the display device 562. The display device 562 may be coupled to the frame 564, which is wearable by a user of the display system or a viewer 560 (hereinafter referred to as "user 560"), and the display device 562 is attached to the user 560. It is configured to be positioned in front of the eye. According to various embodiments, the display device 562 may be a sequential display. The display device 562 may be monocular or binocular. In some embodiments, the display device 562 may be an embodiment of the display 508 in FIG.

In some embodiments, the speaker 566 is coupled to the frame 564 and positioned in close proximity to the user 560's ear canal. In some embodiments, another speaker, not shown, is positioned adjacent to another ear canal of the user 560 to provide stereo / adjustable sound control. The display device 562 is operably coupled to the local data processing module 570 by a wired lead or wireless connectivity 568 or the like, which is secured to a helmet or hat worn by the user 560, which is secured and attached to the frame 564. It may be mounted in a variety of configurations, such as being mounted, built into headphones, or otherwise removably mounted by the user 560 (eg, in a backpack configuration, in a belt coupling configuration).

The local data processing module 570 may include a processor and digital memory such as non-volatile memory (eg, flash memory), both of which can be utilized to assist in data processing, caching, and storage. The data is operably coupled to a) an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope (eg, frame 564). Data captured from a sensor (or otherwise attached to user 560), and / or b) potentially for passage to display device 562 after processing or reading, remote processing module 572 and / or remote data. Contains data obtained and / or processed using repository 574.

In some embodiments, the wearable device may communicate with a remote component. The local data processing module 570 can operate the remote processing modules 572 and the remote data repository 574 with each other by the communication link 576 and 578, respectively, via a wired or wireless communication link or the like. It may be operably combined so that it is combined with and available as a resource to the local data processing module 570. In some embodiments, the head posture tracking components described above may be implemented, at least in part, within the local data processing module 570. In some embodiments, the world reconstruction component 516 in FIG. 3 may be implemented, at least in part, within the local data processing module 570. For example, the local data processing module 570 may be configured to execute computer executable instructions and generate maps and / or physical world representations, at least in part, based on at least a portion of the data.

In some embodiments, processing may be distributed across local and remote processors. For example, local processing may be used to build a map (eg, a tracking map) on a user device based on sensor data collected using the sensors on that user's device. Such maps may be used by applications on the user's device. In addition, previously created maps (eg, canonical maps) may be stored in the remote data repository 574. Suitable stored or persistent maps may be used in place of or in addition to locally created tracking maps on the device, if available. In some embodiments, the tracking map is oriented with respect to the tracking map and one or more persistent features, where the correspondence can be oriented with respect to the position of the wearable device at the time the user turns on the system. Obtained, may be located relative to the stored map so that it is established between the reference map and the obtained. In some embodiments, the persistent map is loaded onto the user device and the user device scans the location to build a complete environment tracking map of the user from the sensor data obtained during the scan. It may be possible to render virtual content without the delay associated with. In some embodiments, the user device may access a remote persistent map (eg, stored in the cloud) without having to download the persistent map onto the user device.

Alternatively, or in addition, tracking maps may be merged with previously stored maps to extend or improve their quality. The process for deciding whether a suitable previously created environment map is available and / or whether to merge the tracking map with one or more stored environment maps is the local data processing module 570 or It may be done in the remote processing module 572.

In some embodiments, the local data processing module 570 may include one or more processors (eg, a graphics processing unit (GPU)) configured to analyze and process data and / or image information. .. In some embodiments, the local data processing module 570 may include a single processor (eg, a single-core or multi-core ARM processor), which limits the computational budget of the local data processing module 570, but is smaller. Will enable the device. In some embodiments, the World Reconstruction Component 516 is a single altitude RISC machine (so that the remaining calculated budget of a single ARM core can be accessed for other uses, such as mesh extraction). The physical world representation may be generated in real time on a non-predetermined space using less computational budget than the ARM) core.

In some embodiments, the remote data repository 574 may include digital data storage equipment, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local data processing module 570, allowing for fully autonomous use from remote modules. In some embodiments, all data is stored and all or most of the calculations are performed within the remote data repository 574, allowing for smaller devices. The World Reconstruction may be stored, for example, in whole or in part within the Repository 574.

In embodiments, the data is stored remotely and accessible over a network, the data may be shared by multiple users of the augmented reality system. For example, the user device may upload its tracking map and extend it into the database of environment maps. In some embodiments, the tracking map upload occurs at the end of the user session with the wearable device. In some embodiments, the tracking map upload is persistent, semi-persistent, intermittent, at a predefined time, after a predefined cycle from the previous upload, or by an event. Can occur when triggered. Tracking maps uploaded by any user device may be used to extend or improve previously stored maps, whether based on data from that user device or any other user device. good. Similarly, the persistent map downloaded to a user device may be based on data from that user device or any other user device. Thus, high quality environmental maps may be readily available to users to improve their experience with AR systems.

In some embodiments, the local data processing module 570 is operably coupled to the battery 582. In some embodiments, the battery 582 is a removable power source such as a commercially available battery. In another embodiment, the battery 582 is a lithium ion battery. In some embodiments, the battery 582 is a system over a longer time cycle in which the user 560 is connected to a power source and does not need to charge the lithium ion battery or shut off the system 580 and replace the battery. Both an internal lithium-ion battery rechargeable by the user 560 and a removable battery are included so that the 580 can be operated during the non-operating time of the system 580.

FIG. 5A illustrates a user 530 wearing an AR display system that renders AR content as the user 530 travels through the physical world environment 532 (hereinafter referred to as "environment 532"). Information captured by the AR system along the user's travel path may be processed into one or more tracking maps. User 530 positions the AR display system at position 534, and the AR display system is in the physical world that can be stored and updated as the real object changes in the passable world for position 534 (eg, in the physical world). Record surrounding information (digital representation of a real object). The information may be stored as poses in combination with images, features, directional audio inputs, or other desired data. Location 534 is aggregated for data entry 536 and processed by at least passable worlds module 538, for example as part of a tracking map, which is implemented, for example, by processing on remote processing module 572 in FIG. May be. In some embodiments, the passable world module 538 indicates the location of an object in the physical world where the processed information is combined with other information about the physical object used in the rendered virtual content. As obtained, the head orientation component 514 and the world reconstruction component 516 may be included.

The passable world module 538, at least in part, determines where and how the AR content 540 can be placed within the physical world, as determined from the data entry 536. AR content is "placed" within the physical world by presenting both a representation of the physical world and AR content through the user interface, and the AR content interacts with objects within the physical world. Objects in the physical world are presented as if the AR content, when appropriate, obscures the user's view of those objects. In some embodiments, the AR content is by appropriately selecting a portion of the fixed element 542 (eg, the table) from the rebuild (eg, the rebuild 518) and determining the shape and position of the AR content 540. , May be installed. As an embodiment, the fixed element may be a table and the virtual content may be positioned to appear on the table. In some embodiments, the AR content may be placed within a structure within the field of view 544, which may be the current field of view or the estimated future field of view. In some embodiments, the AR content may be sustained against model 546 (eg, mesh) of the physical world.

As depicted, the fixed element 542 is a passable world so that the user 530 can perceive content on the fixed element 542 without the need for the system to map to the fixed element 542 each time it is visible to the user 530. It acts as a proxy (eg, a digital copy) for any fixed element in the physical world that can be stored within module 538. The fixed element 542 is therefore a mesh model stored by passable worlds module 538 for future references by multiple users, although determined from a previous modeling session or from a separate user. May be. Therefore, the passable world module 538 recognizes the environment 532 from the previously mapped environment and displays and calculates the AR content without the user 530's device initially mapping all or part of the environment 532. And can save cycles and avoid waiting times for any rendered AR content.

The physical world mesh model 546 may be created by the AR display system, and the appropriate surfaces and metrics to interact with and display the AR content 540 require full or partial remodeling. Instead, it can be stored by passable worlds module 538 for future readings by user 530 or other users. In some embodiments, the data entry 536 was finally installed on the passable worlds module 538, which fixed element 542 of one or more fixed elements is available, on the fixed element 542. To indicate whether AR Content 540 and its identical content should be displayed (such AR Content is "persistent" content regardless of whether the user is viewing a particular passable world model). , Geographic location, user identification, and current activity input.

Even in embodiments where the object is considered fixed (eg, a kitchen table), the passable world module 538 takes into account the potential for changes in the physical world within the model of the physical world. You may update those objects at any time. Models of fixed objects may be updated very infrequently. Other objects in the physical world can be those that are not considered to be moving or otherwise fixed (eg, kitchen chairs). In order to render the AR scene realistically, the AR system may update the positions of these non-fixed objects much more frequently than those used to update the fixed objects. To enable accurate tracking of all objects in the physical world, the AR system may extract information from multiple sensors, including one or more image sensors.

FIG. 5B is a schematic illustration of the visual optics assembly 548 and ancillary components. In some embodiments, two eye tracking cameras 550 are directed towards the user's eye 549 and are metrics of the user's eye 549, such as eye shape, eyelid occlusion, pupil direction, and flash on the user's eye 549. Is detected.

In some embodiments, one of the sensors is a depth sensor 551, such as a flight time sensor, that emits signals to the world, detects reflections of those signals from nearby objects, and is given. You may determine the distance to the object in. The depth sensor can quickly determine, for example, whether an object has entered the user's field of view as a result of either the movement of those objects or a change in the user's posture. However, information about the position of the object in the user's field of view may be collected as an alternative or in addition using other sensors. Depth information may be obtained from, for example, a stereoscopic image sensor or a prenoptic sensor.

In some embodiments, the world camera 552 records, maps, and / or otherwise models the environment 532 and detects inputs that can affect AR content. .. In some embodiments, the world camera 552 and / or camera 553 may be a grayscale and / or color image sensor, which outputs the grayscale and / or color image frame at fixed time intervals. You may. Camera 553 may also capture a physical world image in the user's field of view at a specific time. Pixels in a frame-based image sensor may be iteratively sampled, even if their values are invariant. The world camera 552, camera 553, and depth sensor 551 have separate fields of view of 554, 555, and 556, respectively, and collect data from physical world scenes such as the physical world environment 532 depicted in FIG. 34A. And record.

The inertial measurement unit 557 may determine the movement and orientation of the visual optics assembly 548. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 551 is operably coupled to the eye tracking camera 550 as a confirmation of the measured accommodation for the actual distance the user's eye 549 is looking at.

It should be appreciated that the visual optics assembly 548 may include some of the components illustrated in FIG. 34B and may include components in place of or in addition to the components illustrated. In some embodiments, for example, the visual optics assembly 548 may include two world cameras 552 instead of four. Alternatively, or in addition, the cameras 552 and 553 do not need to capture a visible light image of their full field of view. The visual optics assembly 548 may include other types of components. In some embodiments, the visual optics assembly 548 may include one or more dynamic visual sensors (DVS), the pixels of which respond asynchronously to a relative change in light intensity exceeding a threshold. You may.

In some embodiments, the visual optics assembly 548 may not include a depth sensor 551 based on flight time information. In some embodiments, for example, the visual optics assembly 548 may include one or more plenoptic cameras, from which pixels may capture the light intensity and angle of incident light. Depth information can be determined. For example, the prenoptic camera may include an image sensor overlaid with a transmissive diffraction mask (TDM). Alternatively, or in addition, the prenoptic camera may include an image sensor, including an angle-sensitive pixel and / or a phase-sensitive autofocus pixel (PDAF) and / or a microlens array (MLA). Such a sensor may serve as a depth information source in place of or in addition to the depth sensor 551.

Also, it should be understood that the configuration of the components in FIG. 5B is provided as an example. The visual optics assembly 548 may include components with any suitable configuration, which may be configured to provide the user with a practical maximum field of view for a particular set of components. For example, if the visual optics assembly 548 has one world camera 552, the world cameras may be installed within the central region of the visual optics assembly instead of the sides.

Information from the sensors in the visual optics assembly 548 may be coupled to one or more of the processors in the system. The processor may generate data that can be rendered to make the user perceive the virtual content to interact with objects in the physical world. The rendering may be implemented in any suitable way, including the steps of generating image data that depict both physical and virtual objects. In other embodiments, the physical and virtual content may be portrayed in one scene by modulating the opacity of the display device, where the user sees through the physical world. Opacity may be controlled to create the appearance of the virtual object and block the user from seeing the objects in the physical world that are included by the virtual object. In some embodiments, the image data may be modified to be perceived by the user so that when the virtual content is viewed through the user interface, it interacts realistically with the physical world (eg, the content). Clipping and consideration of occlusion), may include only virtual content.

The location on the visual optics assembly 548, where the content may be displayed to create an impression of the object at a particular location, may depend on the physics of the visual optics assembly. In addition, the posture of the user's head with respect to the physical world and the direction the user's eyes are looking at are within the physical world content, which is displayed at a specific location on the visual optical system assembly where the content will appear. Will affect the location. A sensor as described above is such that the processor receiving the sensor input can calculate where the object should be rendered on the visual optics assembly 548 to create the desired appearance for the user. This information may be collected and / or supplied with information from which this information can be calculated.

Regardless of how the content is presented to the user, the model of the physical world correctly calculates the characteristics of the virtual object that can be influenced by the physical object, including the shape, position, motion, and visibility of the virtual object. Can be used as it can be. In some embodiments, the model may include a reconstruction of the physical world, eg, reconstruction 518.

The model may be created from data collected from sensors on the user's wearable device. However, in some embodiments, the model may be created from data collected by multiple users, which may be aggregated within a computing device remote from all users (and "cloud". It may be "inside").

The model may be created, at least in part, by a world reconstruction system such as, for example, the world reconstruction component 516 of FIG. 3, which is depicted in more detail in FIG. 6A. World reconstruction component 516 may include a perception module 660 that can generate, update, and memorize representations for parts of the physical world. In some embodiments, the perception module 660 may represent a portion of the physical world within the reconstruction range of the sensor as multiple voxels. Each voxel corresponds to a given volume of 3D cube in the physical world, contains surface information, and may indicate whether the surface is within the volume represented by the voxel. A voxel whose corresponding volume has been determined to contain the surface of a physical object, has been determined to be empty, or has not yet been measured using a sensor, and therefore its value is unknown. A value may be assigned to indicate whether or not. Values indicating voxels determined to be empty or unknown need not be explicitly stored, and voxel values are optional, including not storing information about voxels determined to be empty or unknown. It should be understood that in the preferred method of the above, it may be stored in computer memory.

In addition to generating information for sustained world representation, the perception module 660 may identify and output changes in the area around the user of the AR system. Indications of such changes trigger updates to 3D data stored as part of a sustained world, or generate AR content, update AR content, trigger component 604, etc. You may trigger the function of.

In some embodiments, the perception module 660 may identify changes based on a signed distance function (SDF) model. The perception module 660 may be configured to receive sensor data such as, for example, a depth map 660a and a head posture 660b, and then fuse the sensor data to the SDF model 660c. The depth map 660a may provide the SDF information directly, or the image may be processed to arrive at the SDF information. SDF information represents the distance from the sensor used to capture that information. Since those sensors can be part of the wearable unit, the SDF information can represent the physical world from the wearable unit's point of view, and thus from the user's point of view. Head posture 660b may allow SDF information to be associated with voxels within the physical world.

In some embodiments, the perception module 660 may generate, update, and store representations for parts of the physical world within the perceptual range. The perceptual range may be determined, at least in part, based on the reconstruction range of the sensor, which may be determined, at least in part, based on the limits of the observation range of the sensor. As a specific embodiment, an active depth sensor that operates using an active IR pulse operates reliably over a range of distances and can range from a few centimeters or a few tens of centimeters to a few meters. Can be created.

The world reconstruction component 516 may include additional modules that can interact with the perception module 660. In some embodiments, the sustained world module 662 may receive representations for the physical world based on the data obtained by the perception module 660. The Sustained World Module 662 may also contain representations of various formats of the physical world. For example, 3D metadata 662b such as voxels may be stored with the mesh 662c and the plane 662d. In some embodiments, other information such as depth maps may also be stored.

In some embodiments, representations of the physical world, such as those illustrated in FIG. 6A, are about the physical world as compared to sparse maps, such as tracking maps based on feature points, as described above. Can provide relatively dense information.

In some embodiments, the perception module 660 may include a module that produces representations for the physical world in various formats, including, for example, mesh 660d, planes, and semantics 660e. Representations for the physical world may be stored across local and remote storage media. Representations for the physical world may be described, for example, in different coordinate frames, depending on the location of the storage medium. For example, the representation for the physical world stored within the device may be described in a coordinate frame local to the device. Representations for the physical world may have counterparts stored in the cloud. Corresponds in the cloud may be described in coordinate frames shared by all devices in the XR system.

In some embodiments, these modules are used for data within the perceptual range of one or more sensors at the time the representation was generated, as well as data captured at a previous time and information within the Sustained World Module 662. Based on this, the expression may be generated. In some embodiments, these components may act on depth information captured using a depth sensor. However, AR systems may include visual sensors and may generate such representations by analyzing monocular or binocular visual information.

In some embodiments, these modules may act on the realm of the physical world. Those modules may be triggered to update a sub-region of the physical world when the perception module 660 detects a change in the physical world within that sub-region. Such changes may be detected, for example, by detecting new surfaces within the SDF model 660c, or by other criteria such as changes in the values of a sufficient number of voxels representing subregions.

World reconstruction component 516 may include component 664, which may receive representations of the physical world from the perception module 660. Information about the physical world may be pull-delivered by these components, for example, according to usage requirements from the application. In some embodiments, the information may be push-delivered to the components used, such as by indicating changes in pre-identified areas or changes in physical world representation within the perceptual range. Component 664 may include, for example, a game program and other components that perform processing for visual occlusion, physics-based interactions, and environmental inference.

In response to a query from component 664, the perception module 660 may send representations for the physical world in one or more formats. For example, the Perception Module 660 may transmit a surface representation when component 664 indicates that its use is for a visual occlusion or physics-based interaction. Perception module 660 may transmit meshes, planes, and semantics of the physical world when component 664 indicates that its use is for environmental inference.

In some embodiments, the perception module 660 may include a component that provides format information to the component 664. An embodiment of such a component may be the raycasting component 660f. The used component (eg, component 664) may, for example, query information about the physical world from a particular point of view. The raycasting component 660f may be selected from one or more representations of the physical world data in the field of view from that point of view.

As should be understood from the above description, the perception module 660 or another component of the AR system may process the data and create a 3D representation of part of the physical world. The data to be processed, at least in part, is based on the camera cone and / or depth image, thinning out part of the 3D reconstructed volume, extracting and sustaining planar data, maintaining the consistency of the neighborhood system. While capturing, sustaining, and updating 3D reconstructed data in blocks that allow local updates, occlusion data is derived from a combination of one or more depth data sources, generating such scenes with occlusion data. Can be reduced by providing to the application and / or performing multi-step mesh simplification. The reconstruction may contain different levels of sophistication data, including, for example, raw data such as live depth data, fused 3D data such as voxels, and calculated data such as mesh.

In some embodiments, the components of the passable world model may be distributed, some running locally on the XR device and some on the network connected to the server or. Separately, it is executed remotely, such as in the cloud. The processing of information and the distribution of storage between the local XR device and the cloud can affect the functionality and user experience of the XR system. For example, reducing the processing on the local device by allocating the processing to the cloud can enable longer battery life and reduce the heat generated on the local device. However, allocating much more processing to the cloud can create unwanted latency that causes an unacceptable user experience.

FIG. 6B illustrates a distributed component architecture 600 configured for spatial computing, according to some embodiments. The distributed component architecture 600 may include passable world components 602 (eg, PW538 in FIG. 5A), Lumin OS 604, API 606, SDK 608, and application 610. Lumin OS 604 may include a Linux®-based kernel with custom drivers compatible with XR devices. The API 606 may include an application programming interface that gives the XR application (eg, application 610) access to the spatial computing features of the XR device. The SDK 608 may include a software development kit that allows the creation of XR applications.

One or more components within Architecture 600 may create and maintain models of passable worlds. In this embodiment, the sensor data is collected on the local device. The processing of the sensor data may be performed partially locally on the XR device and partially in the cloud. The PW538 may include, at least in part, an environment map created based on data captured by AR devices worn by multiple users. During an AR experience session, individual AR devices (such as the wearable devices described above in connection with FIG. 4) may create a tracking map, which is a type of map.

In some embodiments, the device may include components that build both sparse and dense maps. The tracking map may serve as a sparse map and may include head postures of AR devices scanning the environment as well as information about objects detected in the environment in each head posture. Their head postures may be maintained locally on a device-by-device basis. For example, the head posture on each device can be relative to the initial head posture when the device is turned on for that session. As a result, each tracking map can be local to the device that creates it. The dense map may contain surface information, which may be represented by mesh or depth information. Alternatively, or in addition, the dense map may contain higher levels of information derived from surface or depth information such as the location and / or properties of planes and / or other objects.

The creation of dense maps may be independent of the creation of sparse maps in some embodiments. The creation of dense and sparse maps may be performed, for example, within a separate processing pipeline within the AR system. Separating the processing may allow, for example, the generation or processing of different types of maps to be performed at different rates. Sparse maps may be refreshed, for example, at a faster rate than dense maps. However, in some embodiments, the processing of dense and sparse maps may be relevant even if they are performed within different pipelines. Changes in the physical world exposed within a sparse map can, for example, trigger an update of the dense map, or vice versa. Moreover, maps can be used together, even if they are created independently. For example, a coordinate system derived from a sparse map may be used to define the position and / or orientation of objects in a dense map.

Sparse and / or dense maps may survive for reuse by the same device and / or for sharing with other devices. Such survival may be achieved by storing information in the cloud. The AR device may send the tracking map to the cloud and, for example, merge it with an environment map selected from the surviving maps previously stored in the cloud. In some embodiments, the selected surviving maps may be sent from the cloud to the AR device for merging. In some embodiments, the surviving map may be oriented with respect to one or more persistent coordinate frames. Such maps can serve as a reference map because they can be used by any of multiple devices. In some embodiments, the passable world model may include or be created with one or more canonical maps. Although the device performs some actions based on the coordinate frame local to the device, it also uses the canonical map by determining the conversion between that coordinate frame local to the device and the canonical map. good.

The canonical map may arise as a tracking map (TM) (eg, TM1102 in FIG. 31A), which can be promoted to the canonical map. A canon map uses the information in the canon map to use the information in the canon map within the physical world around the device once the device accessing the canon map determines the transformation between its local coordinate system and the coordinate system of the canon map. It may survive so that it can determine the location of the objects represented in the canonical map of. In some embodiments, the TM may be a head posture sparse map created by an XR device. In some embodiments, the canonical map is such that the XR device sends one or more TMs to the cloud server to merge with the additional TMs captured by the XR device or by another XR device at different times. Then, it may be created.

Criteria maps or other maps may provide information about the part of the physical world represented by the data processed to create the individual maps. FIG. 7 depicts an exemplary tracking map 700 according to some embodiments. The tracking map 700 may provide a plan view 706 of a physical object in the corresponding physical world represented by the point 702. In some embodiments, the map point 702 may represent a feature of a physical object that may contain multiple features. For example, each corner of the table can be a feature represented by a point on the map. Features may be derived from processed images as available using the sensors of the wearable device in the augmented reality system. Features may be derived, for example, by processing the image frame output by the sensor and identifying the features based on a large gradient in the image or other suitable criteria. Further processing may limit the number of features within each frame. For example, the process may select features that are likely to represent persistent objects. One or more heuristics may be applied for this selection.

The tracking map 700 may include data about points 702 collected by the device. Postures may be stored for each image frame with data points contained within the tracking map. The orientation can represent the orientation in which the image frame is captured so that the feature points within each image frame can be spatially correlated. The posture may be determined by positioning information that can be derived from a sensor such as an IMU sensor on the wearable device. Alternatively, or in addition, the orientation may be determined by matching the image frame with another image frame that depicts the overlap of the physical world. The relative attitude between the two frames can be calculated by finding such a positional correlation, which can be accomplished by matching a subset of feature points within the two frames. Relative orientation can be appropriate for tracking maps, as the map can be relative to the device-local coordinate system established based on the device's initial orientation when the tracking map construction begins. ..

Not all feature points and image frames collected by the device can be reserved as part of the tracking map, as much of the information collected using the sensor is likely to be redundant. Rather, only certain frames may be added to the map. The frames may be selected based on one or more criteria such as the degree of overlap with image frames already in the map, the number of new features they contain, or the quality metric for features in the frame. Image frames that are not added to the tracking map may be discarded or used to revise the location of the feature. As a further alternative, all or most of the image frames, represented as a set of features, may be reserved, but a subset of those frames may be designated as keyframes, which may be further processed. Used for.

Keyframes may be processed to produce the keyrig 704. Keyframes may be processed to produce a three-dimensional set of feature points and stored as a key rig 704. Such processing may involve, for example, a step of comparing image frames simultaneously derived from two cameras and stereoscopically determining the 3D position of the feature point. Metadata such as posture may be associated with these keyframes and / or keyrigs.

The environment map may have one of a plurality of formats, for example, depending on the storage location of the environment map, including the local storage device and the remote storage device of the AR device. For example, a map in a remote storage device may have higher resolution than a map in a local storage device on a wearable device if memory is limited. In order to send a higher resolution map from remote storage to local storage, the map is stored in the map with the number of poses per area of the physical world and / or the number of feature points stored per pose. May be downsampled or otherwise converted to a suitable format, such as by reducing. In some embodiments, slices or parts of the high resolution map from the remote storage may be sent to the local storage and the slices or parts are not downsampled.

The environment map database may be updated as new tracking maps are created. To determine which of the potentially very large number of environment maps in the database should be updated, the step to update is one or more environments stored in the database associated with the new tracking map. It may include steps to efficiently select the map. One or more selected environment maps may be ranked by relevance, with one or more of the highest ranked maps being the higher ranked selected environment map and the new tracking map. May be selected to merge and process to create one or more updated environment maps. When a new tracking map represents a part of the physical world where there is no existing environment map to update over it, the tracking map may be stored in the database as a new environment map.

View independent display

Described herein are methods and devices for providing virtual content using an XR system, independent of the location of the eye viewing the virtual content. Traditionally, virtual content is re-rendered in response to any movement of the display system. For example, if a user wearing a display system visually recognizes a virtual representation of a three-dimensional (3D) object on the display and roams around the area where the 3D object appears, the 3D object occupies the real space of the user. , Should be re-rendered per viewpoint so that it has the perception of roaming around the object. However, re-rendering consumes significant computational resources in the system and causes artifacts due to latency.

We can use the head posture (eg, the location and orientation of the user wearing the XR system) to render virtual content independently of eye rotation in the user's head. Recognized and acknowledged its true value. In some embodiments, the dynamic map of the scene is independent of eye rotation in the user's head and / or, for example, sensor deformation caused by heat generated during fast computational intensive movements. Virtual content that interacts with the dynamic map may then be generated based on multiple coordinate frames in real space across one or more sessions so that it can be rendered robustly. In some embodiments, the configuration of the plurality of coordinate frames is such that the first XR device worn by the first user and the second XR device worn by the second user have a common location in the scene. It may be possible to recognize. In some embodiments, the configuration of the plurality of coordinate frames may allow the user wearing the XR device to view the virtual content in the same place in the scene.

In some embodiments, the tracking map may be constructed within a world coordinate frame, which may have a world origin. The world origin may be the first attitude of the XR device when the power is turned on. The world origin may be aligned with gravity so that the developer of the XR application can obtain gravity alignment without extra work. Different tracking maps may be constructed within different world coordinate frames because the tracking maps can be captured by the same XR device and / or different XR devices worn by different users in different sessions. In some embodiments, the session of the XR device may start after the device is powered on and continue until it is powered off. In some embodiments, the XR device may have a head coordinate frame, which may have a head origin. The head origin may be the current posture of the XR device when the image is taken. Differences between the head poses of the world coordinate frame and the head coordinate frame may be used to estimate the tracking route.

In some embodiments, the XR device may have a camera coordinate frame, which may have a camera origin. The camera origin may be the current orientation of one or more sensors in the XR device. The present inventors recognized that the configuration of the camera coordinate frame enables the robust display of virtual content independently of the eye rotation in the user's head, and acknowledged its true value. The configuration also enables a robust display of virtual content, eg, independent of sensor deformation due to heat generated during operation.

In some embodiments, the XR device comprises a head mountable frame that allows the user to adhere to its head and include two waveguides, one in front of each of the user's eyes. It may have a head unit. The waveguide can be transparent so that ambient light from the real-world object can pass through the waveguide and the user can see the real-world object. Each waveguide may allow the light projected from the projector to pass through the individual eye of the user. The projected light can form an image on the retina of the eye. The retina of the eye therefore receives ambient and projected light. At the same time, the user may see a real-world object and one or more virtual objects created by the projected light. In some embodiments, the XR device may have sensors that detect real-world objects around the user. These sensors may be, for example, cameras that capture images that can be processed to identify the location of real-world objects.

In some embodiments, the XR system may assign coordinate frames to the virtual content as opposed to binding the virtual content within the world coordinate frame. Such a configuration allows virtual content to be described regardless of where it was rendered for the user, eg, a persistent coordinate frame (PCF) described in connection with FIGS. 14-20C. ) Etc. can be tied to a more persistent frame position and rendered in a defined location. When the location of the object changes, the XR device may detect the change in the environment map and determine the movement of the head unit worn by the user with respect to the real world object.

FIG. 8 illustrates a user experiencing virtual content as rendered in a physical environment by the XR system 10, according to some embodiments. The XR system may include a first XR device 12.1, a network 18, and a server 20, which are worn by the first user 14.1. User 14.1 exists in a physical environment with real objects in the form of table 16.

In the illustrated embodiment, the first XR device 12.1 includes a head unit 22, a belt pack 24, and a cable connection 26. The first user 14.1 fixes the head unit 22 to the head and the belt pack 24 remote from the head unit 22 on the waist. The cable connection 26 connects the head unit 22 to the belt pack 24. The head unit 22 is used to display a virtual object or a plurality of objects to the first user 14.1 while allowing the first user 14.1 to see a real object such as a table 16. Including technology. The belt pack 24 mainly includes the processing and communication capability of the first XR device 12.1. In some embodiments, the processing and communication capabilities are wholly or partially within the head unit 22 so that the belt pack 24 can be removed or located within another device such as a backpack. It may be resident.

In the illustrated embodiment, the belt pack 24 is connected to the network 18 via a wireless connection. The server 20 is connected to the network 18 and holds data representing local content. The belt pack 24 downloads data representing local content from the server 20 via the network 18. The belt pack 24 provides data to the head unit 22 via the cable connection 26. The head unit 22 may include a display having a light source, such as a laser light source or a light emitting diode (LED) light source, and a waveguide that guides the light.

In some embodiments, the first user 14.1 may mount the head unit 22 on its head and the belt pack 24 on its lumbar region. The belt pack 24 may download image data representing virtual contents from the server 20 via the network 18. The first user 14.1 may see the table 16 through the display of the head unit 22. The projector, which forms part of the head unit 22, may receive image data from the belt pack 24 and generate light based on the image data. Light may travel through one or more of the waveguides that form part of the display of the head unit 22. The light may then exit the waveguide and propagate onto the retina of the eye of the first user 14.1. The projector may generate light in a pattern that is replicated on the retina of the eye of the first user 14.1. The light that hits the retina of the eye of the first user 14.1 is selected so that the first user 14.1 perceives the image at a preselected depth behind the waveguide. It may have a depth. In addition, the eyes of the first user 14.1 are slightly such that the brain of the first user 14.1 perceives a three-dimensional image or multiple images at a selected distance from the head unit 22. You may receive different images. In the illustrated embodiment, the first user 14.1 perceives the virtual content 28 above the table 16. The percentage of the location and distance from the virtual content 28 and the first user 14.1 varies with the data representing the virtual content 28 and used to display the virtual content 28 to the first user 14.1. Determined by the coordinate frame of.

In the illustrated embodiment, the virtual content 28 is invisible from the perspective of the drawing and visible to the first user 14.1 through the use of the first XR device 12.1. The virtual content 28 may initially reside as a data structure in the visual data and an algorithm in the belt pack 24. The data structure can then be exposed as light when the projector of the head unit 22 produces light based on the data structure. Although the virtual content 28 does not exist in the three-dimensional space in front of the first user 14.1, the virtual content 28 is still 3 for the purpose of exemplifying what the wearer of the head unit 22 perceives. Please understand that it is represented in FIG. 1 in dimensional space. Visualization of computer data in three-dimensional space is intended to illustrate how data structures that facilitate rendering perceived by one or more users are interrelated within the data structures within the belt pack 24. Can be used in the description.

FIG. 9 illustrates the components of the first XR device 12.1 according to some embodiments. The first XR device 12.1 includes a head unit 22, for example, a rendering engine 30, various coordinate systems 32, various origin and destination coordinate frames 34, and various origin / destination coordinate frame conversions. It may include various components that form part of the visual data and algorithm, including the vessel 36. The various coordinate systems may be based on the intrinsic properties of the XR device, or may be determined by reference to other information such as persistent attitudes or persistent coordinate systems as described herein. ..

The head unit 22 may include a head mountable frame 40, a display system 42, a real object detection camera 44, a movement tracking camera 46, and an inertial measurement unit 48.

The head mountable frame 40 may have a shape that can be fixed to the head of the first user 14.1 in FIG. The display system 42, the real object detection camera 44, the movement tracking camera 46, and the inertial measurement unit 48 are mounted on the head mountable frame 40 and can therefore move with the head mountable frame 40.

The coordinate system 32 may include a local data system 52, a world frame system 54, a head frame system 56, and a camera frame system 58.

The local data system 52 may include a data channel 62, a local frame determination routine 64, and a local frame storage instruction 66. The data channel 62 may be an internal software routine, a hardware component such as an external cable or radio frequency receiver, or a hybrid component such as an open port. The data channel 62 may be configured to receive image data 68 representing virtual content.

The local frame determination routine 64 may be connected to the data channel 62. The local frame determination routine 64 may be configured to determine the local coordinate frame 70. In some embodiments, the local frame determination routine may determine a local coordinate frame based on a real world object or real world location. In some embodiments, the local coordinate frame is the top edge relative to the bottom edge of the browser window, the character's head or foot, the node on the outer surface of the prism or boundary box that surrounds the virtual content, or the face-to-face direction of the virtual content. It may be based on any other suitable location for installing the coordinate frame, which defines the location where the virtual content should be installed (eg, a node such as an installation node or anchor node).

The local frame storage instruction 66 may be connected to the local frame determination routine 64. One of skill in the art will appreciate that software modules and routines are "connected" to each other through subroutines, calls, etc. The local frame storage command 66 may store the local coordinate frame 70 as the local coordinate frame 72 in the origin and destination coordinate frame 34. In some embodiments, the origin and destination coordinate frame 34 may be one or more coordinate frames that can be manipulated or transformed in order for the virtual content to persist between sessions. In some embodiments, the session may be a time cycle between booting up and shutting down the XR device. The two sessions may be two start-up and shutdown cycles for a single XR device, or may be two start-up and shutdown cycles for two different XR devices.

In some embodiments, the origin and destination coordinate frame 34 is during one or more transformations required for the first user's XR device and the second user's XR device to recognize a common location. It may be a coordinate frame involved. In some embodiments, the destination coordinate frame is even the output of a series of calculations and transformations applied to the target coordinate frame for the first and second users to view the virtual content at the same location. good.

The rendering engine 30 may be connected to the data channel 62. The rendering engine 30 may receive image data 68 from the data channel 62 so that the rendering engine 30 can render virtual content based on the image data 68, at least in part.

The display system 42 may be connected to the rendering engine 30. The display system 42 may include a component that converts the image data 68 into visible light. Visible light may form one or two patterns per eye. Visible light can enter the eye of the first user 14.1 in FIG. 8 and can be detected on the retina of the eye of the first user 14.1.

The real object detection camera 44 may include one or more cameras capable of capturing images from different sides of the head mountable frame 40. The mobile tracking camera 46 may include one or more cameras that capture images on the sides of the head mountable frame 40. One set of one or more cameras may be used in place of the two sets of one or more cameras representing the real object detection camera 44 and the motion tracking camera 46. In some embodiments, cameras 44, 46 may capture images. As described above, these cameras may collect data used to build a tracking map.

The inertial measurement unit 48 may include several devices used to detect the movement of the head unit 22. The inertial measurement unit 48 may include a gravity sensor, one or more accelerometers, and one or more gyroscopes. The sensors of the inertial measurement unit 48, in combination, track the movement of the head unit 22 in at least three orthogonal directions and about at least three orthogonal axes.

In the illustrated embodiment, the world frame system 54 includes a world surface determination routine 78, a world frame determination routine 80, and a world frame storage instruction 82. The world surface determination routine 78 is connected to the real object detection camera 44. The world surface determination routine 78 receives an image and / or a key frame, processes the image, and identifies a surface in the image based on the image captured by the real object detection camera 44. The depth sensor (not shown) may determine the distance to the surface. The surface is therefore represented by data in three dimensions, including its size, shape, and distance from the real object detection camera.

In some embodiments, the world coordinate frame 84 may be based on the origin at the time of initialization of the head posture session. In some embodiments, the world coordinate frame may be located at the location where the device was booted up, or may be a new location if head posture is lost during the boot session. In some embodiments, the world coordinate frame may be the origin at the start of the head posture session.

In the illustrated embodiment, the world frame determination routine 80 is connected to the world surface determination routine 78 and determines the world coordinate frame 84 based on the location of the surface as determined by the world surface determination routine 78. The world frame storage instruction 82 is connected to the world frame determination routine 80, and receives the world coordinate frame 84 from the world frame determination routine 80. The world frame storage command 82 stores the world coordinate frame 84 as the world coordinate frame 86 in the origin and destination coordinate frame 34.

The head frame system 56 may include a head frame determination routine 90 and a head frame storage command 92. The head frame determination routine 90 may be connected to the motion tracking camera 46 and the inertial measurement unit 48. The head frame determination routine 90 may use the data from the motion tracking camera 46 and the inertial measurement unit 48 to calculate the head coordinate frame 94. For example, the inertial measurement unit 48 may have a gravity sensor that determines the direction of gravity with respect to the head unit 22. The motion tracking camera 46 may be used by the head frame determination routine 90 to continuously capture images that refine the head coordinate frame 94. The head unit 22 moves when the first user 14.1 in FIG. 8 moves its head. The motion tracking camera 46 and the inertial measurement unit 48 may continuously provide data to the head frame determination routine 90 so that the head frame determination routine 90 can update the head coordinate frame 94.

The head frame storage command 92 may be connected to the head frame determination routine 90 and may receive the head coordinate frame 94 from the head frame determination routine 90. The head frame storage command 92 may store the head coordinate frame 94 as the head coordinate frame 96 in the origin and destination coordinate frame 34. The head frame storage command 92 may repeatedly store the updated head coordinate frame 94 as the head coordinate frame 96 when the head frame determination routine 90 recalculates the head coordinate frame 94. In some embodiments, the head coordinate frame may be the location of the wearable XR device 12.1 relative to the local coordinate frame 72.

The camera frame system 58 may include a camera-specific property 98. The camera specificity 98 may include the dimensions of the head unit 22, which is a feature of its design and manufacture. The camera unique property 98 may be used to calculate the camera coordinate frame 100 stored in the origin and destination coordinate frame 34.

In some embodiments, the camera coordinate frame 100 may include all pupil positions of the left eye of the first user 14.1 in FIG. When the left eye moves from left to right or up and down, the pupil position of the left eye is located within the camera coordinate frame 100. In addition, the pupil position of the right eye is located within the camera coordinate frame 100 for the right eye. In some embodiments, the camera coordinate frame 100 may include the location of the camera relative to the local coordinate frame when the image is taken.

The origin / destination coordinate frame converter 36 may include a local / world coordinate converter 104, a world / head coordinate converter 106, and a head / camera coordinate converter 108. The local / world coordinate converter 104 may receive the local coordinate frame 72 and convert the local coordinate frame 72 into the world coordinate frame 86. The conversion of the local coordinate frame 72 to the world coordinate frame 86 can be represented as a local coordinate frame converted to the world coordinate frame 110 within the world coordinate frame 86.

The world / head coordinate converter 106 may convert from the world coordinate frame 86 to the head coordinate frame 96. The world / head coordinate converter 106 may convert the local coordinate frame converted to the world coordinate frame 110 into the head coordinate frame 96. The transformation can be represented as a local coordinate frame that is transformed into the head coordinate frame 112 within the head coordinate frame 96.

The head / camera coordinate converter 108 may convert from the head coordinate frame 96 to the camera coordinate frame 100. The head / camera coordinate converter 108 may convert the local coordinate frame converted into the head coordinate frame 112 into the local coordinate frame converted into the camera coordinate frame 114 in the camera coordinate frame 100. The local coordinate frame converted to the camera coordinate frame 114 may be incorporated into the rendering engine 30. The rendering engine 30 may render the image data 68 representing the local content 28 based on the local coordinate frame converted into the camera coordinate frame 114.

FIG. 10 is a spatial representation of various origin and destination coordinate frames 34. The local coordinate frame 72, the world coordinate frame 86, the head coordinate frame 96, and the camera coordinate frame 100 are represented in the figure. In some embodiments, the local coordinate frame associated with the XR content 28 is a local and / or world coordinate frame and / or when the virtual content is placed in the real world and thus the virtual content can be viewed by the user. It may have a position and rotation with respect to the PCF (eg, it may provide a node and face-to-face orientation). Each camera may have its own camera coordinate frame 100 that covers all pupil positions in one eye. Reference numbers 104A and 106A represent the transformations performed by the local / world coordinate converter 104, the world / head coordinate converter 106, and the head / camera coordinate converter 108, respectively, in FIG.

FIG. 11 illustrates a camera rendering protocol for converting head coordinate frames to camera coordinate frames, according to some embodiments. In the illustrated embodiment, the pupil for one eye moves from position A to position B. A virtual object intended to appear stationary will be projected onto a depth plane at one of two positions A or B, depending on the position of the pupil (as the camera is a pupil-based coordinate frame). Assuming it is configured to be used). As a result, using a pupil coordinate frame that is converted to a head coordinate frame will cause jitter in the stationary virtual object as the eye moves from position A to position B. This situation is referred to as a view-dependent display or projection.

As depicted in FIG. 12, the camera coordinate frame (eg, CR) is positioned and covers all pupil positions, but the object projection is here consistent regardless of pupil positions A and B. There will be. The head coordinate frame is converted to a CR frame, which is referred to as a view-independent display or projection. Image reprojection may be applied to the virtual content to take into account changes in eye position, however, because the rendering is still in the same position, jitter is minimized.

FIG. 13 illustrates the display system 42 in more detail. The display system 42 includes a stereoscopic analyzer 144 that is connected to the rendering engine 30 and forms part of the visual data and algorithms.

The display system 42 further includes left and right projectors 166A and 166B, and left and right waveguides 170A and 170B. The left and right projectors 166A and 166B are connected to a power source. Each projector 166A and 166B has a separate input for image data to be provided to a separate projector 166A or 166B. The individual projectors 166A or 166B, when fed, generate and emit light in a two-dimensional pattern. The left and right waveguides 170A and 170B are positioned to receive light from the left and right projectors 166A and 166B, respectively. The left and right waveguides 170A and 170B are transparent waveguides.

At the time of use, the user mounts the head mountable frame 40 on the head. The components of the head mountable frame 40 may include, for example, a strap (not shown) that wraps around the back of the user's head. The left and right waveguides 170A and 170B are then located in front of the user's left and right eyes 220A and 220B.

The rendering engine 30 captures the image data it receives into the stereoscopic analyzer 144. The image data is the three-dimensional image data of the local content 28 in FIG. The image data is projected on a plurality of virtual planes. The stereoscopic analyzer 144 analyzes the image data and determines the left and right image data sets based on the image data for projection onto each depth plane. Left and right image datasets are datasets that represent two-dimensional images that are projected in three dimensions and give the user a sense of depth.

The stereoscopic analyzer 144 captures the left and right image data sets into the left and right projectors 166A and 166B. The left and right projectors 166A and 166B then create left and right light patterns. It should be understood that the components of the display system 42 are shown in plan view, but the left and right patterns are two-dimensional patterns when shown in front elevation. Each light pattern contains multiple pixels. For illustration purposes, rays 224A and 226A from two of the pixels are shown to exit the left projector 166A and enter the left waveguide 170A. The rays 224A and 226A are reflected from the side of the left waveguide 170A. Rays 224A and 226A are shown to propagate through internal reflections from left to right within the left waveguide 170A, while rays 224A and 226A are also directed towards the paper using a refractive and reflective system. Please understand that it also propagates.

The rays 224A and 226A exit from the left optical waveguide 170A through the pupil 228A and then enter the left eye 220A through the pupil 230A of the left eye 220A. The rays 224A and 226A then hit the retina 232A of the left eye 220A. Thus, the left light pattern hits the retina 232A of the left eye 220A. The user is given the perception that the pixels formed on the retina 232A are the pixels 234A and 236A, which the user perceives as being at a distance on the side of the left waveguide 170A facing the left eye 220A. .. Depth perception is created by manipulating the focal length of light.

Similarly, the stereoscopic analyzer 144 captures the right image data set into the right projector 166B. The right projector 166B transmits a right light pattern, which is represented by pixels in the form of rays 224B and 226B. The rays 224B and 226B are reflected in the right waveguide 170B and exit through the pupil 228B. The rays 224B and 226B then enter through the pupil 230B of the right eye 220B and hit the retina 232B of the right eye 220B. The pixels of the rays 224B and 226B are perceived as pixels 134B and 236B behind the right waveguide 170B.

The patterns created on the retinas 232A and 232B are individually perceived as left and right images. The left and right images are slightly different from each other due to the function of the stereoscopic analyzer 144. The left and right images are perceived as 3D rendering in the user's brain.

As mentioned, the left and right waveguides 170A and 170B are transparent. Light from real objects such as the table 16 on the left and right waveguides 170A and 170B side facing the eyes 220A and 220B is projected through the left and right waveguides 170A and 170B and hits the retinas 232A and 232B. be able to.

Persistent Coordinate Frame (PCF)

Described herein are methods and devices for providing spatial persistence across user instances within a shared space. Without spatial persistence, virtual content placed in the physical world by the user in a session cannot exist or can be misplaced in the user's view in a different session. Without spatial persistence, virtual content placed within the physical world by one user is the first, even if the second user intends to share the same physical space experience with the first user. It may not exist or may be out of place within the view of two users.

We recognize and understand that spatial persistence can be provided through persistent coordinate frames (PCFs). The PCF may be defined on the basis of one or more points representing features recognized within the physical world (eg, corners, edges). Features may be selected so that they are likely to be identical from one user instance to another user instance in the XR system.

In addition, the drift between tracks, which can deviate the calculated tracking path (eg, camera trajectory) from the actual tracking path, renders the location of the virtual content against a local map based solely on the tracking map. Sometimes it can appear out of place. The tracking map for space may be refined and drift corrected as the XR device collects more information about the scene over time. However, if the virtual content is placed on a real object and stored against the world coordinate frame of the device derived from the tracking map prior to map refinement, then the virtual content is that the real object is map refined. It can appear displaced as if it had moved in between. The PCF may be updated according to the map refinement as the PCF is defined based on the feature and is updated as the feature moves during the map refinement.

The PCF may have 6 degrees of freedom with translation and rotation with respect to the map coordinate system. The PCF may be stored locally and / or in remote storage media. The translation and rotation of the PCF may be calculated with respect to the map coordinate system, for example, depending on the storage location. For example, the PCF used locally by the device may have translation and rotation with respect to the device's world coordinate frame. The PCF in the cloud may have translations and rotations with respect to the canonical coordinate frame of the canonical map.

The PCF may provide a sparse representation of the physical world and provide less than all of the available information about the physical world so that they can be processed and transferred efficiently. Techniques for processing persistent spatial information create dynamic maps based on one or more coordinate systems in real space across one or more sessions, such as application programming interfaces (APIs). ) May include the step of generating a persistent coordinate frame (PCF) over a sparse map that can be exposed to the XR application.

FIG. 14 is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and the binding of XR content to a PCF according to some embodiments. Each block may represent digital information stored in computer memory. For application 1180, the data may represent computer executable instructions. For virtual content 1170, the digital information may define, for example, a virtual object as defined by application 1180. For other boxes, digital information can characterize some aspects of the physical world.

In the illustrated embodiment, one or more PCFs are created from images captured using sensors on a wearable device. In the embodiment of FIG. 14, the sensor is a visual image camera. These cameras may be the same camera used to form a tracking map. Therefore, some of the processes proposed by FIG. 14 may be performed as part of the step of updating the tracking map. However, FIG. 14 illustrates that information that provides survival is generated in addition to the tracking map.

Two images 1110 from two cameras mounted on a wearable device in a configuration that enables stereoscopic image analysis to derive a 3D PCF are processed together. FIG. 14 illustrates image 1 and image 2, each derived from one of the cameras. A single image from each camera is illustrated for convenience. However, each camera may output a stream of image frames, and the process illustrated in FIG. 14 may be performed for a plurality of image frames in the stream.

Therefore, image 1 and image 2 may each be one frame in a sequence of image frames. The process as depicted in FIG. 14 is repeated on the continuous image frame in the sequence until an image frame containing feature points that form a suitable image from which persistent spatial information is formed is processed. May be good. As an alternative, or in addition, the process of FIG. 14 ensures that the user is no longer sufficiently close to the previously identified PCF and uses that PCF to determine its position with respect to the physical world. It can be repeated as the user moves so much that it cannot. For example, the XR system may maintain the current PCF for the user. When that distance exceeds a threshold, the system may switch to a new current PCF closer to the user, which can be generated according to the process of FIG. 14 using image frames obtained at the user's current location.

Content within the physical world where the stream of image frames is likely to be stable and can be easily identified by devices in the vicinity of the area of the physical world depicted in the image frame, even when generating a single PCF. It may be processed to identify the image frame that depicts the image. In the embodiment of FIG. 14, the process begins with the identification of feature 1120 in the image. Features may be identified, for example, by finding locations or other properties of gradients in the image that may correspond to the corners of the object, eg, above a threshold. In the illustrated embodiment, the feature is a point, but other recognizable features such as edges may also be used as an alternative or in addition.

In the illustrated embodiment, feature 1120 with a fixed number N is selected for further processing. The feature points may be selected based on one or more criteria such as the magnitude of the gradient or the proximity to other feature points. Alternatively, or in addition, the feature points may be heuristically selected, such as based on the characteristics, suggesting that the feature points are persistent. For example, heuristics may be defined based on the characteristics of feature points that are likely to correspond to the corners of windows or doors or large furniture. Such heuristics may consider the feature points themselves and those surrounding them. As a specific example, the number of feature points per image may be 100 to 500 such as 200 or 150 to 250.

Descriptor 1130 may be calculated for feature points regardless of the number of feature points selected. In this embodiment, the descriptor is calculated for each selected feature point, but the descriptor may be calculated for a group of feature points, for a subset of feature points, or for all features in the image. good. Descriptors characterize feature points so that feature points that represent the same object in the physical world can be assigned similar descriptors. Descriptors can facilitate the alignment of two frames that can occur when one map is located relative to another. Rather than searching for the relative orientation of the frames, which minimizes the distance between the feature points of the two images, the initial alignment of the two frames is done by identifying the feature points with similar descriptors. May be good. Image frame matching may be based on a step of matching points with similar descriptors, which may involve less processing to calculate the matching of all feature points in the image.

The descriptor may be calculated as a mapping of feature points to the descriptor, or, in some embodiments, a patch of images around the feature points. The descriptor may be a numerical quantity. U.S. Patent Application No. 16 / 190,948 describes the steps to calculate a descriptor for a feature point, which is incorporated herein by reference in its entirety.

In the embodiment of FIG. 14, the descriptor 1130 is calculated for each feature point in each image frame. Based on the descriptor and / or feature points and / or the image itself, the image frame may be identified as key frame 1140. In the illustrated embodiment, the keyframe is an image frame that meets certain criteria, which is then selected for further processing. When creating a tracking map, image frames may be selected as keyframes, for example to add meaningful information to the map, to be integrated into the map. On the other hand, image frames that are substantially overlapped in the area, over which the image frames are already integrated in the map, may be discarded so that they are not keyframes. Alternatively, or in addition, keyframes may be selected based on the number and / or type of feature points in the image frame. In the embodiment of FIG. 14, the keyframe 1150 selected for inclusion in the tracking map may also be treated as a keyframe for determining the PCF, but for the generation of the PCF. Different or additional criteria for selection may be used.

FIG. 14 shows that keyframes are used for further processing, but the information obtained from the images may be processed in other forms. For example, feature points, such as in a key rig, may be processed as an alternative or in addition. Further, although keyframes are described as being derived from a single image frame, it is not necessary that there be a one-to-one relationship between the keyframe and the obtained image frame. The key frame may be obtained from the plurality of image frames, for example, by stitching or aggregating the image frames together so that only the features appearing in the plurality of images are retained in the key frame.

Keyframes may contain image information and / or metadata associated with the image information. In some embodiments, the image captured by the cameras 44, 46 (FIG. 9) may be calculated within one or more keyframes (eg, keyframes 1, 2). In some embodiments, the keyframe may include a camera orientation. In some embodiments, the keyframe may include one or more camera images captured in the camera orientation. In some embodiments, the XR system determines that a portion of the camera image captured in the camera orientation is not useful and therefore may not include that portion within the key frame. Therefore, using keyframes to align new images with earlier knowledge of the scene reduces the use of computational resources in the XR system. In some embodiments, the keyframe may include an image and / or image data at a location with a certain direction / angle. In some embodiments, the keyframe may include a location and direction from which one or more map points can be observed. In some embodiments, the keyframe may include a coordinate frame with an ID. U.S. Patent Application No. 15 / 877,359 describes key frames and is incorporated herein by reference in their entirety.

Some or all of the keyframes 1140 may be selected for further processing, such as the generation of a sustained pose 1150 for the keyframes. The selection may be based on the characteristics of all or a subset of feature points within the image frame. Those properties may be determined by processing the descriptors, features, and / or the image frame itself. As a specific embodiment, the selection may be based on a cluster of feature points identified as likely to be associated with persistent objects.

Each keyframe is associated with the camera orientation from which the keyframe was obtained. For keyframes selected for persistent attitude processing, that attitude information is stored with other metadata about keyframes such as WiFi finger prints and / or GPS coordinates at the time of acquisition and / or at the location of acquisition. You may.

Persistent posture is a source of information that the device can use to orient itself to previously obtained information about the physical world. For example, if a keyframe from which a persistent pose is created is embedded in a map of the physical world, the device will use a sufficient number of feature points in the keyframe associated with the persistent pose. It can then orient itself to its sustained posture. The device may match the sustained posture with the current image obtained around it. The matching may be based on matching the current image with the image 1110, feature 1120, and / or descriptor 1130 that gave rise to a persistent posture, or any subset of the image or its features or descriptors. .. In some embodiments, the current image frame matched to the persistent pose may be another key frame embedded within the tracking map of the device.

Information about persistent posture may be stored in a format that facilitates sharing between multiple applications that can be performed on the same or different devices. In the embodiment of FIG. 14, some or all of the persistent postures may be reflected as a persistent coordinate frame (PCF) 1160. Like a sustained posture, the PCF may also be associated with a map and may include a set of features or other information that the device can use to determine its orientation with respect to the PCF. The PCF defines its transformation to the origin of the map so that the device can determine its position with respect to any object in the physical world reflected in the map by correlating its position to the PCF. It may include a transformation.

Because the PCF provides a mechanism for determining the location for a physical object, an application such as application 1180 serves as an anchor for virtual content 1170, and the location of the virtual object with respect to one or more PCFs. Can be defined. FIG. 14 illustrates, for example, that the application 1 associates its virtual content 2 with PCFs 1 and 2. Similarly, the application 2 associates the virtual content 3 with the PCFs 1 and 2. App 1 is also shown to associate its virtual content 1 with PCF4 and 5, and app 2 is shown to associate its virtual content 4 with PCF3. In some embodiments, PCF3 may be based on image 3 (not shown), as PCF1 and 2 may be based on image 1 and image 2, and PCF4 and 5 may be image 4 and image 5 (not shown). It may be based on (not shown). When rendering the Virtual Content, the device applies one or more transformations to calculate information such as the location of the virtual content with respect to the device's display and / or the location of the physical object with respect to the desired location of the virtual content. May be good. Using the PCF as a reference can simplify such calculations.

In some embodiments, the persistent posture may be a coordinate location and / or orientation with one or more associated keyframes. In some embodiments, the sustained posture may be created automatically after the user has traveled a distance, eg, 3 meters. In some embodiments, the sustained posture may act as a reference point during positioning. In some embodiments, the sustained posture may be stored within a passable world (eg, passable world module 538).

In some embodiments, the new PCF may be determined based on a predefined distance allowed between adjacent PCFs. In some embodiments, one or more sustained postures may be calculated within the PCF as the user travels a predetermined distance, eg, 5 meters. In some embodiments, the PCF may be associated with one or more world coordinate frames and / or reference coordinate frames, for example, within a passable world. In some embodiments, the PCF may be stored in a local and / or remote database, for example, depending on security settings.

FIG. 15 illustrates a method 4700 for establishing and using sustained coordinate frames, according to some embodiments. Method 4700 may begin with the step (act 4702) of capturing an image of the scene (eg, image 1 and image 2 in FIG. 14) using one or more sensors in the XR device. Multiple cameras may be used and one camera may produce multiple images, eg, in a stream.

Method 4700 includes a step (4704) of extracting points of interest (eg, map points 702 in FIG. 7 and features 1120 in FIG. 14) from captured images and a descriptor for the extracted points of interest (eg, in FIG. 14). It may include a step of generating the descriptor 1130) (act 4706) and a step of generating a keyframe (eg, keyframe 1140) based on the descriptor (act 4708). In some embodiments, the method may form a pair of keyframes that compare points of interest within a keyframe and share a predetermined amount of points of interest. The method may use individual pairs of keyframes to reconstruct parts of the physical world. The mapped portion of the physical world may be stored as a 3D feature (eg, key rig 704 in FIG. 7). In some embodiments, selected parts of a pair of keyframes may be used to build 3D features. In some embodiments, the mapping results may be selectively stored. Keyframes that are not used to build 3D features may be associated with 3D features through poses, such as representing the distance between keyframes, with a covariance matrix between the poses of the keyframes. In some embodiments, the pair of keyframes may be determined so that the distance between each of the 3D features constructed balances the amount of calculation required and the level of accuracy of the resulting model. , May be selected to build 3D features so that they are within a predetermined distance. Such an approach makes it possible to provide a model of the physical world with a suitable amount of data for efficient and accurate calculations using the XR system. In some embodiments, the covariance matrix of the two images may include the covariance between the poses of the two images (eg, 6 degrees of freedom).

Method 4700 may include a step (act 4710) of generating a sustained posture based on a key frame. In some embodiments, the method may include the step of generating a sustained posture based on 3D features reconstructed from a pair of keyframes. In some embodiments, sustained posture may be associated with 3D features. In some embodiments, the sustained posture may include a keyframe posture used to build a 3D feature. In some embodiments, the sustained posture may include the average posture of the keyframes used to build the 3D feature. In some embodiments, the sustained posture is such that the distance between nearby persistent postures is within a predetermined value, eg, one meter to 5 meters, any value in between, or any other. It may be generated to be within a suitable value. In some embodiments, the distance between neighboring persistent poses may be represented by a covariance matrix of neighboring persistent poses.

Method 4700 may include the step of generating a PCF based on a sustained posture (act 4712). In some embodiments, the PCF may be associated with 3D features. In some embodiments, the PCF may be associated with one or more sustained postures. In some embodiments, the PCF may include one of the associated persistent postures. In some embodiments, the PCF may include the average posture of the associated persistent posture. In some embodiments, the PCFs such that the distance between neighboring PCFs is within a predetermined value, eg, a range of 3 to 10 meters, any value in between, or any other suitable value. May be generated in. In some embodiments, the distance between neighboring PCFs may be represented by a covariance matrix of neighboring PCFs. In some embodiments, the PCF allows the XR application to access the model of the physical world through the PCF without accessing the model itself, eg, via an application programming interface (API). It may be exposed to the application.

Method 4700 may include associating at least one of the PCF with the image data of the virtual object to be displayed by the XR device (act 4714). In some embodiments, the method may include calculating the translation and orientation of the virtual object with respect to the associated PCF. It should be understood that associating a virtual object with a PCF generated by the device on which the virtual object is installed is not required. For example, the device may read the stored PCF in the canonical map in the cloud and associate the virtual object with the read PCF. It should be understood that virtual objects can move with the associated PCF as the PCF is adjusted over time.

FIG. 16 illustrates the visual data and algorithms of the first XR device 12.1 and the second XR device 12.2, and the server 20 according to some embodiments. The components illustrated in FIG. 16 are associated with steps that generate, update, and / or use spatial information such as persistent attitudes, persistent coordinate frames, tracking maps, or reference maps as described herein. Can act to perform some or all of the actions. Although not shown, the first XR device 12.1 may be configured identically to the second XR device 12.2. The server 20 may have a map storage routine 118, a reference map 120, a map transmitter 122, and a map merge algorithm 124.

The second XR device 12.2, which may be in the same scene as the first XR device 12.1, has a persistent coordinate frame (PCF) integration unit 1300 and image data 68 that can be used to render virtual objects. The application 1302 and the frame embedding generator 308 (see FIG. 21) may be included. In some embodiments, the map download system 126, the PCF identification system 128, the map 2, the location module 130, the reference map incorporater 132, the reference map 133, and the map issuer 136 are in passable world units 1304. May be grouped into. The PCF integration unit 1300 may be connected to the passable worlds unit 1304 and other components of the second XR device 12.2 to allow the PCF to be read, generated, used, uploaded and downloaded.

Maps with PCFs can enable more survival in a changing world. In some embodiments, the step of locating a tracking map, including, for example, matching features for an image, may include selecting features that represent persistent content from a map composed of PCFs. This enables fast matching and / or positioning. For example, people move in and out of a scene, objects such as doors move relative to the scene, the world demands less storage space and transmission rate, and its to individual PCFs and each other to map the scene. Enables the use of relationships (eg, integrated constellations of PCF).

In some embodiments, the PCF integrated unit 1300 is a PCF 1306 previously stored in data storage on a storage unit of a second XR device 12.2, a PCF tracker 1308, and a continuous posture acquirer 1310. , PCF confirmer 1312, PCF generation system 1314, coordinate frame calculator 1316, continuous posture calculator 1318, tracking map and continuous posture converter 1320, continuous posture and PCF converter 1322, PCF and It may include three converters, including an image data converter 1324.

In some embodiments, the PCF tracker 1308 may have on-prompt and off-prompt selectable by application 1302. Application 1302 may be run by the processor of the second XR device 12.2 and may display virtual content, for example. Application 1302 may have a call to switch on the PCF tracker 1308 via an on-prompt. The PCF tracker 1308 may generate a PCF when the PCF tracker 1308 is switched on. Application 1302 may have subsequent calls that may switch the PCF tracker 1308 off via an off-prompt. The PCF tracker 1308 ends PCF generation when the PCF tracker 1308 is switched off.

In some embodiments, the server 20 may include a plurality of persistent postures 1332 and a plurality of PCF 1330s previously stored associated with the criterion map 120. The map transmitter 122 may transmit the reference map 120 to the second XR device 12.2 together with the continuous attitude 1332 and / or the PCF 1330. Persistent posture 1332 and PCF 1330 may be stored on a second XR device 12.2 in association with the reference map 133. When the map 2 locates with respect to the reference map 133, the continuous posture 1332 and the PCF 1330 may be stored in association with the map 2.

In some embodiments, the continuous posture obtainr 1310 may obtain a continuous posture for map 2. The PCF confirmer 1312 may be connected to the continuous posture obtainr 1310. The PCF confirmer 1312 may read the PCF from the PCF 1306 based on the continuous posture read by the continuous posture acquirer 1310. The PCF read by the PCF confirmer 1312 may form an initial group of PCFs used for an image display based on the PCF.

In some embodiments, application 1302 may require that an additional PCF be generated. For example, if the user navigates to an area that was not previously mapped, application 1302 may switch the PCF tracker 1308 on. The PCF generation system 1314 may be connected to the PCF tracker 1308 and start PCF generation based on the map 2 as the map 2 begins to expand. The PCFs generated by the PCF generation system 1314 may form a second group of PCFs that can be used for PCF-based image display.

The coordinate frame calculator 1316 may be connected to the PCF confirmer 1312. After the PCF confirmer 1312 reads the PCF, the coordinate frame computer 1316 may call the head coordinate frame 96 to determine the head posture of the second XR device 12.2. The coordinate frame calculator 1316 may also call the sustained attitude calculator 1318. The continuous attitude calculator 1318 may be directly or indirectly connected to the frame embedding generator 308. In some embodiments, the image / frame may be designated as a keyframe after a threshold distance from the previous keyframe, eg, 3 meters, has been advanced. The continuous posture calculator 1318 may generate a continuous posture based on a plurality of key frames, for example, three key frames. In some embodiments, the persistent posture may be essentially the average of the coordinate frames of multiple keyframes.

The tracking map and the continuous attitude converter 1320 may be connected to the map 2 and the continuous attitude calculator 1318. The tracking map and the continuous attitude converter 1320 may convert the map 2 into a continuous attitude and determine the persistent attitude at the origin with respect to the map 2.

The persistent attitude and PCF converter 1322 may be connected to the tracking map and continuous attitude converter 1320 and further to the PCF confirmer 1312 and the PCF generation system 1314. Persistent Posture and PCF Converter 1322 converts the Sustained Posture (with which the tracking map converted) to PCF from the PCF confirmer 1312 and the PCF generation system 1314 to determine the PCF for the Persistent Posture. May be good.

The PCF and image data converter 1324 may be connected to a continuous posture and PCF converter 1322 and a data channel 62. The PCF and the image data converter 1324 convert the PCF into image data 68. The rendering engine 30 may be connected to the PCF and the image data converter 1324 to display the image data 68 for the PCF to the user.

The PCF integration unit 1300 may store additional PCFs generated within the PCF 1306 using the PCF generation system 1314. PCF1306 may be stored for sustained posture. The map issuer 136 and the PCF1306 and PCF1306 when the map issuer 136 transmits the map 2 to the server 20 and the map issuer 136 also transmits the PCF and persistent attitude associated with the map 2 to the server 20. The associated persistent posture can be read. When the map storage routine 118 of the server 20 stores the map 2, the map storage routine 118 may also store the persistent posture and PCF generated by the second visual device 12.2. The map merge algorithm 124 may create the canonical map 120, along with the persistent pose of map 2 and the PCF, which are associated with the canonical map 120, respectively, and are stored in the persistent posture 1332 and the PCF 1330.

The first XR device 12.1 may include a PCF integration unit similar to the PCF integration unit 1300 of the second XR device 12.2. When the map transmitter 122 transmits the reference map 120 to the first XR device 12.1, the map transmitter 122 is associated with the reference map 120 and the sustained attitude resulting from the second XR device 12.2. 1332 and PCF 1330 may be transmitted. The first XR device 12.1 may store the PCF and sustained posture in a data storage device on the storage device of the first XR device 12.1. The first XR device 12.1 may then utilize the sustained posture and PCF resulting from the second XR device 12.2 for displaying images to the PCF. In addition, or as an alternative, the first XR device 12.1 reads and generates the PCF and sustained posture in a manner similar to the second XR device 12.2, as described above. You may use it, upload it, and download it.

In the illustrated embodiment, the first XR device 12.1 will generate a local tracking map (hereinafter referred to as "map 1"), and the map storage routine 118 will map map 1 to the first XR device. Receive from 12.1. The map storage routine 118 then stores the map 1 on the storage device of the server 20 as a reference map 120.

The second XR device 12.2 includes a map download system 126, an anchor identification system 128, a location identification module 130, a reference map embedding device 132, a local content positioning system 134, and a map issuer 136. ..

At the time of use, the map transmitter 122 transmits the reference map 120 to the second XR device 12.2, and the map download system 126 downloads and stores the reference map 120 as the reference map 133 from the server 20.

The anchor identification system 128 is connected to the world surface determination routine 78. The anchor identification system 128 identifies anchors based on the objects detected by the world surface determination routine 78. The anchor identification system 128 uses the anchor to generate a second map (map 2). As indicated by cycle 138, the anchor identification system 128 continues to identify anchors and keeps updating map 2. The location of the anchor is recorded as 3D data based on the data provided by the world surface determination routine 78. The world surface determination routine 78 receives an image from the real object detection camera 44 and depth data from the depth sensor 135 to determine the location of the surface and its relative distance from the depth sensor 135.

The positioning module 130 is connected to the reference map 133 and the map 2. The positioning module 130 repeatedly attempts to locate the map 2 with respect to the reference map 133. The reference map embedding device 132 is connected to the reference map 133 and the map 2. When the position identification module 130 locates the map 2 with respect to the reference map 133, the reference map incorporater 132 incorporates the reference map 133 into the anchor of the map 2. Map 2 is then updated with missing data contained within the reference map.

The local content positioning system 134 is connected to the map 2. The local content positioning system 134 may be, for example, a system in which the user can locate the local content at a specific location within the world coordinate frame. The local content itself is then tied to one anchor in Map 2. The local / world coordinate converter 104 converts the local coordinate frame into a world coordinate frame based on the settings of the local content positioning system 134. The functions of the rendering engine 30, the display system 42, and the data channel 62 are described with reference to FIG.

The map issuer 136 uploads the map 2 to the server 20. The map storage routine 118 of the server 20 then stores the map 2 in the storage medium of the server 20.

The map merge algorithm 124 merges the map 2 and the reference map 120. When more than two maps, eg, three or four maps, related to the same or adjacent areas of the physical world are stored, the map merge algorithm 124 merges all the maps into the reference map 120. And render the new reference map 120. The map transmitter 122 then transmits the new reference map 120 to all devices 12.1 and 12.2 in the area represented by the new reference map 120. When devices 12.1 and 12.2 locate their individual maps relative to the canonical map 120, the canonical map 120 becomes a promoted map.

FIG. 17 illustrates an example of generating keyframes for a map of a scene, according to some embodiments. In the illustrated embodiment, the first keyframe KF1 is generated for the door on the left wall of the room. The second keyframe KF2 is generated for the area within the corner where the floor, left wall, and right wall of the room meet. A third keyframe, KF3, is generated for the area of the window on the right wall of the room. The fourth keyframe KF4 is generated for the area at the edge of the rug on the floor of the wall. The fifth keyframe KF5 is generated for the area of lag closest to the user.

FIG. 18 illustrates an example of generating a sustained posture for the map of FIG. 17 with some embodiments. In some embodiments, a new sustained posture is created when the device measures the advanced threshold distance and / or when the application requests a new sustained posture (PP). In some embodiments, the threshold distance may be 3 meters, 5 meters, 20 meters, or any other suitable distance. Choosing a smaller threshold distance (eg, 1 m) can result in an increase in computational load as more PPs can be created and managed compared to a larger threshold distance. Choosing a larger threshold distance (eg, 40 m) results in the creation of fewer PPs and less PCF, and the virtual content associated with the PCF is relatively large distance from the PCF. This can result in an increase in virtual content installation error, as it means that the error can increase with increasing distance (eg, 30 m) from the PCF to the virtual content.

In some embodiments, the PP may be created at the start of a new session. This initial PP can be considered zero and can be visualized as the center of a circle with a radius equal to the threshold distance. When the device reaches a circular circumference and, in some embodiments, the application requests a new PP, the new PP may be installed at the device's current location (threshold distance). In some embodiments, the new PP will not be created at the threshold distance if the device is able to find the existing PP within the threshold distance from the new position of the device. In some embodiments, when a new PP (PP1150 in FIG. 14) is created, the device binds one or more of the closest keyframes to the PP. In some embodiments, the location of the PP relative to the keyframe may be based on the location of the device at the time the PP was created. In some embodiments, the PP will not be created as the device travels the threshold distance unless the application requires the PP.

In some embodiments, the application may request a PCF from the device when the application has virtual content to display to the user. The PCF request from the application may trigger a PP request and a new PP will be created after the device has traveled the threshold distance. FIG. 18 allows the closest keyframes (eg, KF1, KF2, and KF3) to be linked by, for example, calculating the relative attitude between the keyframe and the sustained attitude. The continuous posture PP1 is illustrated. FIG. 18 also illustrates a second sustained posture PP2 that can bind the closest keyframes (eg, KF4 and KF5).

FIG. 19 illustrates an example of generating a PCF for the map of FIG. 17 according to some embodiments. In the illustrated embodiment, PCF1 may include PP1 and PP2. As described above, the PCF may be used to display image data for the PCF. In some embodiments, each PCF may have coordinates within another coordinate frame (eg, a world coordinate frame) and, for example, a PCF descriptor that uniquely identifies the PCF. In some embodiments, the PCF descriptor may be calculated based on the feature descriptor of the features in the frame associated with the PCF. In some embodiments, the various constellations of the PCF may be combined and represent the real world in a sustainable manner that requires the transmission of less data and less data.

20A-20C are schematics illustrating examples of establishing and using persistent coordinate frames. FIG. 20A shows two users 4802A, 4802B with separate local tracking maps 4804A, 4804B that are not located with respect to the canonical map. The origins 4806A and 4806B for an individual user are depicted by a coordinate system within that individual area (eg, a world coordinate system). These origins of each tracking map can be local to each user because the origin depends on the orientation of that individual device when the tracking is started.

As the sensor of the user device scans the environment, the device can be classified as a key frame from which those images can be created, as described above in connection with FIG. As such, images may be captured that may contain features that represent persistent objects. In this embodiment, the tracking map 4802A comprises a sustained posture (PP) 4808A and the tracking map 4802B comprises a PP4808B.

Also, as described above in connection with FIG. 14, some of the PPs are classified as PCFs, which are used to determine the orientation of the virtual content to render it to the user. May be good. FIG. 20B shows that an XR device worn by individual users 4802A, 4802B can create local PCF4810A, 4810B based on PP4808A, 4808B. FIG. 20C shows that persistent content 4812A, 4812B (eg, virtual content) can be associated with PCF4810A, 4810B by individual XR devices.

In this embodiment, the virtual content may have a virtual content coordinate frame that can be used by an application that produces the virtual content, regardless of how the virtual content should be displayed. The virtual content may be defined, for example, as a surface such as a mesh triangle at a particular location and angle with respect to the virtual content coordinate frame. In order to render the virtual content to the user, the location of their surface may be determined for the user who will perceive the virtual content.

Linking the virtual content to the PCF can simplify the calculations involved in determining the location of the virtual content for the user. The location of the virtual content for the user may be determined by applying a series of transformations. Some of those transformations can change and may be updated frequently. Others of those conversions can be stable and need not be updated very often or at all. Nevertheless, the transformation is applied with a relatively low computational burden so that the location of the virtual content is updated frequently to the user and can provide a realistic look to the rendered virtual content. obtain.

In the embodiment of FIGS. 20A-20C, the device of user 1 has a coordinate system that may be associated with a coordinate system that defines the origin of the map by the transformation rig1_T_w1. The device of user 2 has a similar conversion rig2_T_w2. These transformations are represented as 6 degree transformations and may specify translations and rotations to align the device coordinate system with the map coordinate system. In some embodiments, the transformation can be represented as two separate transformations, one defining translation and the other defining rotation. Therefore, it should be understood that the transformation can be expressed in a form that simplifies the calculation or otherwise provides an advantage.

The transformation between the origin of the tracking map and the PCF identified by the individual user device is represented as pcf1_T_w1 and pcf2_T_w2. In this example, PCF and PP are the same, as the same transformation also characterizes PP.

The location of the user device for the PCF can therefore be calculated by continuous application of these transformations such as rig1_T_pcf1 = (rig1_T_w1) ^* (pcf1_T_w1).

As shown in FIG. 20C, the virtual content is located relative to the PCF using the transformation of obj1_T_pcf1. The transformation may be set by an application that produces virtual content that may receive information from a world reconstruction system that describes a physical object for the PCF. To render the virtual content to the user, a transformation of the user's device into the coordinate system is calculated, which relates the virtual content coordinate frame to the origin of the tracking map through the transformation obj1_t_w1 = (obj1_T_pcf1) ^* (pcf1_T_w1). It can be calculated by letting it. The conversion may then be associated with the user's device through a further conversion rig1_T_w1.

The location of the virtual content can change based on the output from the application that produces the virtual content. As it changes, the end-to-end transformation from the source coordinate system to the destination coordinate system can be recalculated. In addition, the user's location and / or head posture may change as the user moves. As a result, any end-to-end conversion that depends on the user's location or head posture will change as well as the conversion rig1_T_w1 can change.

The transformation rig1_T_w1 may be updated with the user's movements based on tracking the user's position with respect to stationary objects in the physical world. Such tracking may be performed by a headphone tracking component that processes the sequence of images, or other components of the system, as described above. Such updates may be made by determining the user's attitude towards a stationary reference frame such as PP.

In some embodiments, the location and orientation of the user device may be determined for the nearest sustained posture, or in this embodiment the PCF as a PP is used as the PCF. Such a determination may be made by identifying feature points that characterize the PP within the current image captured using the sensor on the device. Image processing techniques such as stereoscopic image analysis may be used to determine the location of the device for those feature points. From this data, the system can calculate the transformation changes associated with the user's movements based on the relationship rig1_T_pcf1 = (rig1_T_w1) ^* (pcf1_T_w1).

The system may determine and apply the transformations in a computationally efficient order. For example, the need to calculate rig1_T_w1 from the measurements resulting in rig1_T_pcf1 is avoided by both tracking the user posture and defining the location of virtual content for the PP or PCF built on the persistent posture. obtain. Thus, the transformation of the virtual content from the source coordinate system to the destination coordinate system of the user's device may be based on the measured transformation according to the representation (rig1_T_pcf1) ^* (obj1_t_pcf1), the first transformation being the system. Measured by, the latter transformation is provided by the application that defines the virtual content for rendering. In an embodiment where the virtual content is positioned relative to the origin of the map, the end-to-end transformation may associate the virtual object coordinate system with the PCF coordinate system based on further transformations between the map coordinates and the PCF coordinates. In embodiments where the virtual content is positioned for a PP or PCF that is different from the one for which the user location is tracked, a transformation between the two may be applied. Such transformations may be fixed or, for example, determined from a map where both appear.

The transformation-based approach may be implemented within the device, with components, for example, processing sensor data and building tracking maps. As part of that process, those components may identify feature points that can be used as a sustained posture, which in turn may be converted to a PCF. While those components limit the number of persistent poses generated for the map and provide a suitable spacing between persistent poses, as described above in connection with FIGS. 17-19, while It may allow the user to be close enough to a sustained posture location, regardless of location in the physical environment, to accurately calculate the user's posture. To the user, the persistent attitude closest to the user depends on the location of the PP (or PCF, if used) as it is updated as a result of user movement, refinement to the tracking map, or other causes. Any of the transformations used to calculate the location of the virtual content may be updated and stored for use, at least until the user moves away from its persistent posture. It should be noted that by calculating and storing the conversion, the calculation burden each time the location of the virtual content is updated can be relatively low enough to be performed with a relatively short waiting time.

20A-20C illustrate the position with respect to the tracking map, and each device has its own tracking map. However, the transformation may be generated for any map coordinate system. The survival of content across user sessions in an XR system can be achieved by using persistent maps. The user's shared experience can also be facilitated by using a map, to which multiple user devices can be oriented.

In some embodiments, described in more detail below, the location of the virtual content is defined in relation to the coordinates in the canonical map, which is formatted so that one of a plurality of devices can use the map. May be good. Each device may maintain a tracking map and may determine changes in the user's attitude towards the tracking map. In this embodiment, the transformation between the tracking map and the criteria map may be determined through the process of "locating", which is the structure within the tracking map (such as one or more persistent postures) and the criteria. It may be implemented by matching one or more structures of the map (such as one or more PCFs).

Further described below are techniques for creating and using canonical maps in this way.

Deep keyframe

Techniques as described herein rely on image frame comparisons. For example, new images may be captured using a sensor worn by the user to establish the location of the device with respect to the tracking map, and the XR system may be a set of images used to create the tracking map. Within, an image may be searched that shares at least a predetermined amount of points of interest with the new image. As an example of another scenario involving comparison of image frames, the tracking map first finds an image frame associated with a persistent posture in the tracking map that is similar to the image frame associated with the PCF in the reference map. Thereby, it can be located with respect to the reference map. Alternatively, the transformation between the two canonical maps may be calculated by first finding similar image frames in the two maps.

Deep keyframes provide a method for reducing the amount of processing required to identify similar image frames. For example, in some embodiments, the comparison may be made between an image feature in a new 2D image (eg, "2D feature") and a 3D feature in the map. Such comparisons can be made in any suitable method, such as by projecting a 3D image into a 2D plane. Traditional methods, such as Word Bags (BoW), search for 2D features in a new image in the database, including all 2D features in the map, which is significant, especially when the map represents a large area. Calculation resources may be requested. Traditional methods then locate the image, sharing at least one of the 2D features with the new image, which is not useful for locating meaningful 3D features in the map. Can include. Conventional methods then locate 3D features that are not meaningful to the 2D features in the new image.

We have less memory resources (eg, a quarter of the memory resources used by BoW), higher efficiency (eg, 2.5 ms processing time per keyframe, for 500 keyframes). 100 μs) and higher accuracy (eg, 20% better read / replay than BoW for 1,024 dimensional models) and 5% better read / play than BoW for 256 dimensional models) were used. Recognizes and understands techniques for reading images in maps.

To reduce the calculation, a descriptor that can be used to compare an image frame to another image frame may be calculated for the image frame. Descriptors may be stored in place of, or in addition to, image frames and feature points. Persistent postures and / or PCFs can be generated from image frames, from which the descriptor of the image frame or multiple frames from which each persistent posture or PCF is generated is one of the persistent postures and / or PCFs. It may be stored as a part.

In some embodiments, the descriptor may be calculated as a function of feature points within the image frame. In some embodiments, the neural network is configured to calculate a unique frame descriptor to represent the image. The image may have a resolution higher than 1 megabyte so that sufficient details of the 3D environment within the field of view of the device worn by the user are captured in the image. The frame descriptor can be much smaller, such as a sequence of numbers, for example any number in or between 128 bytes to 512 bytes.

In some embodiments, the neural network is trained so that the calculated frame descriptor shows the similarity between the images. The image in the map is the closest image in the database that contains the image used to generate the map, which may have a frame descriptor within a given distance to the frame descriptor for the new image. Can be located by identifying. In some embodiments, the distance between the images may be represented by the difference between the frame descriptors of the two images.

FIG. 21 is a block diagram illustrating a system for generating descriptors for individual images, according to some embodiments. In the illustrated embodiment, the frame embedding generator 308 is shown. The frame embedding generator 308 may be used in conjunction with the server 20 in some embodiments, but as an alternative or in addition, in whole or in part, the XR devices 12.1 and 12.2, or others. It may be performed within one of any other device that processes the image for comparison with the image of.

In some embodiments, the frame embedding generator, despite the reduced size, still indicates content in the image, from initial size (eg, 76,800 bytes) to final size (eg, 256). It may be configured to generate a data representation of the image reduced to bytes). In some embodiments, the frame embedding generator may be used to generate a data representation for an image, which may be a keyframe or frame used in other ways. In some embodiments, the frame embedding generator 308 may be configured to transform an image at a particular location and orientation into a unique sequence of numbers (eg, 256 bytes). In the illustrated embodiment, the image 320 captured by the XR device may be processed by the feature extractor 324 to detect the point of interest 322 in the image 320. Points of interest may or may not be derived from the identified feature points, as described above for feature 1120 (FIG. 14) or otherwise as described herein. .. In some embodiments, points of interest may be represented by descriptors, as described above with respect to descriptor 1130 (FIG. 14), which can be generated using the deep sparse feature method. In some embodiments, each point of interest 322 may be represented by a sequence of numbers (eg, 32 bytes). For example, there may be n features (eg, 100), each feature represented by a 32-byte sequence.

In some embodiments, the frame embedding generator 308 may include a neural network 326. The neural network 326 may include a multi-layer perceptron unit 312 and a maximum (max) pool unit 314. In some embodiments, the multi-layer perceptron (MLP) unit 312 may include a multi-layer perceptron, which may be trained. In some embodiments, the point of interest 322 (eg, the descriptor for the point of interest) can be reduced by the Multilayer Perceptron 312 and output as a weighted combination of descriptors 310. For example, the MLP can reduce n features to m features, which is less than n features.

In some embodiments, the MLP unit 312 may be configured to perform matrix multiplication. The multi-layer perceptron unit 312 receives a plurality of points of interest 322 of the image 320 and converts each point of interest into an individual sequence of numbers (eg, 256). For example, there may be 100 features, each feature may be represented by a sequence of 256 numbers. The matrix may be made to have 100 horizontal parallels and 256 vertical columns in this embodiment. Each row may have a series of 256 numbers, some smaller, some larger, varying in size. In some embodiments, the output of the MLP may be an n × 256 matrix, where n represents the number of features extracted from the image. In some embodiments, the output of the MLP may be an m × 256 matrix, where m is the number of points of interest reduced from n.

In some embodiments, the MLP 312 may have a training phase and a working phase in which model parameters for the MLP are determined. In some embodiments, the MLP may be trained as illustrated in FIG. The input training data may include the data in three sets, the three sets comprising 1) a query image, 2) a positive sample and 3) a negative sample. The query image can be considered a reference image.

In some embodiments, the positive sample may comprise an image similar to the query image. For example, in some embodiments, similarity means having the same object in both the query and the positive sample image, but can be viewed from different angles. In some embodiments, similarity is having the same object in both the query and the positive sample image, but with respect to other images (eg, left, right, up, down). Can have an object to be.

In some embodiments, the negative sample may comprise an image that is not similar to the query image. For example, in some embodiments, dissimilar images may not contain any prominent objects in the query image, or only a small portion of the prominent objects in the query image (eg, <10%, 1%). ) Can be contained. Similar images, in contrast, may have, for example, the majority of objects (eg,> 50%, or> 75%) in the query image.

In some embodiments, the points of interest may be extracted from the images in the input training data or converted into feature descriptors. These descriptors may be calculated both for the training image and for the features extracted during operation of the frame embedding generator 308 of FIG. 21 as shown in FIG. In some embodiments, a Deep Sparseness Feature (DSF) process is used to generate a descriptor (eg, a DSF descriptor) as described in US Patent Application No. 16 / 190,948. May be good. In some embodiments, the DSF descriptor is n × 32 dimensions. The descriptor may then be passed through the model / MLP to produce a 256-byte output. In some embodiments, the model / MLP may have the same structure as the MLP 312 so that once the model parameters are set through training, the resulting trained MLP can be used as the MLP 312.

In some embodiments, the feature descriptor (eg, 256 bytes output from the MLP model) is then used only during the triplet margin loss module (training phase of the MLP neural network) and between the used phases. May not be used). In some embodiments, the triplet margin loss module reduces the difference between 256 bytes output from the query image and 256 bytes output from the positive sample, with 256 bytes output from the query image and negative. The parameters for the model may be configured to increase the difference from the 256 bytes output from the sample. In some embodiments, the training phase may include feeding multiple triplet input images into the learning process to determine model parameters. The training process may continue, for example, until the differences for positive images are minimized and the differences for negative images are maximized, or until other suitable termination criteria are reached.

With reference back to FIG. 21, the frame embedding generator 308 may include a pooling layer, which is illustrated here as the max pool unit 314. The maximum pool unit 314 may analyze each column to determine the maximum number within an individual column. The maximum pool unit 314 may combine the maximum value of each column of the number of output matrices of the MLP 312 into, for example, the global feature column 316 of the number of 256. It should be appreciated that images processed within an XR system can preferably have high resolution frames, potentially with millions of pixels. The global feature sequence 316 is a relatively small number that occupies a relatively small amount of memory and is easily searchable compared to an image (eg, with a resolution higher than 1 megabyte). Therefore, it is possible to retrieve the image without analyzing each original frame from the camera, and it is cheaper to store 256 bytes instead of a complete frame.

FIG. 22 is a flow chart illustrating a method 2200 for calculating an image descriptor according to some embodiments. Method 2200 may start from the step of receiving a plurality of images captured by an XR device worn by the user (act 2202). In some embodiments, the method 2200 may include determining one or more keyframes from a plurality of images (act 2204). In some embodiments, the act 2204 may be skipped and / or instead may occur after step 2210.

Method 2200 uses an artificial neural network to identify one or more points of interest in a plurality of images (act 2206) and uses an artificial neural network to provide a feature descriptor for each point of interest. It may include a step to calculate (act 2208). The method uses an artificial neural network, at least in part, for each image to provide a frame descriptor to represent the image based on the calculated feature descriptor for the identified points of interest in the image. It may include a step to calculate (act 2210).

FIG. 23 is a flow chart illustrating a location-locating method 2300 using an image descriptor, according to some embodiments. In this embodiment, a new image frame depicting the current location of the XR device is with the image frame stored in relation to a point in the map (such as a persistent posture or PCF, as described above). May be compared. Method 2300 may start from the step of receiving a new image captured by the XR device worn by the user (act 2302). Method 2300 may include identifying one or more closest keyframes in a database (act 2304), comprising keyframes used to generate one or more maps. In some embodiments, the nearest keyframe may be identified based on rough spatial information and / or previously determined spatial information. For example, rough spatial information may indicate that the XR device is within the geographic area represented by the 50m x 50m area of the map. Image matching may be performed only on points within the area. As another embodiment, based on tracking, the XR system moves the XR device in the direction of the second sustained attitude in the map, where the XR device was previously in close proximity to the first sustained attitude in the map. You may know what was happening. The second persistent posture can be considered the closest persistent posture, and the keyframes stored with it can be considered the closest keyframe. Alternatively, or in addition, GPS data or other metadata such as WiFi finger prints may also be used to select the nearest keyframe or set of nearest keyframes.

Regardless of how the nearest keyframe is selected, the frame descriptor is used to determine if the new image matches any of the selected frames to be associated with the persistent pose of the neighborhood. May be done. The decision compares the frame descriptor of the new image with the frame descriptor of the nearest keyframe or subset of keyframes in the database selected in any other suitable way, and determines the given frame descriptor of the new image. This may be done by selecting a keyframe with a frame descriptor within the distance. In some embodiments, the distance between the two frame descriptors may be calculated by obtaining the difference between the columns of two numbers that may represent the two frame descriptors. In embodiments where the columns are treated as columns of multiple quantities, the differences may be calculated as vector differences.

Once the matching image frame is identified, the orientation of the XR device with respect to that image frame can be determined. Method 2300 is a step of performing feature matching against a 3D feature in a map corresponding to the nearest identified key frame (act 2306) and a device worn by the user based on the feature matching result. It may include a step of calculating the posture (act 2308). Thus, the computationally intensive matching of feature points in the two images may be performed on only one image that has already been determined to be a likely match for the new image.

FIG. 24 is a flow chart illustrating a method 2400 for training a neural network, according to some embodiments. Method 2400 may start from the step of generating a dataset (act 2402), which comprises a plurality of image sets. Each of the plurality of image sets may include a query image, a positive sample image, and a negative sample image. In some embodiments, the plurality of image sets may include, for example, a synthetic record pair configured to teach a neural network basic information such as shape. In some embodiments, the plurality of image sets may include actual recording pairs that can be recorded from the physical world.

In some embodiments, the inliar may be calculated by fitting the elementary matrix between the two images. In some embodiments, sparse duplication may be calculated as an intersection (IoU) on the union of points of interest found in both images. In some embodiments, the positive sample may include at least 20 points of interest that are identical in the query image and serve as inliers. Negative samples may contain less than 10 inlier points. Negative samples may have less than half the sparse points that overlap the analysis points of the query image.

Method 2400 may include the step of calculating the loss (act 2404) by comparing the query image with the positive and negative sample images for each image set. In method 2400, the distance between the frame descriptor generated by the artificial neural network for the query image and the frame descriptor for the positive sample image is the frame descriptor for the query image and the negative sample image. It may include a step (act 2406) of modifying the artificial neural network based on the calculated loss so that it is less than the distance to the frame descriptor for.

Methods and devices configured to generate global descriptors for individual images are described above, but methods and devices are configured to generate descriptors for individual maps. Please understand that it is okay. For example, the map may contain multiple keyframes, each with a frame descriptor as described above. The largest pool unit may analyze the frame descriptors of the map's keyframes and combine the frame descriptors into a unique map descriptor for the map.

In addition, it should be understood that other architectures may also be used for processing, as described above. For example, a separate neural network is described to generate DSF and frame descriptors. Such an approach is computationally efficient. However, in some embodiments, the frame descriptor may be generated from selected feature points without first generating a DSF descriptor.
Map ranking and merging

Described herein are methods and devices for ranking and merging multiple environmental maps within an X Reality (XR) system. Map merging can allow maps that represent overlapping parts of the physical world to be combined to represent larger areas. Ranking maps makes it possible to efficiently implement techniques as described herein, including map merging, which involves selecting maps from a set of maps based on similarity. obtain. In some embodiments, the system may maintain, for example, a set of canonical maps that are formatted in a manner accessible by any of several XR devices. These canonical maps may be formed by merging selected tracking maps from those devices with other tracking maps or previously stored canonical maps. Criteria maps can be used, for example, to select one or more canon maps and merge them with a new tracking map, and / or to select one or more canon maps from a set and use them within the device. It may be ranked.

In order to provide the user with a realistic XR experience, the XR system must know the user's physical surroundings in order to properly correlate the location of the virtual object in relation to the real object. Information about the user's physical surroundings may be obtained from an environmental map of the user's location.

We have an XR system that enables efficient sharing of real / physical world environmental maps collected by multiple users so that those users are present in the world at the same or different times. Recognizing and recognizing its true value, it can provide an improved XR experience to multiple users who share the same world, with or without real and / or virtual content. However, there are significant challenges in providing such a system. Such a system may store multiple maps generated by multiple users, and / or the system may store multiple maps generated at different times. For example, as described above, with respect to actions that can be performed using previously generated maps, such as location identification, the substantive processing is XR the relevant environment map of the same world (eg, the same real world location). May be required to identify from all environment maps collected within the system. In some embodiments, there may be only a small number of environmental maps that the device can access, for example, for location. In some embodiments, there may be a large number of environmental maps that the device can access. We recognize a technique for quickly and accurately ranking environmental map relevance from an environmental map that is considered as any possibility, for example, the population of all criterion maps 120 in FIG. 28. I acknowledged the true value. The high-ranked map then renders the virtual object on the user display to interact realistically with the physical world around the user, or merges the map data collected by the user with the stored map. It may be selected for further processing, such as to create a larger or more accurate map.

In some embodiments, even if the stored map associated with the task for the user at a location within the physical world is identified by filtering the stored map based on multiple criteria. good. Those criteria may show a comparison of the tracking map generated by the user's wearable device at that location with the candidate environment map stored in the database. The comparison is based on the metadata associated with the map, such as Wi-Fi finger prints, detected by the device that produces the map, and / or the set of BSSIDs to which the device is attached while forming the map. It may be carried out. The comparison may also be based on the compressed or decompressed content of the map. Comparisons based on compressed representations may be performed, for example, by comparing vectors calculated from map content. Comparisons based on the unzipped map may be performed, for example, by locating the tracking map within the stored map and vice versa. Multiple comparisons may be performed in order based on the calculation time required to reduce the number of candidate maps for consideration, and comparisons with fewer calculations require more calculations. It is performed in the order prior to the other comparisons.

FIG. 26 depicts an AR system 800 configured to rank and merge one or more environmental maps, according to some embodiments. The AR system may include a passable world model 802 for AR devices. The information for capturing the passable world model 802 may come from a sensor on the AR device, which maps the sensor data stored in the processor 804 (eg, the local data processing module 570 in FIG. 4). It may include computer executable instructions that may perform some or all of the processing to convert to. Such a map may be a tracking map such that it can be constructed as sensor data is collected as the AR device operates in the area. Along with the tracking map, area attributes may be supplied to indicate the area represented by the tracking map. These area attributes may be geographic location identifiers such as IDs used by the AR system to represent coordinates or locations presented as latitude and longitude. Alternatively, or in addition, the area attribute may be a measured property with a high likelihood that is unique for that area. Area attributes may be derived, for example, from the parameters of the radio network detected within the area. In some embodiments, the area attribute may be associated with the unique address of the access point to which the AR system is located and / or connected. For example, area attributes may be associated with 5G base stations / routers, Wi-Fi routers, and equivalent MAC addresses or basic service set identifiers (BSSIDs).

In the embodiment of FIG. 26, the tracking map may be merged with other maps of the environment. The map ranking portion 806 receives the tracking map from the device PW802, communicates with the map database 808, and selects and ranks the environment map from the map database 808. The higher ranked selected map is sent to the map merge portion 810.

The map merge portion 810 may perform the merge process on the map transmitted from the map ranking portion 806. The merge process may involve merging some or all of the tracking maps and ranked maps and transmitting the new merged map to the passable world model 812. The map merge portion may merge maps by identifying maps that depict overlapping parts of the physical world. Those overlaps may be aligned so that the information in both maps can be aggregated into the final map. The criteria map may be merged with other criteria maps and / or tracking maps.

Aggregation may involve the step of extending one map with information from another map. Alternatively, or in addition, the aggregation may involve adjusting the representation of the physical world in one map based on the information in another map. The latter map may represent, for example, that the object that gives rise to the feature point has moved so that the map can be updated based on the latter information. Alternatively, the two maps may characterize the same region with different feature points, and the aggregation may involve selecting a set of feature points from the two maps and better representing the region. .. Regardless of the specific processing that occurs in the process of merging, in some embodiments PCFs from all the maps to be merged are reserved so that the application that positions the content to them can continue to do so. May be done. In some embodiments, map merging can result in redundant persistent postures, some of which may be removed. When the PCF is associated with a persistent posture to be deleted, the step of merging the map may involve modifying the PCF so that it is associated with the persistent posture that remains in the map after the merge.

In some embodiments, they may be refined as the maps are expanded and / or updated. The refinement may involve calculations to reduce internal discrepancies between feature points that are likely to represent the same object in the physical world. Mismatches can result from inaccuracies in the pose associated with keyframes that provide feature points that represent the same object in the physical world. Such discrepancies can result, for example, from an XR device that calculates the attitude to the tracking map, which in turn accumulates errors within the attitude estimation and creates a "drift" over time within the attitude accuracy. As such, it is constructed based on the steps of estimating the posture. The map may be refined by performing bundle adjustments or other actions to reduce feature point discrepancies from multiple keyframes.

Depending on the refinement, the location of the persistent points relative to the origin of the map can change. Therefore, the transformation associated with a persistent posture or its persistent point, such as PCF, can also change. In some embodiments, the XR system is associated with any persistence that has changed in relation to map refinement (whether performed as part of a merge operation or for other reasons). The conversion to be performed may be recalculated. These transformations can be pushed from the component that calculates the transformation to the component that uses the transformation so that any use of the transformation can be based on the updated location of the persistence point.

The passable world model 812 may be a cloud model, which may be shared by multiple AR devices. The passable world model 812 may store the environment map in the map database 808 or otherwise have access to it. In some embodiments, when a previously calculated environment map is updated, previous versions of that map may be removed to remove the stale map from the database. In some embodiments, when a previously calculated environment map is updated, previous versions of the map may be archived and read / view of previous versions of the environment may be enabled. In some embodiments, the permission may be set so that only AR systems with certain read / write access can trigger the deletion / archiving of previous versions of the map.

These environment maps created from tracking maps supplied by one or more AR devices / systems may be accessed by AR devices within the AR system. The map ranking portion 806 may also be used in supplying the environment map to the AR device. The AR device may send a message requesting an environment map for its current location, and the map ranking portion 806 may be used to select and rank the environment map associated with the requesting device. ..

In some embodiments, the AR system 800 may include a downsampling portion 814 configured to receive the merged map from the cloud PW812. The map received and merged from the cloud PW812 may be a storage format for the cloud, which may be a large number of PCFs per square meter or high features such as multiple image frames or large sets of feature points associated with PCFs. It may include resolution information. The downsampling portion 814 may be configured to downsample the cloud format map to a format suitable for storage on the AR device. The device format map has less data, such as less PCF or less data stored per PCF, and may accommodate the limited local computing power and storage space of the AR device.

FIG. 27 is a simplified block diagram illustrating a plurality of reference maps 120 that may be stored in a remote storage medium, eg, a cloud. Each canonical map 120 may include a plurality of canonical map identifiers indicating the location of the canonical map in physical space, such as any location on the planet Earth. These canonical map identifiers include the following identifiers, namely area identifiers represented by longitude and latitude ranges, frame descriptors (eg, global feature sequence 316 in FIG. 21), Wi-Fi finger prints, and It may include one or more of a feature descriptor (eg, feature descriptor 310 in FIG. 21) and a device identification indicating one or more devices that contributed to the map.

In the illustrated embodiment, the reference map 120 may be on the surface of the earth and is therefore geographically arranged in a two-dimensional pattern. The criterion map 120 may be uniquely identifiable by the corresponding longitude and latitude, as any canon map with overlapping longitudes and latitudes can be merged into the new canon map.

FIG. 28 is a schematic diagram illustrating, according to some embodiments, a method of selecting a canonical map that can be used to locate a new tracking map with respect to one or more canonical maps. As an embodiment, the method may start from the step (act 120) of accessing the population of reference maps 120, which may be stored in a database in passable worlds (eg, passable worlds module 538). The canonical map population may include canonical maps from all previously visited locations. The XR system may filter the population of all canonical maps to only a small subset or a single map. It should be understood that in some embodiments it may not be possible to send all the canonical maps to the visual device due to bandwidth limitations. Choosing the subset selected as a likely candidate to match the tracking map for sending to the device can reduce the bandwidth and latency associated with accessing the map's remote database. ..

The method may include filtering the population of the canonical map (act 300) based on the area with a given size and shape. In the embodiment illustrated in FIG. 27, each square may represent an area. Each square may cover 50m x 50m. Each square may have six nearby areas. In some embodiments, the act 300 covers at least one longitude and latitude, including that longitude and latitude of the location identifier received from the XR device, as long as at least one map is present at that longitude and latitude. One matching criterion map 120 may be selected. In some embodiments, the act 300 may select at least one neighborhood canonical map that is adjacent to the matching canonical map and covers longitude and latitude. In some embodiments, the act 300 may select a plurality of matching criteria maps and a plurality of neighborhood criteria maps. Act 300 may, for example, reduce the number of canonical maps by about one tenth, eg, thousands to hundreds, to form a first filtered selection. Alternatively, or in addition, criteria other than latitude and longitude may be used to identify nearby maps. The XR device may have been previously located, for example, as part of the same session, using a canonical map within the set. The cloud service may reserve information about the XR device, including previously located maps. In this embodiment, the map selected in the act 300 may include one covering an area adjacent to the map in which the XR device is located.

The method may include filtering the first filtered selection of the canonical map based on the Wi-Fi finger print (act 302). Act 302 may determine the latitude and longitude based on the Wi-Fi finger print received as part of the location identifier from the XR device. Act 302 may compare the latitude and longitude from the Wi-Fi finger print to the latitude and longitude of the criterion map 120 to determine one or more criterion maps that form a second filtered selection. Act 302 may reduce the number of canonical maps to about one tenth, eg, hundreds to dozens of canonical maps (eg, 50) forming a second choice. For example, the first filtered selection may include 130 canonical maps, the second filtered selection may include 50 out of 130 canonical maps, and out of 130 canonical maps. Cannot contain the remaining 80 of.

The method may include filtering a second filtered selection of the canonical map based on a key frame (act 304). Action 304 may compare the data representing the image captured by the XR device with the data representing the reference map 120. In some embodiments, the data representing the image and / or map may include a feature descriptor (eg, DSF descriptor in FIG. 25) and / or a global feature sequence (eg, 316 in FIG. 21). Act 304 may provide a third filtered selection of canonical maps. In some embodiments, the output of the act 304 may be, for example, only 5 of the 50 criterion maps identified following the second filtered selection. The map transmitter 122 then transmits one or more canonical maps to the visual device based on a third filtered selection. Act 304 may reduce the number of canonical maps to about one tenth, forming a third choice, eg, from tens to single digit number of canonical maps (eg, 5). In some embodiments, the XR device may receive a canonical map within a third filtered selection and attempt to locate it within the received canonical map.

For example, action 304 is based on the global feature sequence 316 of the reference map 120 and the global feature sequence 316 based on the image captured by the visual device (eg, an image that can be part of a local tracking map for the user). , Criteria map 120 may be filtered. Each of the criteria maps 120 in FIG. 27 therefore has one or more global feature columns 316 associated with it. In some embodiments, the global feature sequence 316 may be obtained when the XR device submits the image or feature details to the cloud, where the cloud processes the image or feature details and for the reference map 120. Generate the global feature sequence 316.

In some embodiments, the cloud may receive feature details of the live / new / current image captured by the visual device, and the cloud may generate a global feature sequence 316 for the live image. good. The cloud may then filter the criterion map 120 based on the live global feature column 316. In some embodiments, the global feature sequence may be generated on a local visual device. In some embodiments, the global feature sequence may be generated remotely, eg, on the cloud. In some embodiments, the cloud may transmit the filtered criteria map to the XR device along with the global feature column 316 associated with the filtered criteria map. In some embodiments, when the visual device locates its tracking map relative to the canonical map, it does so by matching the global feature column 316 of the local tracking map with the global feature column of the canonical map. May be good.

It should be understood that the operation of the XR device does not have to perform all of the actions (300, 302, 304). For example, if the population of canonical maps is relatively small (eg, 500 maps), the XR device attempting to locate will be a Wi-Fi finger print (eg, act 302) and a keyframe (eg, act 302). The population of the canonical map is filtered based on the act 304), but the filtering based on the area (eg, act 300) may be omitted. Moreover, the maps do not need to be compared as a whole. In some embodiments, comparison of two maps, for example, may result in the identification of common persistent points such as persistent posture or PCF that appear in both the new map and the map selected from the map population. .. In that case, the descriptors may be associated with persistence points and those descriptors may be compared.

FIG. 29 is a flow chart illustrating a method 900 for selecting one or more ranked environment maps according to some embodiments. In the illustrated embodiment, the ranking step is performed for the user's AR device to create a tracking map. Therefore, tracking maps are available for use in ranking environmental maps. In embodiments, where the tracking map is not available, some or all of the steps of selecting and ranking the environment map that do not explicitly rely on the tracking map may be used.

Method 900 may start from act 902 and a set of maps (which can be formatted as a canonical map) from a database of environmental maps near the location where the tracking map was formed is accessed and then ranked. Can be filtered to. In addition, in act 902, at least one area attribute with respect to the area in which the user's AR device is operating is determined. In a scenario where the user's AR device is building a tracking map, the area attributes may correspond to the area over which the tracking map is created. As a specific embodiment, the area attributes may be calculated based on the signal received from the access point to the computer network while the AR device was calculating the tracking map.

FIG. 30 illustrates an exemplary map ranking portion 806 of an AR system 800, according to some embodiments. The map ranking portion 806 may be performed within a cloud computing environment as it may include a portion that runs on an AR device and a portion that runs on a remote computing system such as the cloud. The map ranking portion 806 may be configured to implement at least a portion of Method 900.

FIG. 31A illustrates examples of the area attributes AA1-AA8 of the tracking map (TM) 1102 and the environment maps CM1-CM4 in the database, according to some embodiments. As shown, the environment map may be associated with multiple area attributes. Area attributes AA1-AA8 are network parameters detected by the AR device that calculates the tracking map 1102, eg, the network basic service set identifier (BSSID) to which the AR device is connected, and / or. For example, it may include the signal strength of the access point received by the wireless network through the network tower 1104. Wireless network parameters may be protocol compliant, including Wi-Fi and 5G NR. In the embodiment illustrated in FIG. 32, the area attribute is a finger print of the area in which the user AR device has collected sensor data and formed a tracking map.

FIG. 31B illustrates an embodiment of the determined geographic location 1106 of tracking map 1102, according to some embodiments. In the illustrated embodiment, the determined geographic location 1106 includes a center of gravity point 1110 and an area 1108 that surrounds the center of gravity point. It should be understood that the geographic location determination of the present application is not limited to the formats shown. The determined geographic location may have any suitable format, including, for example, different area shapes. In this embodiment, the geographic location is determined from the area attributes using a database that associates the area attributes with the geographic location. The database is a commercially available, eg, latitude and longitude, Wi-Fi finger print location-related database that can be used for this operation.

In the embodiment of FIG. 29, the map database comprising environment maps may also include location data for those maps, including latitudes and longitudes covered by the maps. The process in act 902 may involve selecting from its database a set of environmental maps that cover the same latitude and longitude determined for the area attributes of the tracking map.

Act 904 is the first filtering of the set of environment maps accessed in act 902. In Act 902, the environmental map is reserved in the set based on the proximity of the tracking map to the geographic location. This filtering step may be performed by comparing the latitude and longitude associated with the tracking map and the environment map in the set.

FIG. 32 illustrates an embodiment of Act 904 according to some embodiments. Each area attribute may have a corresponding geographic location 1202. The set of environment maps may include an environment map with at least one area attribute that has a geographic location that overlaps the determined geographic location of the tracking map. In the illustrated embodiment, each set of identified environment maps has at least one area attribute, having a geographic location that overlaps the determined geographic location of tracking map 1102, environment maps CM1, CM2. , And CM4. The environment map CM3 associated with the area attribute AA6 is not included in the set because it is outside the determined geographic location of the tracking map.

Other filtering steps are also ultimately processed (for map merging, or for providing passable worlds information to user devices, etc.) to reduce / rank the number of environment maps in the set. It may be carried out on a set of environment maps. Method 900 may include filtering the set of environment maps (Act 906) based on the similarity of one or more identifiers of network access points associated with the environment map of the set of tracking maps and environment maps. During map formation, devices that collect sensor data and generate maps can be connected to the network through network access points, such as through Wi-Fi or similar wireless communication protocols. The access point can be identified by BSSID. User devices may connect to multiple different access points as they travel through the area, collect data, and form maps. Similarly, when multiple devices provide information to form a map, the devices may be connected through different access points and, therefore, for the same reason, used in forming the map. There can be multiple access points. Therefore, there can be multiple access points associated with a map, and a set of access points can be an indication of the location of the map. The strength of the signal from the access point, which can be reflected as an RSSI value, may provide additional geographic information. In some embodiments, the list of BSSID and RSSI values may form an area attribute for the map.

In some embodiments, the step of filtering a set of environment maps based on the similarity of one or more identifiers of a network access point is within the set of environment maps to one or more identifiers of the network access point. Based on this, it may include the step of reserving an environment map with the highest Jaccard similarity to at least one area attribute of the tracking map. FIG. 33 illustrates an embodiment of Act 906 according to some embodiments. In the illustrated embodiment, the network identifier associated with the area attribute AA7 may be determined as the identifier for tracking map 1102. The set of environment maps after act 906 includes environment map CM2, which may have an area attribute within a higher Jaccard similarity with AA7, and environment map CM4, which also includes area attribute AA7. The environment map CM1 is not included in the set because it has the lowest Jaccard similarity with AA7.

The processing in Acts 902-906 may be performed based on the metadata associated with the map without actually accessing the contents of the map stored in the map database. Other processes may involve steps to access the contents of the map. Act 908 shows the steps to access the environment map that remains in the subset after filtering based on the metadata. It should be understood that this action may be performed either earlier or later in the process if subsequent actions can be performed with the content being accessed.

Method 900 may include filtering the set of environment maps (Act 910) based on the similarity of the metrics representing the contents of the environment map of the tracking map and the set of environment maps. Metrics representing the content of tracking maps and environment maps may include a vector of values calculated from the content of the map. For example, a deep keyframe descriptor as described above, calculated for one or more keyframes used in forming a map, is a metric for comparing the map or parts of the map. Can be provided. Metrics may be calculated from the maps read in Act 908, or may be pre-calculated and stored as metadata associated with those maps. In some embodiments, the step of filtering a set of environment maps based on the similarity of the metrics that represent the contents of the environment map of the tracking map and the set of environment maps is a characteristic of the tracking map within the set of environment maps. It may include a step of reserving an environment map with a minimum vector distance between the vector of and the vector representing the environment map in the set of environment maps.

Method 900 may include further filtering the set of environmental maps (Act 912) based on the degree of matching between a portion of the tracking map and a portion of the environmental map of the set of environmental maps. The degree of matching may be determined as part of the positioning process. As a non-limiting example, locating may be performed by identifying key points in tracking and environmental maps that are sufficiently similar that they can represent the same part of the physical world. In some embodiments, the key points may be features, feature descriptors, keyframes, keyrigs, persistent postures, and / or PCFs. The set of key points in the tracking map can then be aligned to produce the best fit with the set of key points in the environmental map. If the mean square distance between the corresponding key points can be calculated and falls below a threshold for a particular region of the tracking map, the tracking map and environment map are used as an indication to represent the same region of the physical world.

In some embodiments, the step of filtering the set of environment maps based on the degree of matching between part of the tracking map and part of the environment map of the set of environment maps is the environment map of the set of environment maps. An environment map with a calculated volume larger than the set's filtered out environment map within a set of environment maps, with steps to calculate the volume of the physical world represented by the tracking map, also represented within. May include a step to reserve. FIG. 34 illustrates an embodiment of Act 912 according to some embodiments. In the illustrated embodiment, the set of environment maps after act 912 includes environment map CM4 having an area 1402 matched to the area of tracking map 1102. The environment map CM1 is not included in the set because it does not have an area matched with the area of the tracking map 1102.

In some embodiments, the set of environment maps may be filtered in the order of actions 906, actions 910, and actions 912. In some embodiments, the set of environment maps may be filtered based on actions 906, actions 910, and actions 912, which is the processing required to perform filtering from lowest to highest. It can be carried out in the order based on. Method 900 may include loading an environment map and a set of data (Act 914).

In the illustrated embodiment, the user database stores an area identification that indicates the area in which the AR device was used. The area identification may be an area attribute, which may include radio network parameters detected by the AR device during use. The map database may store multiple environment maps constructed from the data supplied by the AR device and the associated metadata. The associated metadata may include an area identification derived from the area identification of the AR device that supplied the data from which the environment map was constructed. The AR device may send a message to the PW module to indicate that a new tracking map is being created or is being created. The PW module may calculate the area identifier for the AR device and update the user database based on the received parameters and / or the calculated area identifier. The PW module also determines the area identifier associated with the AR device requesting the environment map, identifies the set of environment maps from the map database based on the area identifier, filters the set of environment maps, and filters the environment. A set of maps may be transmitted to the AR device. In some embodiments, the PW module comprises, for example, the geographic location of a tracking map, the similarity of one or more identifiers of a network access point associated with an environment map of a set of tracking maps and environment maps, tracking maps and environment. Based on one or more criteria, including the similarity of the metrics that represent the contents of the environment map in the set of maps, and the degree of matching between part of the tracking map and part of the environment map in the set of environment maps. You may filter the set of environment maps.

Although some aspects of some embodiments have been described so far, it should be appreciated that various modifications, modifications, and improvements will be readily recalled to those of skill in the art. As an embodiment, embodiments are described in relation to an augmented reality (AR) environment. It should be understood that some or all of the techniques described herein may be applied within an MR environment, more generally other XR and VR environments.

As another embodiment, embodiments are described in the context of devices such as wearable devices. Some or all of the techniques described herein may be implemented via any suitable combination of networks (clouds, etc.), discrete applications, and / or devices, networks, and discrete applications. I want you to understand.

In addition, FIG. 29 provides examples of criteria that can be used to filter candidate maps and result in a set of highly ranked maps. Other criteria may be used in lieu of or in addition to the criteria described. For example, if multiple candidate maps have similar values for the metrics used to filter out less desirable maps, the characteristics of the candidate map determine which map is reserved or filtered out as a candidate map. May be used to. For example, a larger or more dense candidate map may take precedence over a smaller candidate map. In some embodiments, FIGS. 27-28 may illustrate all or part of the systems and methods described in FIGS. 29-34.

35 and 36 are schematic diagrams illustrating an XR system configured to rank and merge multiple environmental maps, according to some embodiments. In some embodiments, passable worlds (PWs) may determine when to trigger a step to rank and / or merge maps. In some embodiments, the step of determining the map to be used is, according to some embodiments, at least in part, based on the deep keyframes described above in relation to FIGS. 21-25. May be good.

FIG. 37 is a block diagram illustrating a method 3700 for creating an environmental map of the physical world according to some embodiments. Method 3700 locates a tracking map captured by a user-worn XR device with respect to a group of canonical maps (eg, the canonical map selected by the method of FIG. 28 and / or the method 900 of FIG. 29). You may start from step (act 3702). Act 3702 may include the step of locating the key rig of the tracking map within the set of reference maps. The position-specific result of each key rig may include the position-specified posture of the key rig and a set corresponding to 2D / 3D features.

In some embodiments, method 3700 may include splitting the tracking map into connected components (act 3704), which merges the map robustly by merging the connected pieces. Can make it possible. Each connected component may include a key rig that is within a predetermined distance. Method 3700 may include a step of merging connected components larger than a predetermined threshold into one or more reference maps (act 3706) and a step of removing the merged connected components from the tracking map. good.

In some embodiments, method 3700 may include merging a group canonical map (act 3708), which is merged with the same connected component of the tracking map. In some embodiments, method 3700 may include the step (act 3710) of promoting the remaining connected components of the tracking map to the canonical map, which are not merged with any canonical map. In some embodiments, method 3700 also includes the step of merging the canonical map (Act 3712), which is merged with the persistent posture of the tracking map and / or the PCF and at least one connected component of the tracking map. good. In some embodiments, method 3700 may include, for example, the step of completing a canonical map (act 3714) by fusing map points and pruning redundant key rigs.

FIGS. 38A and 38B illustrate an environment map 3800 created by updating the reference map 700, which may be promoted from the tracking map 700 (FIG. 7) with a new tracking map, according to some embodiments. As illustrated and illustrated with respect to FIG. 7, the reference map 700 may provide a floor plan 706 of the reconstructed physical object within the corresponding physical world represented by the point 702. In some embodiments, the map point 702 may represent a feature of a physical object that may contain multiple features. The new tracking map may be captured for the physical world, uploaded to the cloud and merged with Map 700. The new tracking map may include map points 3802 and key rigs 3804, 3806. In the illustrated embodiment, the key rig 3804 represents a key rig successfully positioned relative to the reference map, for example by establishing a correspondence with the key rig 704 of the map 700 (as illustrated in FIG. 38B). ). On the other hand, the key rig 3806 represents a key rig that is not located relative to the map 700. The key rig 3806 may be promoted to a separate criteria map in some embodiments.

39A-39F are schematic diagrams illustrating an embodiment of a cloud-based persistent coordinate system that provides a shared experience for users within the same physical space. FIG. 39A shows, for example, that the reference map 4814 from the cloud is received by the XR device worn by the users 4802A and 4802B of FIGS. 20A-20C. The reference map 4814 may have a reference coordinate frame 4806C. Criteria map 4814 may have a PCF4810C with a plurality of associated PPs (eg, 4818A, 4818B in FIG. 39C).

FIG. 39B shows that the XR device has established a relationship between its individual world coordinate system 4806A, 4806B and the reference coordinate frame 4806C. This may be done, for example, by locating the reference map 4814 on a separate device. Positioning the tracking map relative to the canonical map can result in a transformation between its local world coordinate system and the coordinate system of the canonical map on a device-by-device basis.

FIG. 39C shows that the transformation can be calculated between the local PCFs on the individual devices (eg PCF4810A, 4810B) and the individual persistent attitudes on the reference map (eg PP4818A, 4818B) as a result of the localization. (Eg, conversions 4816A, 4816B). By using these transformations, each device may use its local PCF, which processes the image detected using the sensor on the device, determines the location for the local device, PP4818A, 4818B or It can be detected locally on the device by displaying virtual content that is tied to other persistence points in the canonical map. Such an approach may accurately locate the virtual content to each user and allow each user to have the same experience of the virtual content within the physical space.

FIG. 39D shows a continuous posture snapshot from the canonical map to the local tracking map. As can be seen from the figure, the local tracking maps are interconnected via a sustained attitude. FIG. 39E shows that the PCF4810A on the device worn by the user 4802A is accessible through the PP4818A into the device worn by the user 4802B. FIG. 39F shows that tracking maps 4804A, 4804B and criteria 4814 can be merged. In some embodiments, some PCFs may be removed as a result of the merge. In the illustrated embodiment, the merged map includes the PCF4810C of the reference map 4814 but does not include the PCF4810A, 4810B of the tracking maps 4804A, 4804B. The PP previously associated with PCF4810A, 4810B may be associated with PCF4810C after map merging.

40 and 41 illustrate an example of generating a tracking map with the first XR device 12.1 of FIG. FIG. 40 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1) according to some embodiments, which can be generated by the first XR device of FIG. FIG. 41 is a block diagram illustrating a step of uploading a map 1 from a first XR device to the server of FIG. 9 according to some embodiments.

FIG. 40 illustrates map 1 and virtual content (content 123 and content 456) on the first XR device 12.1. Map 1 has an origin (origin 1). Map 1 includes several PCFs (PCFa-PCFd). From the viewpoint of the first XR device 12.1. It has Y and Z coordinates (-1,0,0). Content 123 is associated with PCFa. In this embodiment, the content 123 has an X, Y, and Z relationship with respect to PCFa of (1,0,0). Content 456 has a relationship to PCFb. In this embodiment, the content 456 has an X, Y, and Z relationship of (1,0,0) with respect to the PCFb.

In FIG. 41, the first XR device 12.1 uploads the map 1 to the server 20. The server 20 here has a canonical map based on the map 1. The first XR device 12.1 has a canonical map that is empty at this stage. For the purposes of discussion, the server 20, in some embodiments, does not include maps other than map 1. The map is not stored on the second XR device 12.2.

The first XR device 12.1 also transmits its Wi-Fi signature data to the server 20. Server 20 uses Wi-Fi signature data to record GPS for such other devices, along with intelligence gathered from other devices that were previously connected to server 20 or other servers. A rough location for the first XR device 12.1 may be determined based on the location. The first XR device 12.1 can now terminate the first session (see FIG. 8) and disconnect from the server 20.

42 is a schematic diagram illustrating the XR system of FIG. 16 according to some embodiments, wherein the second user 14.2 after the first user 14.1 ends the first session. Indicates that the second XR device of the XR system has been used to initiate a second session. FIG. 43A is a block diagram showing the start of a second session by a second user 14.2. The first user 14.1 is shown by an imaginary line because the first session by the first user 14.1 has ended. The second XR device 12.2 begins recording the object. Various systems with variable particle size are used to determine that the second session by the second XR device 12.2 is in the same neighborhood as the first session by the first XR device 12.1. May be used by 20. For example, Wi-Fi signature data, Global Positioning System (GPS) positioning data, GPS data based on Wi-Fi signature data, or any other data indicating a location, first and for record of the location. It may be included in the second XR devices 12.1 and 12.2. Alternatively, the PCF identified by the second XR device 12.2 may show similarities to the PCF in Map 1.

As shown in FIG. 43B, the second XR device boots up and begins collecting data such as images 1110 from one or more cameras 44, 46. As shown in FIG. 14, in some embodiments, the XR device (eg, a second XR device 12.2) collects one or more images 1110, performs image processing, and performs one or more images. Features / points of interest 1120 may be extracted. Each feature may be converted to descriptor 1130. In some embodiments, descriptor 1130 may be used to illustrate key frame 1140, which may have the position and orientation of the associated associated image. One or more keyframes 1140 may correspond to a threshold distance from the previous persistent posture 1150, eg, a single persistent posture 1150 that may be automatically generated after 3 meters. One or more sustained postures 1150 may correspond to a single PCF 1160 that may be automatically generated after a given distance, eg, every 5 meters. Over time, additional PCFs (eg, PCF3 and PCF4, 5) may be created as the user continues to roam the user's environment and the XR device continues to collect more data such as images 1110. The application, i.e., two 1180s, may be launched on the XR device and provide virtual content 1170 to the XR device for presentation to the user. The virtual content may have a content coordinate frame associated with it, which may be installed for one or more PCFs. As shown in FIG. 43B, the second XR device 12.2 creates three PCFs. In some embodiments, the second XR device 12.2 may attempt to locate to one or more canonical maps stored on the server 20.

In some embodiments, the second XR device 12.2 may download the reference map 120 from the server 20, as shown in FIG. 43C. Map 1 on the second XR device 12.2 includes PCFad and origin 1. In some embodiments, the server 20 may have a plurality of reference maps for different locations, with the second XR device 12.2 being the first XR device 12 during the first session. It may be determined that it is in the same neighborhood as the neighborhood of .1 and a criterion map for that neighborhood may be transmitted to the second XR device 12.2.

FIG. 44 shows that the second XR device 12.2 initiates identification of the PCF for the purpose of generating map 2. The second XR device 12.2 identifies only a single PCF, ie PCFs 1 and 2. The X, Y, and Z coordinates of PCF1, 2 for the second XR device 12.2 can be (1,1,1). Map 2 has its own origin (origin 2), which may be based on the device 2's head posture at device startup for the current head posture session. In some embodiments, the second XR device 12.2 may immediately attempt to locate Map 2 with respect to the reference map. In some embodiments, map 2 may not be able to be located relative to the reference map (map 1) because the system does not recognize any or sufficient overlap between the two maps (map 1). That is, location identification can fail). In some embodiments, the system may be located based on a PCF comparison between a local map and a canonical map. In some embodiments, the system may be located based on a sustained attitude comparison between the local map and the canonical map. In some embodiments, the system may locate based on a keyframe comparison between the local map and the canonical map.

FIG. 45 shows Map 2 after the second XR device 12.2 has identified additional PCFs (PCF1, 2, PCF3, PCF4, 5) of Map 2. The second XR device 12.2 again attempts to locate Map 2 with respect to the reference map. The location attempt will be successful because Map 2 has been extended to overlap at least part of the canonical map. In some embodiments, overlap between the local tracking map, map 2, and canonical map may be represented by a PCF, persistent posture, keyframes, or any other suitable intermediate or derivative construct.

Further, the second XR device 12.2 associates content 123 and content 456 with PCF1, 2 and PCF3 of map 2. The content 123 has X, Y, and Z coordinates with respect to PCFs 1 and 2 of (1,0,0). Similarly, the X, Y, and Z coordinates for PCF3 in Map 2 are also (1,0,0).

FIGS. 46A and 46B illustrate the successful positioning of Map 2 with respect to the reference map. The overlapping area / volume / section of map 1410 represents the intersection with map 1 and the reference map. The map 2 created PCFs 3 and 4, 5 before locating, and the reference map created PCFa and c before map 2, so different PCFs were created in real space (eg, different maps). ) Was created to represent the same volume.

As shown in FIG. 47, the second XR device 12.2 extends map 2 to include PCFad from the reference map. The inclusion of PCFa-d represents the location of Map 2 with respect to the reference map. In some embodiments, the XR system may perform an optimization step to remove PCFs within 1410, ie duplicate PCFs such as PCF3 and PCF4, 5 from overlapping areas. After the map 2 is located, the installation of virtual content such as content 456 and content 123 will be for the nearest updated PCF in the updated map 2. The virtual content appears in the same real-world location for the user, despite the modified PCF binding for the content and the updated PCF for Map 2.

As shown in FIG. 48, the second XR device 12.2, for example, as the additional PCF (PCFe, f, g, and h) is identified by the second XR device 12.2 Continue to expand Map 2 as you roam the world. Also note that Map 1 is not extended in FIGS. 47 and 48.

Referring to FIG. 49, the second XR device 12.2 uploads the map 2 to the server 20. The server 20 stores the map 2 together with the reference map. In some embodiments, map 2 may be uploaded to server 20 at the end of the session for the second XR device 12.2.

The reference map in the server 20 includes PCFi here, which is not included in the map 1 on the first XR device 12.1. The canonical map on the server 20 can be extended to include the PCFi when a third XR device (not shown) uploads the map to the server 20 and such a map contains the PCFi.

In FIG. 50, the server 20 merges the map 2 with the canonical map to form a new canonical map. The server 20 determines that the PCFad is common to the reference map and the map 2. The server extends the canonical map to include PCFe-h and PCF1 and 2 from the map 2 to form a new canonical map. The canonical maps on the first and second XR devices 12.1 and 12.2 are based on Map 1 and become obsolete.

In FIG. 51, the server 20 transmits the new reference map to the first and second XR devices 12.1 and 12.2. In some embodiments, this can occur when the first XR device 12.1 and the second device 12.2 attempt to locate during a different or new or subsequent session. The first and second XR devices 12.1 and 12.2 are steps to locate their individual local maps (Map 1 and Map 2, respectively) with respect to the new reference map, as described above. Proceed to.

As shown in FIG. 52, the head coordinate frame 96 or "head posture" is associated with the PCF in map 2. In some embodiments, the origin of the map, or origin 2, is based on the head posture of the second XR device 12.2 at the start of the session. As the PCF is created during the session, the PCF is placed with respect to the world coordinate frame, ie origin 2. The PCF of the map 2 serves as a sustained coordinate frame with respect to the reference coordinate frame, and the world coordinate frame may be the world coordinate frame of the previous session (eg, the origin 1 of map 1 in FIG. 40). The conversion from the world coordinate frame to the head coordinate frame 96 is described above with reference to FIG. The head coordinate frame 96 shown in FIG. 52 has only two orthogonal axes at a particular coordinate position of the map 2 with respect to the PCF and at a particular angle with respect to the map 2. However, it should be understood that the head coordinate frame 96 is in a three-dimensional location with respect to the PCF of Map 2 and has three orthogonal axes in the three-dimensional space.

In FIG. 53, the head coordinate frame 96 is moving with respect to the PCF of the map 2. The head coordinate frame 96 is moving because the second user 14.2 has moved its head. The user can move its head in 6 degrees of freedom (6 dof). The head coordinate frame 96 can therefore move from 6 dof, i.e., its previous location in FIG. 52, in three dimensions and about three orthogonal axes to the PCF of Map 2. The head coordinate frame 96 is adjusted when the real object detection camera 44 and the inertial measurement unit 48 in FIG. 9 detect the motion of the real object and the head unit 22, respectively. Further information on head posture tracking is disclosed and incorporated herein by reference in US Pat. No. 6,221,065, entitled "Enhanced Pose Determination for Display Device."

FIG. 54 shows that sound may be associated with one or more PCFs. The user may wear, for example, headphones or earphones with stereoscopic sound. The location of the sound through the headphones can be simulated using conventional techniques. The sound location is located in a stationary position so that when the user rotates his head to the left, the sound location rotates to the right, and thus the user perceives the sound coming from the same location in the real world. You may. In this embodiment, the location of the sound is represented by sound 123 and sound 456. For the purposes of discussion, FIG. 54 is similar to FIG. 48 in its analysis. When the first and second users 14.1 and 14.2 are located in the same room at the same or different times, they perceive sounds 123 and 456 originating from the same place in the real world.

FIGS. 55 and 56 illustrate further implementations of the techniques described above. The first user 14.1 initiates a first session, as described with reference to FIG. As shown in FIG. 55, the first user 14.1 has terminated the first session, as indicated by the imaginary line. At the end of the first session, the first XR device 12.1 uploaded the map 1 to the server 20. The first user 14.1 in turn started a second session at a time after the first session. The first XR device 12.1 does not download the map 1 from the server 20 because the map 1 is already stored on the first XR device 12.1. If the map 1 is lost, the first XR device 12.1 downloads the map 1 from the server 20. The first XR device 12.1 then proceeds to the step of constructing a PCF for map 2, locating it relative to map 1, and further developing the canonical map, as described above. Map 2 of the first XR device 12.1 is then used to relate local content, head coordinate frames, local sounds, etc., as described above.

With reference to FIGS. 57 and 58, it is also possible that more than one user can interact with the server in the same session. In this embodiment, the first user 14.1 and the second user 14.2 are joined by a third user 14.3 with a third XR device 12.3. Each XR device 12.1, 12.2, and 12.3 begins to generate its own map, namely Map 1, Map 2, and Map 3. Maps are progressively uploaded to server 20 as the XR devices 12.1, 12.2, and 12.3 continue to expand maps 1, 2, and 3. The server 20 merges maps 1, 2, and 3 to form a canonical map. The canonical map is then transmitted from the server 20 to each of the XR devices 12.1, 12.2, and 12.3.

FIG. 59 illustrates aspects of a visual method for restoring and / or resetting head posture, according to some embodiments. In the illustrated embodiment, at act 1400, the visual device is powered on. At act 1410, a new session is initiated in response to being powered on. In some embodiments, the new session may include steps to establish a head posture. One or more capture devices on a head-mounted frame, anchored to the user's head, first capture an image of the environment and then capture the surface of the environment by determining the surface from the image. do. In some embodiments, the surface data may also be combined with data from a gravity sensor to establish a head posture. Other suitable methods for establishing head posture may be used.

At act 1420, the processor of the visual device enters a routine for tracking head posture. The capture device captures the surface of the environment as the user moves its head and continues to determine the orientation of the head-mounted frame with respect to the surface.

At act 1430, the processor determines if the head posture has been lost. Head postures result in or move from "edge" cases, such as too many reflective surfaces, low light intensity, empty walls, outdoors, etc., which can result in low feature availability, forming part of the map. , Crowds, etc., can be lost due to dynamic cases. The routine at 1430 allows a certain amount of time, eg, 10 seconds, to elapse to allow sufficient time to determine if head posture has been lost. If no head posture has been lost, the processor returns to 1420 and enters head posture tracking again.

If the head posture is lost in action 1430, the processor enters a routine to restore the head posture in 1440. If the head posture is lost due to low light intensity, a message such as the following message is displayed to the user through the display of the visual device:

The system is detecting low light conditions. Move to an area with more light.

The system will continue to monitor whether sufficient light is available and whether head posture can be restored. Alternatively, the system may determine that the low texture of the surface is causing head posture loss, in which case the user is given the following prompt in the display as a suggestion to improve surface capture: Be:

The system cannot detect enough surfaces with fine textures. Move to an area where the surface texture is not rough and the texture is more refined.

At act 1450, the processor enters a routine to determine if head posture restoration has failed. If the head posture restoration has not failed (ie, the head posture restoration was successful), the processor returns to action 1420 by entering the head posture tracking again. If the head posture restoration fails, the processor returns to action 1410 and establishes a new session. As part of the new session, all cached data will be invalidated and the head posture will be newly established thereafter. Any suitable method of head tracking may be used in combination with the process described in FIG. U.S. Pat. No. 16,221,065 describes head tracking and is incorporated herein by reference in its entirety.

FIG. 60 shows a schematic representation of a machine in an exemplary embodiment of a computer system 1900, which is a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein. It may be carried out according to the embodiment. In an alternative embodiment, the machine may operate as a stand-alone device or be connected (eg, networked) to another machine. Further, although only a single machine is illustrated, the term "machine" also performs a set of instructions (or a set of instructions) individually or together and is any of the methodologies discussed herein. It shall be considered to include any set of machines that carry out one or more of the above.

An exemplary computer system 1900 includes a processor 1902 (eg, a central processing unit (CPU), a graphics processing unit (GPU), or both) and a main memory 1904 (eg, read-only memory (ROM), flash memory, dynamic random). Access memory (DRAM) includes, for example, synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.) and static memory 1906 (eg, flash memory, static random access memory (SRAM), etc.), which are buses. Communicate with each other via 1908.

The computer system 1900 may further include a disk drive unit 1916 and a network interface device 1920.

The disk drive unit 1916 stores on it a set of one or more instructions 1924 (eg, software) that embodies any one or more of the methodologies or functions described herein. , Includes machine-readable media 1922. The software also resides entirely or at least partially within the main memory 1904 and / or the processor 1902 during its execution by the computer system 1900, main memory 1904, and processor 1902, and also constitutes a machine-readable medium. You may.

The software may further be transmitted or received via the network 18 and via the network interface device 1920.

The computer system 1900 includes a driver chip 1950, which is used to drive a projector and generate light. The driver chip 1950 includes its own data storage device 1960 and its own processor 1962.

Although the machine-readable medium 1922 is shown as a single medium in an exemplary embodiment, the term "machine-readable medium" is a single medium or a plurality of media that stores a set of one or more instructions. For example, it should be considered to include centralized or distributed databases and / or associated caches and servers). The term "machine readable medium" is also capable of storing, encoding, or carrying a set of instructions for execution by a machine and to the machine any one or more of the methodologies of the invention. It shall be regarded as including any medium for carrying out. The term "machine readable medium" is therefore considered to include, but is not limited to, solid state memory, optical and magnetic media, and carrier signals.

Although some aspects of some embodiments have been described so far, it should be appreciated that various modifications, modifications, and improvements will be readily recalled to those of skill in the art.

As an embodiment, embodiments are described in relation to an augmented reality (AR) environment. It should be understood that some or all of the techniques described herein may be applied within an MR environment, or more generally, other XR and VR environments.

As another embodiment, embodiments are described in the context of devices such as wearable devices. Some or all of the techniques described herein may be implemented via any suitable combination of networks (clouds, etc.), discrete applications, and / or devices, networks, and discrete applications. I want you to understand.

In addition, FIG. 29 provides examples of criteria that can be used to filter candidate maps and result in a set of highly ranked maps. Other criteria may be used in lieu of or in addition to the criteria described. For example, if multiple candidate maps have similar values for the metrics used to filter out less desirable maps, the characteristics of the candidate map determine which map is reserved or filtered out as a candidate map. May be used to. For example, a larger or denser candidate map may take precedence over a smaller candidate map.

Such modifications, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Further, although the advantages of the present disclosure are shown, it should be understood that not all embodiments of the present disclosure include all the described advantages. Some embodiments may not implement any of the features described as advantageous herein and in some cases. Therefore, the above description and drawings are merely examples.

Some embodiments are sensors configured to capture information about a three-dimensional (3D) environment and output an image, each image comprising a plurality of pixels, a sensor and a computer run. It relates to a portable electronic system including at least one processor configured to execute possible instructions and process an image output by a sensor. A computer-executable instruction is a step of receiving multiple images captured by a sensor and a step of identifying one or more features within a plurality of pixels for each image of a subset of images with respect to at least a subset of the images. And each feature has a step corresponding to one or more pixels, a step to calculate a feature descriptor for each feature of one or more features, and at least a partial image for each subset of images. It comprises instructions for and with the step of calculating a frame descriptor to represent the image based on the calculated feature descriptor in the image.

In some embodiments, the sensor comprises at least a million pixel circuit. The frame descriptor for each of the plurality of images comprises 512 or a smaller number.

In some embodiments, the computer executable instructions are further generated from steps to build a map of at least part of the 3D environment, feature descriptors for individual frames, and at least partially individual frames. Provides instructions for and for associating parts of the map.

In some embodiments, a computer executable instruction is one or more from a plurality of images, at least in part, as a subset of the plurality of images, based on the location of the image and the multiple pixels of the plurality of images. Provides instructions for the step of selecting a keyframe of.

In some embodiments, the computer executable instruction comprises an instruction for the step of identifying one or more frames associated with a map of the 3D environment for the keyframes of one or more keyframes. One or more frames have a frame descriptor less than the threshold distance from the frame descriptor for the key frame.

In some embodiments, the computer executable instruction for calculating the frame descriptor comprises an artificial neural network.

In some embodiments, the artificial neural network is trained on the basis of similar and different images, receives multiple values representing features in the image as inputs, and outputs a weighted combination of multiple values representing features. It comprises a multi-layer perceptron unit configured to provide as and a maximum pooling unit configured to select a subset of the output of the multi-layer perceptron unit as the frame descriptor.

Some embodiments relate to a method of operating a computing system to generate a map of at least a portion of a three-dimensional (3D) environment based on sensor data collected by a device worn by a user. The method uses a step of receiving a plurality of images captured by a device worn by the user, a step of determining one or more keyframes from the plurality of images, and a first artificial neural network. A step of identifying one or more points of interest in one or more key frames, a step of calculating a feature descriptor for each point of interest using a first artificial neural network, and one or more keys. For each frame, a second artificial neural network is used to represent the keyframe, at least in part, based on the calculated feature descriptor for the identified point of interest within the keyframe. Includes a step to calculate.

In some embodiments, the first and second artificial neural networks are subnets of the artificial neural network.

In some embodiments, the frame descriptor is unique to each keyframe.

In some embodiments, each one or more keyframes has a resolution higher than 1 megabyte. The frame descriptor for each one or more keyframes is a column with less than 512 numbers.

In some embodiments, each feature descriptor is a 32-byte column.

In some embodiments, the frame descriptor is generated by maximally pooling the feature descriptor.

In some embodiments, the method is one or more in a database comprising a step of receiving a new image captured by a device worn by the user and a keyframe used to generate a map. A step of identifying the nearest keyframe, wherein one or more nearest keyframes include a step having a frame descriptor within a predetermined distance of the frame descriptor for the new image.

In some embodiments, the method is based on a step of performing feature matching against a 3D map point of a map corresponding to one or more identified keyframes and a feature matching result. Includes a step of calculating the posture of the device worn by the user.

In some embodiments, the step of determining one or more keyframes from a plurality of images is a step of comparing the pixels of the first image with the pixels of the second image taken immediately after the first image. , Includes a step of identifying the second image as a key frame when the difference between the pixels of the first image and the pixels of the second image is above or below the threshold.

In some embodiments, the method is a step of training a second artificial neural network by generating a dataset with multiple image sets, where the plurality of image sets are query images, positive, respectively. The loss is calculated by comparing the query image with the positive and negative sample images for each step and image set of multiple image sets in the dataset, including sample images and negative sample images. The distance between the step and the frame descriptor generated by the second artificial neural network for the query image and the frame descriptor for the positive sample image is negative with the frame descriptor for the query image. Includes a step of modifying the second artificial neural network based on the calculated loss to exceed the distance to the frame descriptor for the sample image of.

Some embodiments relate to computing environments for cross-reality systems. The computing environment includes a database that stores multiple maps. Each map contains information that represents an area of the 3D environment. The information representing each region comprises a frame descriptor representing the image of the region, and the non-transient computer storage medium, when executed by at least one processor, exhibits multiple features within the image within the computing environment. By identifying, the image captured by the portable device is processed, a feature descriptor is calculated for each of the features, and at least in part, a calculation for one or more identified points of interest in the image. Based on the feature descriptors made, the frame descriptor to represent the image is calculated and in the database based on the comparison between the calculated frame descriptor and the frame descriptor stored in the map's database. Memorize computer-executable instructions to select a map of.

In some embodiments, the frame descriptor is unique to the frame stored in the database.

In some embodiments, the image has a resolution higher than 1 megabyte. The frame descriptor calculated to represent the image is a column, which is less than 512 numbers.

In some embodiments, a computer-executable instruction is a step of processing a dataset that comprises a plurality of image sets, where the plurality of image sets are a query image, a positive sample image, and a negative sample image, respectively. For the query image, including the steps to calculate the loss for a set of images in multiple image sets in the dataset, and the step to compare the query image with the positive and negative sample images. The distance between the frame descriptor generated by the artificial neural network and the frame descriptor for the positive sample image is between the frame descriptor for the query image and the frame descriptor for the negative sample image. It comprises an artificial neural network trained by the steps of modifying the artificial neural network based on the calculated loss so that it is less than the distance of.

In some embodiments, modifying the artificial neural network comprises modifying a copy of the artificial neural network on a portable device in a computing environment.

In some embodiments, the computing environment comprises a cloud platform and a plurality of portable devices that communicate with the cloud platform. The cloud platform comprises a database and computer executable instructions for selecting maps. Computer-executable instructions for processing images captured by the portable device are stored on the portable device.

Some embodiments include a first processor, a first computer-readable medium connected to the first processor, and a first origin coordinate frame stored on the first computer-readable medium. A first destination coordinate frame stored on a computer-readable medium, a first data channel that receives data representing local content, and a position of local content that can be executed by a first processor. After converting the first coordinate frame converter that converts the first origin coordinate frame to the first destination coordinate frame and the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, With respect to an XR system, including a first XR device, including a first display system, adapted to display local content to a first user.

In some embodiments, the step of storing the first origin coordinate frame, the step of storing the first destination coordinate frame, the step of receiving the data representing the local content, and the positioning of the local content are first. After converting the origin coordinate frame of the to the first destination coordinate frame and the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, the local content is displayed to the first user. It relates to a visual recognition method including a step to be performed.

Some embodiments are map storage routines that store a first map, which is a canonical map with multiple persistent coordinate frames (PCFs), where each PCF in the first map is a set of coordinates. A map storage routine that is connected to a real object detection device and a real object detection device that is positioned to detect the location of the real object, and detects the PCF of the second map based on the location of the real object. PCF identification system, each PCF of the second map is connected to a PCF identification system having a set of coordinates, a reference map and a second map, and the first PCF of the second map. Execute to locate the second map with respect to the reference map by matching the first PCF of the reference map and matching the second PCF of the second map with the second PCF of the reference map. With respect to the XR system, including possible, locating modules.

In some embodiments, the real object detection device is a real object detection camera.

In some embodiments, the XR system is further connected to a canonical map and a second map, which can be implemented to incorporate a third PCF of the canonical map into the second map. Be prepared.

In some embodiments, the XR system is a head unit that further comprises a head mountable frame, wherein the real object detection device is mounted on the head mountable frame, the head unit, and local content. Connected to and local to a local content positioning system and a local content positioning system that receives image data, is connected to the data channel, and is capable of executing local content to relate to one PCF of the canonical map. It is equipped with a display system that displays content.

In some embodiments, the XR system further comprises a local / world coordinate converter that transforms the local coordinate frame of the local content into the world coordinate frame of the second map.

In some embodiments, the XR system further stores a first world frame determination routine, which calculates a first world coordinate frame, and a first world coordinate frame, based on the PCF of the second map. World frame storage command, head frame determination routine that calculates the head coordinate frame, and stores the first head coordinate frame, which changes according to the movement of the head mountable frame, head frame storage It includes a command and a world / head coordinate converter that converts the world coordinate frame into a head coordinate frame.

In some embodiments, the head coordinate frame changes relative to the world coordinate frame as the head mountable frame moves.

In some embodiments, the XR system further comprises at least one sound element associated with at least one PCF in the second map.

In some embodiments, the first and second maps are created by an XR device.

In some embodiments, the XR system further comprises first and second XR devices. Each XR device is a head unit having a head mountable frame, and the real object detection device is mounted on the head mountable frame, and receives image data of the head unit and local contents. A display system that is connected to a channel and a data channel and is connected to a local content positioning system and can display local content so that the local content can be associated with one PCF in the canonical map. And include.

In some embodiments, the first XR device creates a PCF for the first map, the second XR device creates a PCF for the second map, and the location module. It forms part of a second XR device.

In some embodiments, the first and second maps are created in the first and second sessions, respectively.

In some embodiments, the XR system further comprises a server and a map download system that forms part of the XR device and downloads a first map from the server via a network.

In some embodiments, the locating module repeatedly attempts to locate the second map relative to the reference map.

In some embodiments, the XR system further comprises a map issuer that uploads a second map to the server over the network.

In some embodiments, there is a step of storing a first map, which is a canonical map with multiple PCFs, where each PCF of the canonical map has a set of coordinates, a step and the location of the real object. The step to detect and the step to detect the PCF of the second map based on the location of the real object, where each PCF of the second map has a set of coordinates, the step and the second map. By matching the first PCF to the first PCF of the first map and the second PCF of the second map to the second PCF of the reference map, the second map is matched to the reference map. The present invention relates to a visual recognition method including a step of identifying a position.

Some embodiments include a processor, a computer-readable medium connected to the processor, a plurality of reference maps on the computer-readable medium, and a separate reference map identifier on the computer-readable medium associated with each individual reference map. The location, which is different from each other and uniquely identifies the reference map, and the location, which is on a computer-readable medium and is executable by the processor, receives and stores the location identifier from the XR device. The detector is on a computer-readable medium, runn by the processor, and compares the location identifier with the criterion map identifier to determine one or more criteria maps that form the first filtered selection. , A first filter and one or more of the canonical maps transmitted to the XR device on a computer-readable medium, which is executable by the processor and is based on the first filtered selection. With respect to an XR system, including a server, which may have a map transmitter.

In some embodiments, the canonical map identifier includes longitude and latitude, respectively, and the location identifier includes longitude and latitude, respectively.

In some embodiments, the first filter comprises at least one matching criterion map covering the longitude and latitude, including the longitude and latitude of the location identifier, and the longitude and adjacent to the first matching criterion map. A neighborhood area filter that selects at least one neighborhood map that covers the latitude.

In some embodiments, the location identifier comprises a WiFi finger print. The XR system is also a WiFi finger print filter, which is on a computer readable medium and the processor determines the latitude and longitude based on the WiFi finger print, and the latitude and longitude and reference map from the WiFi finger print filter. Latitude and longitude are compared to determine one or more criterion maps that form the second filtered selection within the first filtered selection, and the map transmitter is based on the second selection. It comprises a second filter capable of transmitting one or more criterion maps and not transmitting a criterion map based on the first selection outside the second selection.

In some embodiments, the first filter is on a computer-readable medium, the processor determines the latitude and longitude based on the WiFi finger print, and the latitude and longitude from the WiFi finger print filter and the canonical map. A WiFi finger print filter that can be run to compare the latitude and longitude of and determine one or more canonical maps that form a first filtered selection.

In some embodiments, the XR system is further on a computer-readable medium, executable by a processor, receiving multiple features of an image and converting each feature into a separate number of columns, multi-layered. A perceptual unit and a maximum pool unit on a computer-readable medium that can be run by a processor and combines the maximum value of each number of columns into a global feature column that represents an image, with each criterion map being global. The position identifier having at least one of the feature sequences and received from the XR device is captured by the XR device, which is advanced to determine the global feature sequence of the image by the multi-layer perception unit and the maximum pool unit. One that compares the largest pool unit, including the features of the image, with the global feature column of the image and the global feature column of the canonical map to form a third filtered selection within the second filtered selection. A keyframe filter that determines the above criterion map, the map transmitter transmits one or more criterion maps based on a third selection and is based on a second selection outside the third selection. It is equipped with a keyframe filter that does not transmit the reference map.

In some embodiments, the XR system is on a computer-readable medium, runn by a processor, receiving multiple features of an image and converting each feature into a separate number of columns, multi-layered perception. A unit and a maximum pool unit on a computer-readable medium that is runnable by the processor and combines the maximum value of each number of columns into a global feature column that represents an image, and each criterion map is a global feature. The location identifier having at least one of the columns and received from the XR device was captured by the XR device, which is advanced to determine the global feature sequence of the image by the multi-layer perception unit and the maximum pool unit. Further equipped with a maximum pool unit, including image features, the first filter is a keyframe filter that compares the global feature column of the image with the global feature column of the canonical map to determine one or more canonical maps. be.

In some embodiments, the XR system is a head unit that comprises a head mountable frame, wherein the real object detection device is mounted on the head mountable frame, the head unit, and an image of local content. A local content positioning system that receives data, is connected to the data channel, and is capable of performing local content to relate to one PCF in the canonical map, and is connected to the local content positioning system and local content. The XR device includes a display system and a display system.

In some embodiments, the XR device is a map storage routine that stores a first map, which is a canonical map with multiple PCFs, where each PCF in the first map has a set of coordinates. , A map storage routine, a real object detection device positioned to detect the location of a real object, and a PCF of a second map that is connected to the real object detection device and is based on the location of the real object. A PCF identification system, each PCF of the second map is connected to a PCF identification system having a set of coordinates and a reference and a second map, and the first PCF of the second map is the reference map. By matching to the first PCF and matching the second PCF of the second map to the second PCF of the canonical map, it is possible to locate the second map with respect to the canonical map. , Includes locating modules.

In some embodiments, the real object detection device is a real object detection camera.

In some embodiments, the XR system comprises a canonical map embedding device that is connected to a canonical map and a second map and can be implemented to incorporate a third PCF of the canonical map into the second map. ..

Some embodiments are steps of storing a plurality of criterion maps on a computer-readable medium, where each criterion map has a separate canon map identifier associated with a separate canon map, which is a canonical map identifier. A step that uniquely identifies the criteria map, which is different from each other, and a step that receives and stores the location identifier from the XR device using a processor connected to a computer-readable medium, and a location identifier and criterion using the processor. Multiple canonical maps are created based on the first filtered selection using a processor and the steps of comparing map identifiers to determine one or more canonical maps that form a first filtered selection. The present invention relates to a visual recognition method including a step of transmitting to an XR device.

Some embodiments are on a processor, a computer-readable medium connected to the processor, and a computer-readable medium that can be executed by the processor and receive multiple features of the image, each feature individually. With a multi-layered perception unit that transforms into a number of columns, and a maximum pool unit that is on a computer-readable medium and is processor-executable and combines the maximum value of each number of columns into a global feature column that represents an image. With respect to the XR system, including.

In some embodiments, the XR system is a plurality of criteria maps on a computer-readable medium, with each criterion map having at least one of the global feature columns associated with it. Features of an image captured by an XR device, on a computer-readable medium, runn by a processor and processed by a multi-layer perception unit and a maximum pool unit to determine the global feature sequence of the image. A keyframe that compares the position detector received from the XR device with the global feature column of the image and the global feature column of the canonical map to determine one or more canonical maps that form part of the filtered selection. It comprises a filter and a map transmitter that is on a computer readable medium and is executable by the processor and transmits one or more of the canonical maps to the XR device based on the filtered selection. ..

In some embodiments, the XR system is a head unit that comprises a head mountable frame, wherein the real object detection device is mounted on the head mountable frame, the head unit, and an image of local content. A local content positioning system that receives data, is connected to the data channel, and is capable of performing local content to relate to one PCF of the canonical map, and is connected to the local content positioning system and local content. It is equipped with a display system to display.

In some embodiments, the XR system is a head unit that comprises a head mountable frame, wherein the real object detection device is mounted on the head mountable frame, the head unit, and an image of local content. A local content positioning system and a local content positioning system that receive data, are connected to the data channel, and are capable of executing the local content to relate to one PCF of the canonical map. A display system that displays an XR device, including a display system, wherein the matching step is performed by matching the global feature column of a second map to the global feature column of a canonical map. Be prepared.

In some embodiments, a processor is used to receive multiple features of an image, a processor is used to convert each feature into a separate number of columns, and a processor is used to convert each feature. It relates to a visual method including a step of combining the maximum value of a column into a global feature column representing an image.

Some embodiments operate a computing system, identify one or more environment maps stored in a database, and a tracking map calculated based on sensor data collected by a device worn by the user. In a method of merging with, the device receives the signal of the access point to the computer network while calculating the tracking map, and based on the characteristics of communication with the access point, at least one area attribute of the tracking map. A step to determine, a step to determine the geographic location of the tracking map based on at least one area attribute, and a step to identify the set of environment maps stored in the database corresponding to the determined geographic location. , A step to filter the set of environment maps based on the similarity of one or more identifiers of the network access points associated with the environment map of the set of tracking maps and environment maps, and the environment map of the set of tracking maps and environment maps. An environment map based on the degree of matching between the steps of filtering a set of environment maps based on the similarity of the metrics that represent the content of the map and part of the tracking map and part of the environment map of the set of environment maps. With respect to methods, including steps to filter a set of.

In some embodiments, the step of filtering a set of environment maps based on the similarity of one or more identifiers of a network access point is to one or more identifiers of the network access point within the set of environment maps. Based on, it comprises the step of reserving an environment map with the highest Jaccard similarity to at least one area attribute of the tracking map.

In some embodiments, the step of filtering a set of environment maps based on the similarity of the metrics that represent the contents of the environment map of the tracking map and the set of environment maps is a characteristic of the tracking map within the set of environment maps. Includes the step of reserving an environment map with a minimum vector distance between the vector of and the vector representing the environment map in the set of environment maps.

In some embodiments, the metrics representing the content of the tracking map and the environment map include a vector of values calculated from the content of the map.

In some embodiments, the step of filtering the set of environment maps based on the degree of matching between part of the tracking map and part of the environment map of the set of environment maps is the environment map of the set of environment maps. An environment map with a step to calculate the volume of the physical world represented by the tracking map, also represented within, and a calculated volume within the set of environment maps that is larger than the environment map filtered out of the set. Includes steps to reserve.

In some embodiments, the set of environment maps is first based on the similarity of one or more identifiers, then based on the similarity of the metrics that represent the content, and then part of the tracking map. Filtered based on the degree of matching between and part of the environment map.

In some embodiments, the environment map is based on the similarity of one or more identifiers, the similarity of the metrics that represent the content, and the degree of matching between part of the tracking map and part of the environment map. The filtering of the set is performed in an order based on the processing required to perform the filtering.

In some embodiments, the environment map is based on the similarity of one or more identifiers, the similarity of the metrics that represent the content, and the degree of matching between part of the tracking map and part of the environment map. , Selected based on the filtering of the set of environment maps, the information is loaded onto the user device from the selected environment map.

In some embodiments, the environment map has the similarity of one or more identifiers, the similarity of the metrics that represent the content, and the degree of matching between part of the tracking map and part of the environment map. Based on the filtering of the set of environment maps, the tracking map is merged with the selected environment map.

Some embodiments are cloud computing environments for augmented reality systems configured for communication with a plurality of user devices comprising sensors, indicating an area in which the plurality of user devices are used. A user database that stores area identification, where area identification consists of a user database with parameters of the wireless network detected by the user device at the time of use, and data supplied by multiple user devices and associated metadata. A map database that stores multiple constructed environment maps, with associated metadata having area identification derived from the area identification of multiple user devices that supplied the data from which the map was built. Area identification is performed by a map database and at least one processor in a cloud computing environment, with the parameters of the wireless network detected by the user device that supplied the data from which the map was built. The message is received from multiple user devices, including the parameters of the wireless network detected by the user device, the area identifier for the user device is calculated, and based on the received parameters and / or the calculated area identifier. , Update the user database, receive requests for the environment map from multiple user devices, determine the area identifier associated with the user device requesting the environment map, and at least partially based on the area identifier, the environment. The steps of identifying a set of maps from the map database, filtering the set of environment maps, transmitting the filtered set of environment maps to the user device, and filtering the set of environment maps from there are requests for the environment map. Stores computer-executable instructions based on the similarity between the wireless network parameters detected by the user device and the wireless network parameters in the map database for the environmental map in the set of environmental maps. Concerning a cloud computing environment with a non-transient computer storage medium.

In some embodiments, the computer executable instruction is further configured to receive a tracking map from the user device requesting the environment map when executed by at least one processor in the cloud computing environment. The step of filtering a set of is further based on the similarity of the metrics that represent the contents of the tracking map and the environment map of the environment map set.

In some embodiments, the computer executable instruction is further configured to receive a tracking map from the user device requesting the environment map when executed by at least one processor in the cloud computing environment. The step of filtering a set of is further based on the degree of matching between a portion of the tracking map and a portion of the environment map of the environment map set.

In some embodiments, the wireless network parameter comprises a network basic service set identifier (BSSID) to which the user device is connected.

In some embodiments, the step of filtering a set of environment maps based on the similarity of wireless network parameters is with multiple BSSIDs stored in the user database associated with the user device requesting the environment map. It involves calculating the similarity to the BSSID stored in the map database associated with the environment map of the set of environment maps.

In some embodiments, the area identifier indicates a geographical location by longitude and latitude.

In some embodiments, the step of determining the area identifier comprises accessing the area identifier from the user database.

In some embodiments, the step of determining the area identifier comprises receiving the area identifier in a message received from a plurality of user devices.

In some embodiments, the radio network parameters are protocol compliant, including Wi-Fi and 5G NR.

In some embodiments, the computer executable instruction is further configured to receive a tracking map from the user device when executed by at least one processor in the cloud computing environment, filtering the set of environment maps. The steps are further based on the degree of matching between part of the tracking map and part of the environment map in the set of environment maps.

In some embodiments, the computer executable instruction is further based on the user device that receives the tracking map from the user device and supplies the tracking map when executed by at least one processor in the cloud computing environment. Determine the area identifier associated with the tracking map, select a second set of environment maps from the map database, at least in part, based on the area identifier associated with the tracking map, and based on the tracking map received. , The step of updating and updating the map database is configured to include the step of merging one or more environment maps in a set of received tracking maps and a second environment map.

In some embodiments, the computer executable instructions are further executed by at least one processor in the cloud computing environment to receive a portion of the tracking map and a second set of environment maps for the environment map. The step is configured to filter the second set of environment maps based on the degree of matching with some, and the step of merging the tracking map and one or more environment maps in the second set of environment maps is , Includes a step of merging one or more environment maps in a set of tracking maps and a filtered second environment map.

Some embodiments are PCF identification systems and maps that are connected to a real object detection device that detects multiple surfaces of a real world object and generate a map based on the real world object. Based on, a persistent coordinate frame (PCF) generation system that generates a first PCF and associates a map with the first PCF, and first and second XR devices on the first and second XR devices, respectively. At least the first and first of the first and second XR devices, which store the second storage medium and the first PCF in the first and second storage media of the first and second XR devices, respectively. The present invention relates to an XR system including two processors.

In some embodiments, the XR system can be run by at least one processor, a keyframe generator that converts multiple camera images into multiple individual keyframes, and at least one processor. And it can be run by a persistent posture calculator and at least one processor to generate a sustained pose by averaging multiple keyframes, transforming the tracking map into a sustained pose and tracking map. A tracking map and a sustained attitude converter that determines the sustained attitude at the origin with respect to, and a first PCF that can be executed by at least one processor and that converts the sustained attitude to a first PCF and a sustained attitude. A PCF and an image data converter that can be run by at least one processor and that converts the first PCF to image data, and an image to the first PCF that determines the PCF. It is equipped with a display device that displays data to the user.

In some embodiments, the detection device is a detection device for the first XR device that is connected to the first XR device processor.

In some embodiments, the map is a first map on a first XR device, and the processor that produces the first map is the first XR device processor of the first XR device.

In some embodiments, the processor that produces the first PCF is the first XR device processor of the first XR device.

In some embodiments, the processor that associates the first PCF with the first map is the first XR device processor of the first XR device.

In some embodiments, the XR system is run by the application, which is run by the first XR device processor, and the first PCF tracker, which is run by the first XR device processor, from the application. It comprises a first PCF tracker that includes an on-prompt for switching and that produces a first PCF only if the first PCF tracker is switched on.

In some embodiments, the first PCF tracker has an off-prompt for switching the first PCF tracker off from the application, and the first PCF tracker is the first PCF tracker. When switched off, the first PCF generation ends.

In some embodiments, the XR system is runnable by a first XR device processor, a map issuer that transmits the first PCF to the server, and a server processor of the server. A map storage routine that stores the first PCF on the storage device of the server, a step of transmitting the first PCF to the second XR device using the server processor of the server, and a second of the second XR devices. It comprises a map download system that can be run by two XR device processors and downloads a first PCF from a server.

In some embodiments, the XR system is runn by an application, runn by a second XR device processor, and a second PCF tracker turned on from the application, runn by a second XR device processor. It comprises a second PCF tracker that includes an on-prompt for switching, with a second PCF tracker that produces a second PCF only if the second PCF tracker is switched on.

In some embodiments, the second PCF tracker has an off-prompt for switching the second PCF tracker off from the application, and the second PCF tracker is a second PCF tracker. When switched off, the second PCF generation ends.

In some embodiments, the XR system comprises a map issuer that is executable by a second XR device processor and transmits a second PCF to the server.

In some embodiments, the XR system is runnable by a first XR device processor, a persistent posture acquirer that downloads the persistent posture from the server, and a first XR device processor. There is a PCF confirmer that reads the PCF from the first storage device of the first XR device based on a sustained attitude, and can be executed by the first XR device processor from the first storage device. A coordinate frame processor that calculates a coordinate frame based on the read PCF is provided.

Some embodiments include a step of detecting multiple surfaces of a real-world object using at least one detection device and a step of generating a map based on the real-world object using at least one processor. , A step of generating a first PCF based on a map using at least one processor, a step of associating a map with a first PCF using at least one processor, and a first and second XR. A visual method comprising storing the first PCF in the first and second storage media of the first and second XR devices, respectively, using at least the first and second processors of the device. Regarding.

In some embodiments, the visual method uses at least one processor to convert multiple camera images into multiple individual keyframes, and at least one processor to average multiple keyframes. By doing so, with at least one processor, there is a step to generate a persistent posture, and with at least one processor, there is a step to convert the tracking map into a persistent posture and determine the persistent posture at the origin with respect to the tracking map. , A step of converting a persistent posture to a first PCF and determining a first PCF for a sustained posture, a step of converting the first PCF to image data using at least one processor, and a display device. Includes a step of displaying image data for the first PCF to the user using.

In some embodiments, the detection device is a detection device for the first XR device that is connected to the first XR device processor.

In some embodiments, the map is a first map on a first XR device, and the processor that produces the first map is the first XR device processor of the first XR device.

In some embodiments, the processor that produces the first PCF is the first XR device processor of the first XR device.

In some embodiments, the processor that associates the first PCF with the first map is the first XR device processor of the first XR device.

In some embodiments, the visual method is a step of running the application using the first XR device processor and a first PCF tracking on-prompt from the application using the first XR device processor. A step of switching on a device, wherein the first PCF tracker includes a step of generating a first PCF only when the first PCF tracker is turned on.

In some embodiments, the visual method is a step in which the first PCF tracker can be switched off at an off-prompt from an application using the first XR device processor, the first PCF tracker. Includes the step of ending the first PCF generation when the first PCF tracker is switched off.

In some embodiments, the visual method uses a first XR device processor to transmit the first PCF to the server and a server processor to use the server's server processor to transfer the first PCF to the server's storage device. The steps stored above, the step of transmitting the first PCF to the second XR device using the server processor of the server, and the first step using the second XR device processor of the second XR device. Includes a step of receiving the PCF from the server.

In some embodiments, the visual method is a step of running the application using a second XR device processor and a second PCF tracking on-prompt from the application using the second XR device processor. A step of switching the device on, wherein the second PCF tracker includes a step of generating a second PCF only if the second PCF tracker is turned on.

In some embodiments, the visual method is a step of switching off the second PCF tracker at an off-prompt from the application using the first XR device processor, where the second PCF tracker is , Includes a step to end the second PCF generation when the second PCF tracker is switched off.

In some embodiments, the visual method comprises uploading a second PCF to a server using a second XR device processor.

In some embodiments, the visual method uses a first XR device processor to determine a persistent posture from the server and a first XR device processor to use a PCF based on the persistent posture. From the first storage device of the first XR device, and a step of calculating a coordinate frame based on the PCF read from the first storage device using the first XR device processor. include.

Some embodiments are feasible and sustainable by the first XR device processor, the first XR device storage device connected to the first XR device processor, and the first XR device processor. A PCF reader that reads the PCF from the first storage device of the first XR device, which can be executed by a download system that downloads the posture from the server and a first XR device processor, based on a sustained posture. On a first XR device processor, including a coordinate frame processor that is executable by the first XR device processor and calculates coordinate frames based on the PCF read from the first storage device. With respect to an XR system comprising a first XR device, which may include a set of instructions.

In some embodiments, the first XR device processor of the first XR device is used to download the persistent posture from the server, and the first XR device processor is used based on the persistent posture. , A step of reading the PCF from the first storage device of the first XR device, and a step of calculating a coordinate frame based on the PCF read from the first storage device using the first XR device processor. Concerning visual recognition methods, including.

Some embodiments are feasible with a server processor, a server storage device connected to the server processor, and a server server processor, and a first PCF associated with a map is stored in the server server. Includes a map storage routine that stores on the device and a map transmitter that can be executed using the server processor and transmits the map and the first PCF to the first XR device. Get about XR devices, including servers.

In some embodiments, the server processor of the server is used to store the first PCF associated with the map on the server storage device of the server, and the server processor of the server is used to store the map and the first PCF. The present invention relates to a visual recognition method including a step of transmitting a PCF to a first XR device.

In some embodiments, the processor of the XR device uses a capture device on a head-mounted frame that is anchored to the user's head to capture the environment and determine the orientation of the head-mounted frame. The step of entering the head posture tracking and the step of determining whether the head posture was lost due to the inability to orient the head-mounted frame by the processor, and the head posture was lost. If so, the present invention relates to a visual method including a step of establishing a head posture by entering a posture restoration mode by a processor and determining the orientation of the head-mounted frame.

In some embodiments, if the head posture is not lost, the processor enters the tracking of the head posture.

In some embodiments, the posture restoration involves displaying a message to the user, along with suggestions for improving the capture of the environment by the processor.

In some embodiments, the proposal is at least one of a step of increasing light and a step of refining the texture.

In some embodiments, the visual method initiates a new session, comprising the step of determining if the restore failed by the processor and, if the restore fails, the step of establishing the head posture by the processor. Including steps to do.

In some embodiments, the visual method comprises displaying to the user a message that the processor will initiate a new session.

In some embodiments, the visual method comprises stepping into tracking the head posture by a processor if the head posture is not lost.

Some embodiments relate to a method of operating a computing system to render a virtual object in a scene comprising one or more physical objects. The method captures multiple images of a scene from one or more sensors of a first device worn by the user, and at least in part, one or more sustained postures based on the images. And the information of multiple images can be accessed at different times by one or more applications launched on the first device and / or the second device via a persistent coordinate frame. Includes, at least in part, a step of generating a persistent coordinate frame based on one or more calculated persistent poses.

In some embodiments, at least in part, the step of calculating one or more persistent postures based on a plurality of images is a step of extracting one or more features from each of the plurality of images and one. A step to generate a descriptor for each of one or more features, and at least partly to generate a keyframe for each of multiple images based on the descriptor, and at least partially one or more keys. Includes steps to generate one or more sustained postures based on the frame.

In some embodiments, at least in part, the step of generating a persistent coordinate frame based on one or more calculated persistent poses is that the first device is from the location of the previous persistent coordinate frame. Includes a step to generate a persistent coordinate frame as it travels a given distance.

In some embodiments, the predetermined distance is 2 to 20 meters and is based on both the consumption of computational resources of the device and the installation error of the virtual object.

In some embodiments, the method comprises the step of generating an initial sustained posture when the first device is powered on, and the initial sustained posture and threshold distance where the first device is the center of a circle. Includes the step of generating a first sustained posture at the current location of the first device upon reaching a circular circumference with a radius equal to.

In some embodiments, the circle is the first circle. The method further provides that when the device reaches the circumference of the second circle with a first sustained posture as the center of the circle and a radius equal to twice the threshold distance, at the current location of the first device. It involves the step of generating a second sustained posture.

In some embodiments, when the first device finds an existing sustained posture within a threshold distance from the initial sustained posture, the first sustained posture is not generated.

In some embodiments, the first device associates the first sustained posture with one or more of a plurality of keyframes within a predetermined distance to the first sustained posture. ..

In some embodiments, the first persistent attitude is not generated unless the application launched on the first device requires a sustained attitude.

Some embodiments relate to electronic systems that can be carried by the user. The electronic system is configured to capture images of one or more physical objects in the scene, execute computer-executable instructions with one or more sensors, and render virtual content in the scene. A computer-executable instruction that is configured to execute an application and a computer-executable instruction and is configured to provide image data about virtual content to the application, wherein the computer-executable instruction is at least partially. Includes at least one processor with instructions for generating sustained coordinate frames based on the captured image.

The aforementioned embodiments of the present disclosure can be implemented in any of a number of methods. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code runs on any suitable processor or set of processors, whether provided within a single computer or distributed among multiple computers. be able to. Such processors are one or more of integrated circuit components, including commercially available integrated circuit components known in the art, such as CPU chips, GPU chips, microprocessors, microprocessors, or coprocessors, to name a few. It may be implemented as an integrated circuit together with the processor of. In some embodiments, the processor may be implemented in a custom circuit such as an ASIC, or in a semi-custom circuit resulting from configuring a programmable logic device. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific embodiment, some commercially available microprocessors have multiple cores such that one or a subset of those cores can constitute the processor. However, the processor may be implemented using circuits in any suitable format.

Further, it should be understood that a computer can be embodied in any of several forms such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computer may be embodied in a device, including a personal digital assistant (PDA), a smartphone, or any suitable portable or fixed electronic device, which is generally not considered a computer but has suitable processing power. good.

The computer may also have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for the visual presentation of the output, or a speaker or other sound generating device for the audible presentation of the output. Examples of input devices that can be used for user interfaces include keyboards and pointing devices such as mice, touchpads, and digitized tablets. As another embodiment, the computer may receive input information through speech recognition or in other audible formats. In the illustrated embodiment, the input / output device is shown as physically separate from the computing device. However, in some embodiments, the input and / or output device may be physically integrated into the same unit as the processor or other element of the computing device. For example, the keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output device may be completely disconnected from the computing device and functionally integrated through a wireless connection.

Such computers may be interconnected by one or more networks of any suitable form, including as a local area network or wide area network, such as a corporate network or the Internet. Such networks may be based on any suitable technique, may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

Also, the various methods and processes outlined herein are encoded as software that can be run on one or more processors that employ any one of the various operating systems or platforms. You may. In addition, such software may be written using any of several suitable programming languages and / or programming or scripting tools and may also be run on a framework or virtual machine. It may be compiled as executable machine language code or intermediate code.

In this aspect, the disclosure is in one or more programs that, when run on one or more computers or other processors, implement methods of implementing the various embodiments of the present disclosure discussed above. Encoded computer-readable storage medium (or multiple computer-readable media) (eg, computer memory, one or more floppy® discs, compact discs (CDs), optical discs, digital video discs (DVDs), It may be embodied as a magnetic tape, flash memory, circuit configuration within a field programmable gate array or other semiconductor device, or other tangible computer storage medium). As will be apparent from the above embodiments, the computer-readable storage medium may reserve information for a sufficient amount of time to provide computer-executable instructions in a non-transient form. Such computer-readable storage media or media, as described above, is one or more such that one or more programs stored on it implement various aspects of the present disclosure. It can be transportable so that it can be loaded on different computers or other processors. As used herein, the term "computer-readable storage medium" includes only computer-readable media that can be considered manufactured (ie, manufactured) or machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

The term "program" or "software", as described above, may be any type of computer code or that may be employed to program a computer or other processor to implement various aspects of the present disclosure. As used herein in a general sense, to refer to a set of computer-executable instructions. In addition, according to one aspect of this embodiment, one or more computer programs that, when executed, perform the methods of the present disclosure do not need to reside on a single computer or processor, but of the present disclosure. It should be understood that it can be distributed in a modular fashion among several different computers or processors to implement various aspects.

Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. In general, a program module includes routines, programs, objects, components, data structures, etc. that perform a particular task or implement a particular abstract data type. Typically, the functionality of the program modules may be combined or distributed, as desired, in various embodiments.

Further, the data structure may be stored in a computer-readable medium in any suitable form. For simplicity of illustration, a data structure may be shown to have fields associated through locations within the data structure. Such relationships may also be achieved by allocating storage for fields with locations in computer-readable media that convey the relationships between the fields. However, any suitable mechanism, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements, is used to establish relationships between information within fields of a data structure. May be done.

The various aspects of the present disclosure may be used alone, in combination, or in various sequences not specifically discussed in the aforementioned embodiments, and therefore their uses are described in the above description or described. Not limited to the details and arrangement of components illustrated in the drawings. For example, the aspects described in one embodiment may be combined with the aspects described in another embodiment in any manner.

Further, the present disclosure may be embodied as a method for which an embodiment thereof is provided. The actions performed as part of the method may be ordered in any suitable method. Thus, although shown as a continuous action in an exemplary embodiment, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing several actions simultaneously.

The use of ordering terms such as "first," "second," "third," etc. in a claim to modify a claim element alone is one claim compared to another element. It does not imply any priority, precedence, or order of the elements, or the time order in which the act of the method takes place, but the term of order is used to distinguish the claim elements (due to the use of the term of order). ) Used only as a marker to distinguish one claim element with a certain name from another with the same name.

Also, the terms and terminology used herein are for explanatory purposes only and should not be considered limiting. The use of "including", "with", or "having", "containing", "with", and variants thereof herein are listed as items and their use thereof. Means to include equals and additional items.

Claims (28)

It ’s an electronic system,
One or more sensors configured to capture information about a three-dimensional (3D) environment, wherein the captured information comprises a sensor and a plurality of images.
At least one processor configured to execute a computer executable instruction and generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer executable instruction further comprises.
Identifying multiple features within the plurality of images and
To select a plurality of keyframes from the plurality of images, at least in part, based on the plurality of characteristics of the selected keyframes.
Generating one or more coordinate frames, at least in part, based on the identified characteristics of the selected keyframe.
An electronic system comprising at least one processor comprising instructions for storing the one or more coordinate frames as one or more persistent coordinate frames in association with a map of the 3D environment. The one or more sensors include a plurality of pixel circuits arranged in a two-dimensional array such that each image of the plurality of images comprises a plurality of pixels.
Each feature corresponds to multiple pixels,
The electronic system according to claim 1. Identifying a plurality of features in the plurality of images is a predetermined feature of the group of pixels based on a measurement of similarity to the group of pixels depicting a portion of a persistent object as the identified feature. The electronic system of claim 1, comprising selecting a number less than the maximum value of. To store the one or more coordinate frames is to store the one or more coordinate frames for each of the one or more coordinate frames.
The electronic system of claim 1, comprising storing a descriptor representing at least a subset of the features in the selected keyframe from which the coordinate frames were generated. To store the one or more coordinate frames is to store the one or more coordinate frames for each of the one or more coordinate frames.
The electronic system of claim 1, comprising storing at least a subset of the features within the selected keyframe from which the coordinate frames were generated. To store the one or more coordinate frames is to store the one or more coordinate frames for each of the one or more coordinate frames.
The transformation between the coordinate frame of the map of the 3D environment and the persistent coordinate frame,
The electronic system of claim 1, comprising storing geographic information indicating the location of the selected keyframe in the 3D environment from which the coordinate frames were generated. The electronic system of claim 6, wherein the geographical information comprises a WiFi finger print at the location. The electronic system of claim 1, wherein the computer executable instruction comprises an instruction for calculating a feature descriptor for an individual feature using an artificial neural network. The first artificial neural network is a first artificial neural network.
The computer executable instruction is to calculate a frame descriptor to represent a key frame, at least in part, based on the calculated feature descriptor for the identified feature within the key frame. With instructions for implementing a second artificial neural network to be constructed,
The electronic system according to claim 8. The computer executable instruction further
An application programming interface, said application programming interface, is configured to provide an application running on a portable electronic system with information that characterizes the persistent coordinate frames of the one or more persistent coordinate frames. Application programming interface and
Instructions for refining the map of the 3D environment based on the second plurality of images,
Adjusting one or more of the persistent coordinate frames, at least in part, based on the second plurality of images.
The electronic system of claim 1, comprising instructions for providing notification of the adjusted persistent coordinate frame through the application programming interface. 10. The aspect of claim 10, wherein adjusting the one or more persistent coordinate frames comprises adjusting the translation and rotation of the one or more persistent coordinate frames with respect to the origin of the map of the 3D environment. Electronic system. The electronic system comprises a wearable device, and the one or more sensors are mounted on the wearable device.
The map is a tracking map calculated on the wearable device.
The origin of the map is determined based on where the device is powered on.
The electronic system according to claim 11. The electronic system comprises a wearable device, and the one or more sensors are mounted on the wearable device.
The computer executable instruction further
Tracking the movement of the portable device and
An instruction to generate one or more coordinate frames and / or an instruction to store one or more persistent coordinate frames based on the tracked motion indicating the motion of the wearable device over a threshold distance. The electronic system according to claim 1, further comprising an instruction for controlling the timing of execution, wherein the threshold distance is 2 to 20 meters. A method of operating an electronic system and rendering virtual content in a 3D environment equipped with a portable device, wherein the method uses one or more processors.
Maintaining a coordinate frame local to the portable device based on the output of one or more sensors on the portable device.
Acquiring the stored coordinate frame from the stored spatial information about the 3D environment,
To calculate the transformation between the coordinate frame local to the portable device and the acquired stored coordinate frame,
Receiving the specifications of a virtual object that has a coordinate frame local to the virtual object and the location of the virtual object relative to the selected stored coordinate frame.
A method comprising rendering the virtual object onto the display of the portable device at a determined location, at least in part, based on the calculated transformation and the received location of the virtual object. 14. The method of claim 14, wherein acquiring the stored coordinate frame comprises acquiring the coordinate frame through an application programming interface (API). The portable device comprises a first portable device comprising a first processor of the one or more processors.
The system further comprises a second portable device comprising a second processor of the one or more processors.
The processors on each of the first and second devices
To get the same stored coordinate frame,
To calculate the transformation between a coordinate frame local to an individual device and the same stored coordinate frame acquired above,
Receiving the specifications of the virtual object and
14. The method of claim 14, wherein the virtual object is rendered on a separate display. The first and second devices, respectively,
With a camera configured to output multiple camera images,
A keyframe generator configured to convert multiple camera images into multiple keyframes,
A continuous posture calculator configured to generate a continuous posture by averaging the plurality of key frames.
A tracking map and a continuous posture converter configured to convert the tracking map to the continuous posture and determine the continuous posture with respect to the origin of the tracking map.
A Sustained Posture and Persistent Coordinate Frame (PCF) transducer configured to convert the Sustained Posture to a PCF.
16. The method of claim 16, comprising a map issuer configured to transmit spatial information including the PCF to a server. 16. The method of claim 16, further comprising running an application to generate the specification of the virtual object and the location of the virtual object with respect to the selected stored coordinate frame. Maintaining a coordinate frame local to the portable device on the portable device is for each of the first and second portable devices.
Capturing multiple images of the 3D environment from one or more sensors on the portable device.
At least in part, calculating one or more sustained postures based on the plurality of images.
At least in part, including generating spatial information about the 3D environment based on one or more of the calculated sustained attitudes.
The method further comprises transmitting the generated spatial information to a remote server for each of the first and second portable devices.
Acquiring the stored coordinate frame includes receiving the stored coordinate frame from the remote server.
The method according to claim 16. At least in part, calculating the one or more sustained postures based on the plurality of images can be done.
Extracting one or more features from each of the plurality of images,
Generating a descriptor for each of the above-mentioned one or more features,
Generating keyframes for each of the plurality of images, at least in part, based on the descriptor.
19. The method of claim 19, comprising generating the one or more sustained postures based on the one or more keyframes, at least in part. Generating one or more of the sustained postures mentioned above
20. The method of claim 20, comprising selectively generating a sustained posture based on the portable device traveling a predetermined distance from another persistent posture location. The first and second devices, respectively,
16. The method of claim 16, comprising a download system configured to download the stored coordinate frames from a server. An electronic system for maintaining persistent spatial information about a 3D environment for rendering virtual content on each of a plurality of portable devices.
A networked computing device
With at least one processor
With at least one storage device connected to the processor
A map storage routine, wherein the map storage routine receives a plurality of maps from the portable devices of the plurality of portable devices and stores the map information on the at least one storage device. A map storage routine, which can be executed using a processor, wherein each of the received maps has at least one coordinate frame.
It is a map transmitter, and the map transmitter is
Receiving location information from the portable devices of the plurality of portable devices,
To select one or more maps from the stored maps,
By transmitting information from the selected one or more maps to the portable devices of the plurality of portable devices, the transmitted information is a map of the selected one or more maps. An electronic system comprising a networked computing device, comprising a map transmitter, which can be performed using the at least one processor to do so, including a coordinate frame. The coordinate frame is
23. The electronic system of claim 23, comprising a computer data structure comprising coordinate frames comprising information that characterizes a plurality of features of the object in the 3D environment. 23. The electronic system of claim 23, wherein the information characterizing the plurality of features comprises a descriptor that characterizes the region of the 3D environment. 23. The electronic system of claim 23, wherein each coordinate frame of the at least one coordinate frame comprises a persistent point characterized by features detected in sensor data representing the 3D environment. 26. The electronic system of claim 26, wherein each coordinate frame of the at least one coordinate frame comprises a sustained posture. 26. The electronic system of claim 26, wherein each coordinate frame of at least one coordinate frame comprises a persistent coordinate frame.

Images