JP2023511332A - Augmented reality map curation - Google Patents

Abstract

Augmented reality devices may communicate with a map server via an API interface to provide mapping data, and may receive map data from a map server, which may be implemented in a reference map. A map quality visualization, including quality indicators for multiple cells of the environment, may be provided to the user as an overlay to the current real-world environment seen through the AR device. These visualizations may include, for example, map quality minimaps and/or map quality overlays. The visualization provides a guide to the user that allows the map to be updated more efficiently, thereby improving the map quality and the user's positioning within the map.

Description

TECHNICAL FIELD This disclosure relates to virtual and augmented reality imaging and visualization systems, including mixed reality, and more particularly to systems and methods for displaying and interacting with virtual content.

Modern computing and display technologies are facilitating the development of systems for so-called "virtual reality," "augmented reality," and "mixed reality" experiences, in which digitally reproduced images are real. presented to the user in a manner that appears or can be perceived as such. Virtual reality (VR) scenarios typically involve the presentation of computer-generated image information without transparency to other actual, real-world visual inputs. Augmented Reality (AR) scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality (MR) is a type of augmented reality in which physical and virtual objects can coexist and interact in real time. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

Implementation details of one or more of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. Neither this description nor the following detailed description is intended to define or limit the scope of the inventive subject matter.

Augmented reality devices may communicate with a map server via an API interface to provide mapping data, and may receive map data from a map server, which may be implemented in a reference map. A map quality visualization, including quality indicators for multiple cells of the environment, may be provided to the user as an overlay to the current real-world environment seen through the AR device. These visualizations may include, for example, map quality minimaps and/or map quality overlays. The visualization provides a guide to the user that allows the map to be updated more efficiently, thereby improving the map quality and the user's positioning within the map.

FIG. 1 illustrates an example of an AR device that can be configured to provide AR/VR/MR scenes.

FIG. 2 is a block diagram of an embodiment of an AR environment.

FIG. 3 is a block diagram illustrating an example of data flow between multiple users and a map server.

FIG. 4 is a flow chart illustrating one embodiment of the map creation process.

FIG. 5A is a flowchart illustrating one embodiment of a process for curating maps as may be implemented by a developer or user.

FIG. 5B illustrates an exemplary user interface that may be provided upon activation of the map curation tool.

FIG. 5C shows an exemplary map quality map represented by the controller UI, which can be configured to follow the controller's movement, such that as the user moves throughout the environment, the map quality minimap is easily accessible. Illustrate the minimap.

FIG. 5C1 illustrates another view of the controller UI and associated map information.

FIG. 5C2 is a top view of an exemplary minimap.

FIG. 5D illustrates an exemplary map quality overlay overlaying the actual real-world area of the environment with color indicators of map quality within cells of the environment.

Figures 5E, 5F, and 5G are exemplary user interfaces including waypoint guide animations. Figures 5E, 5F, and 5G are exemplary user interfaces including waypoint guide animations. Figures 5E, 5F, and 5G are exemplary user interfaces including waypoint guide animations.

FIG. 6 is a flow chart illustrating one embodiment of a process for determining updates to map quality indicators.

Figure 7A is a flow chart illustrating one embodiment of a process for determining a cell quality score.

FIG. 7B illustrates an exemplary cell.

FIG. 7C illustrates the same exemplary cell of FIG. 7B, with multiple images superimposed on the corresponding viewing direction quadrants of the cell.

FIG. 8 is an exemplary user interface for the minimap.

9A-9E illustrate an exemplary user interface displayed to the user via the AR device as the user moves around the environment and acquires images that are used to improve map quality. . 9A-9E illustrate an exemplary user interface displayed to the user via the AR device as the user moves around the environment and acquires images that are used to improve map quality. . 9A-9E illustrate an exemplary user interface displayed to the user via the AR device as the user moves around the environment and acquires images that are used to improve map quality. . 9A-9E illustrate an exemplary user interface displayed to the user via the AR device as the user moves around the environment and acquires images that are used to improve map quality. . 9A-9E illustrate an exemplary user interface displayed to the user via the AR device as the user moves around the environment and acquires images that are used to improve map quality. .

Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example implementations described herein and are not intended to limit the scope of the disclosure.

DETAILED DESCRIPTION Embodiments of the present disclosure are directed to devices, systems, and methods for facilitating virtual or augmented reality interactions. As one exemplary embodiment, one or more user input devices may be used to interact in a VR, AR, or MR session. Such sessions may include virtual elements or objects in three-dimensional space. The one or more user input devices are further used for pointing, selecting, annotating, and drawing, among other actions on virtual objects, real objects, or the void in an AR or MR session. may be For ease of reading and understanding, certain systems and methods discussed herein refer to augmented reality environments and other “augmented reality” or “AR” components such as “AR devices” or “AR systems”. Point. These descriptions of "augmented reality" or "AR" specifically refer to "mixed reality,""virtualreality,""VR,""MR," and equivalents, and each of those "real environments" also specifically It should be construed to include the same as stated.
overview

To facilitate understanding of the systems and methods discussed herein, some terms are explained below. The terms set forth below and other terms used herein are intended to include the description provided, the ordinary and customary meaning of the term, and/or any other implied meaning associated with the individual term. should be construed and such interpretation is consistent with the context of the term. Accordingly, the following discussion does not limit the meaning of these terms, but provides exemplary explanations only.

Normative Map: A map that may be usable by multiple AR and non-AR (eg, smart phone) devices. A reference map may synchronize a common set of Persistent Coordinate Frames (PCFs) across devices, thereby enabling a multi-user experience. In some embodiments, the reference map may be dynamically updated over time by one or more users and may represent a digital replica of the real world.

Tracking map: generally a local map used by a particular AR or non-AR device, but a tracking map can be shared (e.g., at a common location) and available to multiple users It may be used to generate and/or update certain reference maps.

Localization: Determination of a location within a map based on matching sensor input (eg, images from a front-facing camera of a headset) with corresponding map data. For example, an AR system may process an image from a camera and determine whether features in the image match certain features in the map. If a match is found, the AR system can then determine the user's position and orientation based on the matched features.

Cell Quality Subscore: Indicates the amount of map data associated with a particular viewing direction that can be used to locate a user within a determined cell.

Cell Saturation Indicator: Indicates whether the user has been located within the determined cell for at least a threshold period of time.

Cell Score: Indicates the likelihood of location within a cell, which may be determined based on the Cell Quality Subscore and the Cell Saturation Indicator.

Application Programming Interfaces (APIs): APIs are generally defined communication channels that allow two devices to exchange information between each other in a more direct manner than might otherwise be possible; protocol, settings, and so on. In some embodiments, the API registration module issues tokens to individual devices that authorize such direct communication, thereby allowing specific computing devices (e.g., receiving, processing, storing, It may be configured to register individual devices (eg, AR devices, computing devices, Internet of Things devices, sensors, etc.) for communication with a central server that provides individual devices. Thus, a computing system can establish secure and direct communication channels with multiple devices via APIs.
Exemplary AR system

An AR device (also referred to herein as an Augmented Reality (AR) system), such as the example discussed below with reference to FIG. can be configured. Images may be still images, frames of video, or videos, in combination or the like. For the purposes of this disclosure, the term "AR" is used synonymously with the terms "MR" or "VR."

FIG. 1 illustrates an example of an AR device 100, which can be configured to provide AR/VR/MR scenes. AR device 100 may also be referred to as AR system 100 . AR device 100 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of display 220 . Display 220 may be coupled to frame 230 that is wearable by user 210 (which may also be referred to herein as a wearer or viewer). The display 220 can be positioned in front of the user's 210 eyes. The display 220 can present AR/VR/MR content to the user. Display 220 may comprise a head mounted display (HMD) that is worn on the user's head.

In some implementations, speaker 240 is coupled to frame 230 and positioned adjacent the user's ear canal (in some implementations, another speaker, not shown, is positioned adjacent the user's other ear canal). , providing stereo/shapeable sound control). Display 220 may include an audio sensor (eg, a microphone) to detect audio streams from the environment and/or capture ambient sounds. In some implementations, one or more other audio sensors not shown may be positioned to provide stereo sound reception. Stereo sound reception can be used to determine the location of the sound source. The AR device 100 can perform voice or speech recognition on audio streams.

AR device 100 may include an outward-facing imaging system that observes the world within the user's surrounding environment. The AR device 100 can also include an inward-facing imaging system that can track the user's eye movements. An inward-facing imaging system can track either one eye movement or both eye movement. The inward-facing imaging system may be mounted on frame 230 and processes the image information obtained by the inward-facing imaging system to, for example, determine the pupil diameter or orientation of the user's 210 eye, eye may be in electrical communication with processing modules 260 and/or 270, which may determine movement of the eye, or eye pose. An inward-facing imaging system may include one or more cameras or other imaging devices. For example, at least one camera may be used to image each eye. The images obtained by the camera determine the pupil size or eye pose for each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. may be used to

As an example, the AR device 100 can use an outward-facing imaging system or an inward-facing imaging system to obtain an image of the user's pose. An image may be a still image, a frame of video, or a video.

The display 220 may be fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removed by the user 210, such as by wired leads or a wireless connection. It can be operatively coupled 250 to a local data processing module 260, which can be mounted in a variety of configurations, such as being operably mounted (eg, in a rucksack configuration, in a belt-tied configuration).

Local processing and data module 260 may comprise a hardware processor and digital memory, such as non-volatile memory (eg, flash memory), both of which may be used to assist in processing, caching, and/or storing data. can be utilized. The data may include a) image capture devices (e.g., cameras in an inward-facing imaging system or outward-facing imaging systems), audio sensors (e.g., microphones), inertial measurement units (IMUs), accelerometers , compass, global positioning system (GPS) unit, wireless device, or gyroscope (eg, which may be operatively coupled to frame 230 or otherwise attached to user 210). or b) may include data obtained or processed using remote processing module 270 or remote data repository 280, possibly for processing or retrieval and subsequent passage to display 220. Local processing and data module 260 connects communication links 262 or 264 to remote processing module 270, such as via wired or wireless communication links, such that these remote modules are available as resources to local processing and data module 260. or may be operably coupled to remote data repository 280 . Additionally, remote processing module 270 and remote data repository 280 may be operatively coupled to each other.

In some implementations, remote processing module 270 may comprise one or more processors configured to analyze and process data or image information. In some implementations, remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some implementations, all data is stored and all computations (e.g., AR processes discussed herein) are performed in local processing and data modules, fully autonomous from remote modules. can enable the use of In other implementations, some or all of the calculations of certain AR processes discussed herein are performed remotely, such as on a network attached server.

The AR device may combine GPS and data obtained by a remote computing system (e.g., remote processing module 270, another user's AR device, etc.), which provides further information about the user's environment. be able to. As an example, an AR device can determine a user's location based on GPS data and retrieve a world map (which can be shared by multiple users) that includes virtual objects associated with the user's location.

In many implementations, an AR device may include other components in addition to or as an alternative to the AR device components described above. AR devices may include, for example, one or more tactile devices or components. A tactile device or component may be operable to provide a sense of touch to a user. For example, a tactile device or component may provide a haptic sensation of pressure or texture when touching virtual content (eg, virtual objects, virtual tools, other virtual structures). The sense of touch may reproduce the sensation of a physical object represented by a virtual object, or may reproduce the sensation of an imaginary object or character (eg, a dragon) represented by virtual content. In some implementations, tactile devices or components may be worn by a user (eg, user wearable gloves). In some implementations, the tactile device or component may be held by the user.

An AR device may include, for example, one or more physical objects that are manipulable by a user to allow input to or interaction with the AR device. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects such as metal or plastic pieces, walls, table surfaces. In some implementations, the totem may not actually have any physical input structure (eg, keys, triggers, joysticks, trackballs, rocker switches). Alternatively, the totem may simply provide a physical surface, and the AR device may render the user interface so that it appears to the user to be on one or more surfaces of the totem. . For example, an AR device may render images of computer keyboards and trackpads to appear to reside on one or more surfaces of the totem. For example, an AR device may render a virtual computer keyboard and virtual trackpad to appear visible on the surface of a thin rectangular plate of aluminum that acts as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, AR devices may detect user manipulations or interactions or touches with the rectangular plate as selections or inputs made via a virtual keyboard or virtual trackpad. User input devices 466 (shown in FIG. 4) may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. , may be a totem implementation. Alone or in combination with postures, users may use totems to interact with AR devices or other users.

Examples of tactile devices and totems that can be used with the AR devices, HMDs, and display systems of the present disclosure are described in US Patent Publication No. 2015/0016777, which is incorporated herein by reference in its entirety. be done.

The exemplary components depicted and/or described above with reference to FIG. 1 are for illustration purposes only. Multiple sensors and other functional modules are shown together for ease of illustration and explanation. Some implementations may include only one or a subset of these sensors or modules. Furthermore, the locations of these components are not limited to the locations depicted in FIG. Some components may be mounted on or housed within other components, such as belt-mounted components, handheld components, or helmet components.
An example of mapping the user's environment

FIG. 2 is a block diagram of an example AR environment 200 . The AR environment 200 accepts inputs (e.g., visual input 202 from AR devices, stationary inputs 204 such as room cameras, sensory inputs 206 from various sensors, gestures, totems, eye tracking, user input from AR devices, etc.). from one or more user AR devices (eg, AR device 100) or stationary indoor systems (eg, indoor cameras, etc.). AR devices use a variety of sensors (e.g., accelerometers, gyroscopes, temperature sensors, motion sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.). can determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different viewpoints or different cues. Image data obtained by a camera (such as an indoor camera and/or an outward-facing imaging system camera) may be reduced to a set of mapping points.

One or more object recognizers 208 crawl through the received data (e.g., set of points), recognize or map the points, tag the images, and use the map database 212 to extract semantic information. can be attached to an object. The map database 212 may comprise various points and their corresponding objects collected over time. Various devices and map databases can be interconnected through a network (eg, LAN, WAN, etc.) to access the cloud.

Based on this information and the point sets in the map database, object recognizers 208a-208n may recognize objects in the environment. For example, an object recognizer can identify faces, people, windows, walls, user input devices, televisions, documents (eg, passports, driver's licenses, passports as described in the security embodiments herein), the user's environment. Other objects, etc. within can be recognized. One or more object recognizers may be specialized for objects with certain properties. For example, object recognizer 208a may be used to recognize faces, while another object recognizer may be used to recognize documents.

Object recognition may be performed using various computer vision techniques. For example, AR devices analyze images obtained by outward-facing imaging systems to provide scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), object pose estimation, facial recognition ( (e.g., from images on documents or people in the environment), learning, indexing, motion estimation, or image analysis (e.g., identifying indicia in documents such as photographs, signatures, identification information, travel information, etc.) can do. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms are Scale Invariant Feature Transform (SIFT), Speed Up Robust Features (SURF), Oriented FAST and Rotation Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual simultaneous localization and mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g. Kalman filter, Extended Kalman filter) etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbors (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms ( For example, support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised/unsupervised models, etc.).

Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithms, etc.), Artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., Stacked Gneralization, etc.) , and/or other machine learning algorithms. In some implementations, individual models can be customized for individual datasets. For example, an AR device can generate or store a base model. The base model is used as a starting point and includes data types (e.g. a particular user in a telepresence session), datasets (e.g. a set of additional images acquired of a user in a telepresence session), conditional situations , or additional models specific to other variations may be generated. In some implementations, wearable HMDs can be configured to utilize multiple techniques to generate models for analysis of aggregated data. Other techniques may include using predefined thresholds or data values.

Based on this information and the set of points in the map database, the object recognizers 208a-208n may recognize the object, augment the object with semantic information, and give life to the object. For example, if an object recognizer recognizes that a set of points is a door, the system may associate some semantic information (eg, a door has a hinge and 90 degrees centered on the hinge). degree shift). If the object recognizer recognizes that the set of points is a mirror, the system may associate the semantic information that the mirror has a reflective surface that can reflect images of objects in the room. Semantic information can include affordances of objects, as described herein. For example, semantic information may include normals of objects. The system can assign a vector whose direction indicates the normal of the object. Over time, the map database grows as the system (which may be locally resident or accessible through a wireless network) accumulates more data from around the world. Once the object is recognized, the information may be transmitted to one or more AR devices. For example, AR environment 200 may include information about scenes taking place in California. Environment 200 may be transmitted to one or more users in New York. Based on the data received from the FOV camera and other inputs, the object recognizer and other software components are configured so that the scene can be accurately "passed" to a second user who may be in a different part of the world. Points collected from different images can be mapped, objects can be recognized, and so on. The environment 200 may also use topology maps for localization purposes.
Exemplary Map API Settings

FIG. 3 is a block diagram illustrating an exemplary data flow between multiple users and a map server 310 (also referred to as a "cloud server" or simply "the cloud"). In this example, API interface 330 is implemented to enable more direct and efficient communication between users 302 , 304 and map server 310 . In some embodiments, map server 310 may be part of remote processing and data module 260 (FIG. 1), or map server 310 is a separate cloud server accessible to multiple users. may be

As shown in the example of FIG. 3 , developer 302 may communicate with map server 310 to provide mapping data, which may be implemented in canonical map 320 and displayed to developer 302 . , and/or may be received from map server 310 via API interface 330, which may be used to guide developers in obtaining additional mapping data. Similarly, a user 304, such as a consumer, who purchases an AR device also interfaces with map server 310 via API interface 330 to provide additional map data to map server 310; It may both receive from map server 310 .

In the FIG. 3 embodiment, map server 310 also includes image store 322, which stores images obtained from various sources (eg, AR devices) that are linked with associated portions of reference map 320. may contain.

In some embodiments, the map server 310 evaluates map data from multiple entities (e.g., multiple users) for map data quality and then compares the high quality map data with any existing maps in the reference map. map data (eg, promote a tracking map to a reference map). For example, multiple users may exist in a common environment and transmit images to map server 310 . In some embodiments, the integration of map data from multiple users is performed in real-time so that each of the users can benefit from increased map quality. In some embodiments, the map quality is updated separately within each AR device based on the images acquired by the AR device, and on a periodic basis (e.g., nightly), as part of the reference map, and other user's image data. Thus, each AR device can utilize improved quality maps immediately, and later, potentially even higher quality maps, as the map server integrates map data from other users.
Illustrative Map Creation Process

A user, such as developer 302 or user 304, has not previously interacted with or visited its current environment, has not previously scanned its current environment, or the AR system If you are failing to recognize the environment, you may create a map of the environment. FIG. 4 illustrates an exemplary flow chart of map creation process 400 . In some embodiments, a process similar to FIG. 4 may be implemented to refine the current map, even though the user may be relocated within an existing map. (eg, by transmitting additional data at block 422, as discussed below). Additionally, even if the user is not in an immersive flow designed to help the user gather data to improve the map, the AR device can, for example, update the reference map. Some or all of the processes of FIG. 4 may still be implemented to provide background transmission of mapping data to the cloud, which may be used for

At the start mapping block 410, the AR system can decide whether to start scanning or mapping the user's environment. For example, the AR system can determine if a start condition is met to begin scanning the environment. In some examples, the start conditions include detecting user movement into a location new and/or unknown to the system, input from one or more sensors, and/or user input. be able to. User input may include affirmative or negative responses to one or more prompts. One or more prompts are used to initiate prompts, whether the user is a new or existing user, whether the user has previously traversed the environment to create a map, or It can differ based on any number of AR system requirements, such as the type of program being used. As another example, developer 302 may enter the mapping workflow in a different manner than user 304 . For example, the developer 302 may associate software applications developed by the developer to enable the developer 302 to explore the environment and gather sensor data that can be used to build maps of the environment. may initiate a new environment mapping process.

In some embodiments, a welcome user interface may be presented when the user enters the mapping process, such as via block 410 of FIG. The welcome interface may include a dialog that prompts the user to scan or continue scanning the user's environment. In some examples, the welcome interface may receive user input, and the AR system may or may not start scanning based on the user input. Additionally or alternatively, the AR system may move the user to another prompt based on user input.

At scanning block 412, the AR system may begin a scanning process, which may include providing the user with a guide of areas in the environment for which additional images should be acquired. In some embodiments, the scanning process may be a process that has a gamified component that helps the user direct movement around the environment and collect data within the space. . For example, an AR system generates one or more graphics (also called waypoints), displays them around the user's environment, and prompts the user to interact with the graphics until an exit criterion is met. may be instructed to As used herein, a waypoint may refer to a particular location within a map and/or a graphic (or other indication) of a particular location within a map. Thus, a waypoint may include a graphic that marks a particular location within the map and/or directs the user to the waypoint location. During the user's interaction with one or more graphics, the AR system may collect data about the user's environment.

In some embodiments, the AR system may check whether the user's environment is known or known at map recognition block 414 . The AR system may perform this check during or after scan block 412 . For example, the AR system may perform a scanning process at block 412, and the AR system may check at intervals during the scanning process whether the user's environment matches the known environment. Good (eg, the AR system can match one or more PCFs found in the user's current environment with one or more PCFs in the user's saved map). If the map is recognized by the AR system, the AR system may restore AR content associated with the recognized map at block 420 before entering the landscape at block 424 . If the map is not recognized by the AR system, the system can check map quality at block 416 .

In map quality block 416, the AR system determines that the map generated based on the data collected during scan block 412 (and/or combined with data stored within the virtual world map) is current and/or Or for future use, it can be checked if the quality is high enough to provide a high quality user experience. The quality metric can be any suitable criterion for assessing map quality, such as the number of keyframes, PCFs, or other data associated with meshes or other map characteristics in the user's environment. For example, the AR system may determine whether sufficient PCFs have been found or generated based on the collected data to make the user's space identifiable in future scans. The number of PCFs may be any suitable number, such as 1, 2, 3, or 5 PCFs in the user's environment. However, other numbers are also possible. For example, the number of PCFs required for a particular environment may be dynamically determined by the AR system, such as based on analysis of collected scan data and/or map data previously associated with the environment. . Once the AR system determines that the map exceeds the quality threshold, the AR system may use the collected data to save the map at block 422 . Further discussion and examples of map quality determination are provided below.

At save block 422, the AR system may save the map to remote or local memory for retrieval by a user or third party. For example, the AR system of user 304 (FIG. 3) may transmit map data obtained from the AR system of user 304 to map server 310 via API interface 330 . Additionally or alternatively, the system may prompt the user to enter other information associated with the map, such as a name or geophysical location, to be stored as metadata with the map.

If the map quality is not sufficient to provide a high quality user experience, the AR system may determine at decision block 418 whether the user wishes to continue scanning or mapping the user's environment. can. For example, the AR system can prompt the user to continue or stop scanning the scanning process. The AR system may receive user input in response to the prompt to continue scanning the environment at block 412 or enter the landscape at block 424 .

Additionally or alternatively, the AR system can stop the map creation process 400 and enter the landscape at block 424 at any time. For example, the user can enter an exit or skip command during the scanning process at block 412 . The AR system can then abort the scanning process at block 412 or enter the landscape at block 424 .
Exemplary map curation

FIG. 5A is a flowchart illustrating one embodiment of a process for curating maps as may be performed by developer 302 or user 304 . In general, the map curation process optimizes the collection of sensor data (e.g., image data within the environment to be mapped) to improve map quality, as discussed herein. and guides to users. In some embodiments, communication between users (eg, either developer 302 or consumer user 304) may occur via API communication channels as illustrated in FIG. As discussed further below, real-time map quality data may be calculated and provided to users as part of the map curation process. For example, a map visualization may be displayed within the AR device that guides the user to areas of the environment (eg, real-world space to be mapped) that currently have low map quality. Improvements in map quality provide a more realistic and reproducible experience for users. For example, as the quality of maps increases, digital content associated with the real world better survives between sessions of a particular user or between multiple users. Another potential benefit of having a higher quality map is that end-users entering the area will be more quickly, with a higher probability, and more likely to find specifics within the area with improved map quality. Depending on the location, it can be located in the map from more perspectives.

In one embodiment, a welcome screen may be provided to the developer that allows selection of either guided setup or map creation mode. For example, FIG. 5B illustrates an exemplary user interface that may be provided upon activation of the map curation tool. Map creation or refinement mode (e.g., the "Create Map" button in the user interface of FIG. 5B, or in embodiments where at least some map data associated with the environment is already stored in the cloud or elsewhere , “Refine” button), a map-quality visualization may be provided to the user as an overlay to the current real-world environment seen through the AR device. These visualizations may include, for example, map quality minimaps and/or map quality overlays. Additional visual components, such as meshes, PCFs, and the like, which may be available in further developing the map, may also be displayed. In some embodiments, the map creation mode is first entered to build a map, but the map creation (or refinement) mode is also used when the user already has an existing map (e.g., very few existing maps). map data). As discussed further below, new map data may be merged with any existing cloud/reference map.

FIG. 5C illustrates an exemplary map quality minimap 502, represented by controller UI 504, such that the map quality minimap is easily accessible as the user moves through the environment. For example, it may be configured to follow the movement of a handheld totem operated by a user. In some embodiments, any user input device or object, such as a user's hand or spoon, may be used in place of the controller and have its own associated UI. In general, the map quality minimap 502 shows the user's current location, with the user icon 501 at the center of the visualization, and a plurality of cell-by-cell quality indicators around the user icon 501 . In some implementations, the mini-map 502 is referenced in related U.S. Patent Application No. 62/965,708 (Attorney Docket No. MLEAP.298PR/ML-1007USPRV) (incorporated herein by reference in its entirety for all purposes) to follow the position of the controller. , may have a “delayed” tracking property. In some embodiments, the minimap 502 may automatically rotate so that the user remains facing north, e.g., the minimap rotates/rotates depending on where the user is facing. deviate. The user also has the option to keep the minimap in a "north up" orientation so that the orientation of the user icon 501 at the center of the minimap is rotated while the orientation of the minimap is not changed. may In some embodiments, the mini-map may be tilted up toward the user, such as based on the user's current head pose. In embodiments where a controller UI is included in the visualization (e.g., controller UI 504 in FIG. 5C), the UI is directed upward toward the user, such as at a predetermined adjustable tilt (e.g., 10, 20, or 30 degree tilt). , may be oriented for easy viewing by the user.

In the example of FIG. 5C, additional map information 503 is displayed in association with controller UI 504 . For example, a map status may be displayed, indicating whether the map is "live" or "fixed." In some embodiments, developers are provided options to pin (or lock) maps or portions of maps from further updates. For example, the user may provide a long press of the trigger (eg, 2 seconds or more) or other specified input to freeze or unpin the map.

An overall map quality indicator 506 may also be displayed to the user, which uses color (or in other embodiments, other visualization effects) to indicate that the map is currently of low quality (e.g., red ), average quality (eg yellow), or high quality (eg green). Additional map information 503 may also include hints or help on how to proceed with additional mappings or finish the map curation process. FIG. 5C1 illustrates another view of the controller UI 504 and associated map information 503. FIG. In this example, overall map quality indicator 506 is displayed at a 3D Z position, farther away from controller 504 than mapping hint information 507 .

FIG. 5C2 is a top view of an exemplary minimap 502. FIG. In this example, mini-map 502 includes a defined radius of the map (eg, reference and/or tracking maps) centered on the user's current location. In this example, each cell of the map is represented by a 27×27 mm cell and the radius of the minimap is 145 mm. Using these exemplary dimensions, the mini-map contains cell quality indicators for 4-5 cells north, south, west, and east from the central cell. In this example, the mini-map fades out starting at a certain distance from the outer radius, such as 115 mm in the example of FIG. 5C2. In other embodiments, other dimensions for the cells and minimap, whether static or dynamic (eg, adjustable by the user), may also be used and/or other dimensions for the minimap's components. visualization may also be used.

FIG. 5D illustrates an exemplary map quality overlay 506 overlaying the actual real-world area of the environment along with color indicators of map quality within cells of the environment. A map quality minimap 505 is displayed in conjunction with a map quality overlay 506, as shown in FIG. 5D. Thus, depending on the embodiment, one or both of the map quality minimap and/or the map quality overlay may be displayed to the user as part of the map curation process.

Referring again to FIG. 5B, the user may select a guided setup option (eg, the “Guided Setup” button in the user interface of FIG. 5B) to provide additional guidance in mapping the environment. . In one exemplary implementation, guided setup provides the user with waypoint indicators, such as using gamified animations, to direct the user to areas of the map where additional images are needed. . These additional guiding features may be provided in conjunction with or independent of map quality visualization. Figures 5E, 5F, and 5G are exemplary user interfaces including waypoint guide animations. Related U.S. Patent Application No. 62/937,056, entitled "Mapping and Localization of a Passable World" (No. and is incorporated herein by reference in its entirety for all purposes.

Returning to FIG. 5A, in some embodiments, software developer 302 (FIG. 3), to collect initial map data, etc., associated with the environment (eg, the real-world space to be mapped): Some or all of the methods of FIG. 5A may be implemented. In some embodiments, one or more other users 304 update the reference map, provide additional map data that can be used to improve the quality of the map, etc. Some or all of the methods of 5A (eg, the “subsequent scanning of the environment” portion of the flowchart) may be implemented.

Starting at block 510, an initial scan of the environment to be mapped may be obtained. For example, a developer (or other user) may use an AR device to obtain image data of an environment or area, such as an office space, that does not already have any data in the reference map. In some implementations, developers may obtain map data from other sources, such as using one or more LIDAR sensors (which may be manually moved around the environment or automatically (whether or not it is moved through the environment, e.g., robotically controlled). In some embodiments, the developer may walk around the environment and obtain images and/or other sensor data that can be used to create a mesh of the environment. As the developer moves around the environment to be mapped, the AR system obtains image data along the user's movement path that can be processed into one or more tracking maps. As used herein, tracking maps generally refer to local maps used by a particular AR system, although tracking maps may be shared among multiple users (e.g., common location ), may be used to generate and/or update reference maps that are available to multiple users, as discussed further below.

Moving to block 520, the map data obtained at block 510 may be uploaded to a server, such as map server 310 (FIG. 3), where it may be processed for inclusion within the reference map. For example, a server may receive map data from a myriad of entities, process the map data, determine the most authoritative map data, and merge them into a canonical map.

Next, at block 530, the map data are stitched and/or merged together to generate or update the reference map so that it can be stored as the reference map 320 associated with the map server 310 (FIG. 3). Therefore, the reference map can occur as a tracking map. The reference map allows a device accessing the reference map to use the information in the reference map to map the physical surroundings of the device once a transformation between its local coordinate system and the coordinate system of the reference map has been performed. It may be persisted so that the location of objects represented on the reference map within the world can be determined. Thus, in some embodiments, tracking map data received from an AR device may be used to generate an initial reference map of the environment. Data stitching and merging may identify overlapping portions of images and common reference points within such images to improve map quality. Subsequent map data may then be integrated into the reference map, and additional map data may improve the quality of the map, as discussed below.

Reference maps, tracking maps, and/or other maps may provide information about the portion of the physical world represented by the data processed to create the individual maps. For example, a map may provide a floor plan of physical objects in the corresponding physical world. In some embodiments, a map point may be associated with a physical object or a feature of a physical object (eg, if the physical object includes multiple features). For example, each corner of the table can be a separate feature represented by a separate point on the map (eg, four map points associated with the four corners of the table). Features may be derived from processing images such as may be obtained with the sensors of an AR device within an augmented reality system.

Continuing at block 540, the user may then access the reference map generated at block 530, such as via API communication with map server 310 (FIG. 3). A user may be a developer or any other user. At block 550, the user acquires additional images of the environment to locate the user within the environment. For example, when a user's AR device is activated, images from one or more cameras may be acquired and used to determine the user's location within the map. If localization does not occur immediately, the user may move around the environment and acquire additional images until localization within the reference map occurs.

Next, at block 570, one or more map quality indicators are dynamically generated to provide a visual indicator of map quality to the user as the user moves further around the environment. Advantageously, such an indicator can be used by the user to more efficiently navigate to areas of the map where additional images can provide the greatest quality improvement.

Finally, at block 580 the additional images obtained by the AR system are provided to the map server 310 where they can be analyzed and used to update the reference map. Therefore, in conjunction with the map quality interaction described above, obtaining additional map data useful in updating the reference map can be performed more efficiently.

In some embodiments, the mesh of the mapped environment may be displayed and updated as the user interacts with the AR device through the process of FIG. 5A. In some embodiments, the mesh is of low quality for which additional images are useful in improving the quality of the map (and correspondingly the localization efficiency within that portion of the map). Wax indicates a portion of the map. In other embodiments, the mesh may not be displayed to the user during some or all of the map curation process of FIG. 5A.

FIG. 6 is a flow chart illustrating one embodiment of a process for determining updates to map quality indicators. In some implementations, a user of an AR device is located with reference to the user's current environment using a location map, which can be a tracking map, a reference map, and/or some combination of maps. . Real-time images acquired from the AR device's camera system (e.g., one or more of the most recently captured) are then used to determine the location of the user in an attempt to determine the user's location within the map. It may be compared against a specific map. Successful localization depends on the quality of the localization map. Therefore, a map quality metric that indicates the likelihood of success for using a map for localization may be advantageous. As discussed in more detail below, the map quality metric is determined based on fiducial points captured in different images, such as the overall amount of fiducial points and/or the simultaneous visibility of fiducial points between images. may The reference points may be map points representing features of interest, such as corners, edges, etc., which may be used to identify objects in the environment for localization purposes.

In the example of FIG. 6, the map is segmented into a plurality of cells, such as in a grid pattern, and each cell is scored to determine one or more quality indicators. In some embodiments, a grid of cells associated with a map may be referred to as a pose grid block (or PGB). Advantageously, the cell quality score is dynamically generated, e.g., as the user moves around the environment, and may benefit from improvements to map quality and/or additional map data for the current cell. Real-time feedback of neighboring cell indications may be provided to the user.

Beginning at block 610, the map is segmented into multiple cells, each associated with a defined area of the real-world environment. Depending on the implementation, the map may be some or all of the reference maps associated with the environment in which the user is located, such that it may be transmitted (from the map server 310) to the AR system during localization. may For example, each cell may be associated with a predetermined area of 4m×4m, 2m×2m, 1m×1m, or any other size or shape of the real-world environment.

Moving to block 620, the cell in which the user is currently located is determined, such as based on location techniques discussed elsewhere herein.

Next, at block 630, a cell quality score is determined for the cell in which the user is currently located (eg, as determined at block 620). In one embodiment, a cell quality subscore in the range of 0 to 1 is determined for each of multiple viewing directions of the cell. A cell quality score may then be determined based on the subscores, such as by averaging the subscores. For example, a cell that is divided into four viewing directions may have four corresponding cell quality subscores in the range 0-1 (or other range). The four subscores (e.g. 0.5, 0.6, 0.7, 0.25) may be averaged to obtain a cell quality score, e.g. (0.5+0.6+0.7+0.25) /4 yields a cell quality score of 0.5125. In other implementations, other score ranges may be used and the overall cell quality score may be calculated differently. For example, in some embodiments the cell quality score is influenced by surrounding cells. For example, the cell quality scores of the cell in directions north, south, west, and east of the current cell are used in calculating the current cell quality score, such as by associating a weight to each of the surrounding cell quality scores. may be In other embodiments, cells may be scored in other manners. A further example of a scoring methodology is described in further detail with reference to FIG. 7A, which is a flowchart illustrating an exemplary method of determining cell quality scores. Proceeding to FIG. 7A, at block 710, the AR system and/or map server 310 (FIG. 3) determines an image associated with the current cell (eg, the 2m×2m area in which the user is currently located). . The image may be stored locally, such as as part of the tracking map, and/or from the cloud (e.g., the map server of FIG. 3) in response to a request for an image associated with the current cell in the reference map. 310).

Moving to block 720, the images identified in block 710 are grouped based on cell viewing direction and/or some other segmentation. For example, each image may be associated with one of four viewing directions (eg, north, south, east, and west, or up, down, left, and right). Other amounts and orientations of viewing direction may also be used in other embodiments. In the example of FIG. 7B, an exemplary cell 749 is illustrated. Exemplary cell 749 includes four quadrants or viewing directions, with user 742 positioned in the center of the cell.

FIG. 7C illustrates the same cell 749, where multiple images 750A-750E are superimposed within the cell 749 from which the images were obtained (eg, by another user or the current user in the cell). and positioned. Image icons 750A-750E also indicate the viewing direction from which the image was obtained, which points from the narrower side of icon 750 toward (and beyond) the wider side. Thus, in the present example, images 750A, 750B, and 750C show content from an east viewing direction within cell 749, e.g. contains the image obtained from the imaging device in cell 749, looking at . For example, if north is zero degrees, south is 180 degrees, and east is 90 degrees, the range of viewing directions associated with the east viewing direction is (90-X) to ( 90+X), where X is 45 or less. Similar viewing direction ranges may be determined for North, South, and West using the same X value. However, in some implementations, X is greater than 45, e.g., 60, so that some images taken from viewing angles between the two directions can be included within the two groups of images. may be For example, in some embodiments with overlapping viewing direction areas, images taken from northeast viewing angles may be included in both the north (740N) and east (740E) image groups. In some embodiments, other number groups (eg, 2, 3, 10, etc.) and/or other X values (1, 10, 80, etc.) may also be used.

Still referring to FIG. 7C, image 750E includes content from a south viewing direction and image 750D includes content from a west viewing direction. Thus, after categorizing these exemplary images, three images are associated with the east viewing direction (associated with group 740E in cell 749), one image is associated with the west facing group (740W) and the south facing group (740S) and none of the images are associated with the north facing group (740N).

Determining the viewing direction of the image and its corresponding viewing direction group does not depend on the particular location of the user within the cell, the image may have been taken from any location within the cell 749. .

Returning to Figure 7A, at block 730, an image score is determined for each image. For example, the image score may be based on the amount of reference points contained within the image. In some embodiments, image scoring may be performed early in the process, such as prior to block 710 . For example, images may be scored as they are received by the AR system and/or cloud server. In some embodiments, image scores are normalized, such as by the highest amount of reference points in any image associated with the map. For example, if the maximum fiducial point in any image associated with the map is 10, then an image with two fiducial points may be normalized to an image score of 0.20, with seven fiducial points. Images may be normalized to an image score of 0.70. Using normalized image scores may provide a more comprehensive global map-quality view than using only the raw amount of fiducials in the image.

Further discussion of image scoring and grouping of images based on viewing direction is provided in U.S. Patent Application Serial No. No. ML-0750US), incorporated herein by reference in its entirety for all purposes.

Next, at block 740, for each viewing direction, a cell quality subscore may be generated, such as based on the image score of the image associated with the particular viewing direction. Thus, referring to Example 7C, nine cell quality sub-scores may be calculated for viewing direction group 740E based on identifying three reference points within each of images 750A-750C (eg, 3+3+3=9 ). In embodiments where the image score is normalized, the normalized score may be used in calculating the cell quality subscore. For example, if images 750A, 750B, 750C have normalized image scores of 0.7, 0.3, and 0.5, then the cell quality subscores for group 740E are the fractions of their normalized image scores. It may be an average, ie 0.5.

In some embodiments, the cell quality subscore may further be based on the simultaneous visibility of fiducial points within images of a particular viewing direction group. For example, there are two simultaneous visible reference points in each of images 750A and 750B, three simultaneous visible reference points in each of images 750B and 750C, and one simultaneous visible reference point in each of images 750A and 750C. If so, six cell quality sub-scores may be calculated for viewing direction group 740E (eg, 2+3+1=6). Other variations of determining cell quality sub-scores may also be used in other embodiments.

A cell quality score may then be calculated based on the cell's cell quality subscore. For example, four cell quality subscores may be averaged to determine a cell quality score. In some embodiments, the cell quality score may be normalized to a general range such as 0-1, 0-10, or 0-100. For illustrative purposes, cell quality scores ranging from 0 to 1 are discussed herein.

Returning to FIG. 6, once the cell quality score is calculated, a cell saturation indicator is determined for the current cell. In one embodiment, the cell saturation indicator is based on the amount of time the user has been located within the current cell (eg, within a 2m x 2m area of the current cell). A numerical score, such as the number of seconds the user has been in the current cell, may be determined, or in some embodiments whether the current user has been in the cell for more than a certain period of time. An indicator of whether is determined. For example, the threshold time is that if the user has been in the current cell for 10 seconds or more, the cell saturation indicator is positive (or 1), while the user has been in the cell for less than 10 seconds. , the cell saturation indicator may be set to 10 seconds such that it is negative (or zero). Other threshold times may also be used in other embodiments. In some embodiments, the saturation subscore is for each of the cell's multiple viewing directions (eg, the four viewing directions 740N, 740S, 740W, 740E in FIG. 7C), such as the viewing directions for which the cell quality subscore is generated. may be calculated to A cell saturation score (eg, the sum, average, weighted average, etc. of the cell's subscores), which is a representation of multiple saturation subscores associated with the cell, may be calculated.

At block 650, an adjusted cell quality score (or "cell score") is determined based on the cell quality score (block 630) and the cell saturation indicator (block 640). For example, the cell score may be based on the cell quality score and some weighting value when the cell saturation indicator is positive. For example, in one implementation, if the cell quality score is less than a predetermined amount (e.g., 0.5), the cell score is determined as the cell quality score +0.5 increase due to the positive cell saturation indicator. be done. If the cell saturation indicator is negative, the cell score may equal the cell quality score. In other embodiments, the cell saturation indicator and cell quality score may be combined in different proportions to determine the cell score.

Moving to block 660, once the cell score is calculated, an updated map visualization may be provided to the AR device to allow the user to visualize the cell score. In some embodiments, cell scores are indicated using colors such as red, orange, yellow, green, etc., and/or gradients between such colors. For example, a cell score of 1 may result in a green indicator for the cell, while a yellow indicator for the cell is displayed if the cell score is greater than or equal to 0.5 but less than 1. Similarly, cell scores that are 0.25 or less may be shown as red. A color gradient may be used to more precisely indicate where the score lies on the zero to one scale. In another example, the color gradient for the scores may be determined based on the gradient between two colors, such as green for a score of 1 and red (orange) for a score of zero. As discussed earlier, the visualization may include a mapping guide that directs the user to areas of the environment within which cell scores are lower and for which additional images are desirable.

FIG. 8 is an exemplary user interface for minimap 800 . In this example, the minimap 800 has some cells 802 (e.g., green circles) with high cell scores, four cells 804 (e.g., yellow circles) with medium cell scores, and four cells 804 (e.g., yellow circles) with low cell scores. One cell 806 (eg, red circle) is shown. In addition, the cell in which the user is currently located, cell 802A, has an orange-colored central portion and indicates a mean or median score, with a directional score towards cell 806 being low (red). , toward cell 804A is moderate (yellow), the directional score in the other two directions is high (green), and so on.

As described elsewhere herein, the range of values for ranking cell scores as high, medium, low, and/or any other variation in between varies based on implementation. can. Additionally, other colors may be used to represent cell score values and/or other visual indicators, such as the association of object size and quality level within each cell (e.g., cells with the highest scores is filled, and the cell contains only a dot in the center of the cell, or nothing, when the score is the lowest). In some embodiments, such as the example of FIG. 8, colored dots are included only in cells for which some quality data exists (eg, cells with cell scores). Because even a very short visit to a cell typically allows the collection of sufficient data to calculate a cell score (even though it may be a lower cell score), this embodiment The lack of colored dots in the may indicate that the user was not in that cell.

In the example of FIG. 8, current cell 802A shows the cell quality score in each of the four viewing directions along with the overall cell score (eg, orange) in the center of cell 802A. Thus, the colors in cell subscoring 810 indicate the green, red, yellow, and green cell quality subscores associated with the quality of the image obtained from that viewing direction from within cell 802A (starting north going clockwise). In some embodiments, multiple directional quality indicators are calculated and stored for each cell, but an overall cell score may be shown only for cells outside the current cell in which the user is located. This may reduce user interface complexity while still providing an overall quality score for any cell for which a score was determined.

In the embodiment of FIG. 8, if the user enters cell 804A, currently indicated by a yellow dot, in response to entering the cell the user icon is positioned within cell 804A and the cell: It may be updated to show four directional scores for cell 804A. For example, cell 804A may be associated with three green and one red directional quality indicators, but a user would not know the direction with which each score is associated until entering cell 804A. Once the user enters cell 804A (and a directional score indicator is shown in cell 804A), if the red directional quality indicator is to west, it prompts the user to look west. and will acquire additional images in that direction that can be used to improve the quality of the map in that direction, increasing both the West Quality subscore and the overall cell score.

In some embodiments, an overall map quality score may be calculated based on one or more of the map's cell scores. For example, in one embodiment, the overall map quality score is the average of the cell scores of all cells of the map. In another embodiment, the overall map quality score may be based on the quantity or percentage of cells in the map that have been assigned cell scores. For example, if a map contains 100 cells, but only 53 of the cells have an associated cell score, the overall map score may be 0.53 or 53%. In some embodiments, an overall map quality score may be calculated based on the portion of the map shown in the mini-map (eg, the cells immediately surrounding the user) rather than the entire map. The overall map quality score can be shown in associated map information 503, as illustrated with reference to controller UI 504 in FIG.

9A-9E illustrate an exemplary user interface displayed to the user via the AR device as the user moves around the environment and acquires images that are used to improve map quality. do. Starting with FIG. 9A, a minimap 902 contains a user icon 901 in the center cell of the map. At this stage of the mapping process, the cell in which the user is currently located has been assigned a cell score, as indicated by the lack of colored circles within any other cell of the minimap 902. It is the only cell. In this embodiment, the user interface also includes a mesh 910 indicator in the form of white dots throughout the lower right portion of the exemplary user interface of FIG. Indicates the extent of meshing for a particular area. Additional mapping information 903 is also displayed in this example, including overall map quality 906, indicating that the overall map quality is currently 0%.

Moving to FIG. 9B, the user has moved into another cell and an additional cell score is shown. In some embodiments, cell scores are recalculated at each map event, such as when a new image is obtained by the AR device or when the user moves across cell boundaries. As shown in the example of FIG. 9, a viewing direction indicator 905 is shown next to the user icon 901 to help orient the user relative to the minimap. A visual indicator 905 advantageously indicates the user's real-time orientation. In other embodiments, the user's orientation may be displayed in other manners. FIG. 9C further illustrates how the minimap is updated in real time as the user moves around the environment. Similarly, mesh indications are also updated in real time. FIG. 9D again shows an overall map indicator 906, but this time due to the user moving around the environment and allowing the system to develop cell scores for three cells. with an indication that the map quality is 33%. In FIG. 9E, the map quality indicator 906 shows an increase to 51%, where four cells have cell scores. The mini-map thus provides the user with an immediate indication of refinement, which can serve as an incentive for the user to continue moving around the environment and acquiring additional image data.
Example implementation

The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of the present disclosure, some non-limiting features will now be briefly discussed. The following paragraphs describe various exemplary implementations of the devices, systems, and methods described herein. A system of one or more computers performs a particular action or Can be configured to perform actions. One or more computer programs may be configured to perform a particular operation or action by including instructions which, when executed by a data processing apparatus, cause the apparatus to perform the action.

Example 1: A computerized method for storing software instructions executable by one or more hardware computer processors and a computing system to implement the computerized method. segmenting a digital map into a plurality of cells, each a defined area of the digital map and an actual area of the digital map; determining the cell in which the wearable headset is located; and associating the user with the determined cell that can be used to locate the user within the determined cell. determining a cell quality score, determined a cell saturation indicator indicating whether the user has been located within the determined cell for at least a threshold period of time, and combining the cell quality score and the cell saturation score; determining a cell score; and updating a user interface visible within the wearable headset to indicate the cell score for the determined cell.

Example 2: The cell quality score and cell saturation indicator are determined by a server remote to the wearable headset and transmitted to the wearable headset via an application programming interface (API) communication channel to Example 1. The described computerized method.

Example 3: The computerized method of Example 2, wherein the wearable headset initiates the (API) communication channel by transmitting credentials for direct access to map data from a remote server.

Example 4: The computerized method of Example 1, wherein the cell quality score is 0-1, with 0 indicating the lowest cell quality and 1 indicating the highest cell quality.

Example 5: The computerized method of Example 1, wherein if the cell saturation indicator is positive, the cell score is the sum of the cell quality score and 0.5, and the maximum cell score is 1.

Example 6: The computerized method of Example 1, wherein if the cell saturation indicator is negative, the cell score is the cell quality score.

Example 7: The computerized method of Example 1, wherein the user interface includes a minimap of a portion of the cell and its corresponding cell score.

Example 8: The computerized method of Example 7, wherein the minimap is associated with the location of the user input device such that the minimap moves in conjunction with the user input device.

Example 9: The computerized method of Example 8, wherein the minimap is displayed in front of the user input device.

Example 10: The computerized method of Example 7, further comprising rotating the minimap to maintain a user orientation in response to user movement.

Example 11: The computerized method of Example 1, wherein the user interface includes a map-quality overlay in which the cell score indicator overlays a corresponding portion of the real-world environment.

Example 12: The computerized method of Example 1, wherein the user interface includes at least one cell quality subscore determined based on images acquired from a particular viewing direction.

Example 13: The computerized method of Example 12, wherein the at least one cell quality subscore comprises a north viewing direction subscore, a south viewing direction subscore, a west viewing direction subscore, and an east viewing direction subscore.

Example 14: The computerized method of Example 13, wherein the cell quality subscore is indicated for the current cell.

Example 15: The computerized method of Example 14, wherein the cell quality subscores are indicated as colored areas around the cell score indicators.

Example 16: The computer of Example 7, wherein cell scores are indicated using colors in the user interface, lower cell scores being a first color and higher cell scores being a second color. method.

Example 17: The computerized method of Example 16, wherein the first color is red and the second color is green.

Example 18: The computerized method of Example 7, wherein the user interface indicates determined cells with user icons.

Example 19: The computerized method of Example 1, wherein the plurality of cells are in a grid pattern.

Example 20: A computerized method storing software instructions executable by one or more hardware computer processors and a computing system to perform the computerized method, one or a map associated with the environment of the wearable headset via an application programming interface implemented by a computing system having a non-transitory computer-readable storage device above it and configured to communicate with a map server; accessing the data and displaying, via the wearable headset, a minimap indicating the quality of the map data in each of a plurality of cells of the map; and via one or more sensors of the wearable headset. obtaining images of the environment as the user moves around the environment; determining updates to the quality of the map data based on the obtained images of the environment; updating the minimap; and indicating updates to the quality of the data.

Example 21: The computerized method of Example 20, further comprising transmitting at least some of the images of the environment via an application programming interface.

Example 22: The computerized method of example 20, wherein the map server is configured to update the reference map of the environment based on the image of the environment.

Example 23: The computerized method of Example 22, wherein the map server is further configured to receive images of the environment from one or more other users.

Example 24: The computerized method of Example 20, wherein map quality is determined based on cell quality scores and saturation indicators for individual cells of the map.

Example 25: The computerized method of Example 24, wherein the cell quality score and saturation indicator for a particular cell are determined while the user is positioned within the particular cell.

Example 26: The computerized method of Example 24, further comprising determining an overall map quality indicator based at least on cell quality scores for individual cells of the map.

As noted above, implementation of the illustrative examples provided above may include hardware, methods or processes, and/or computer software on a computer-accessible medium.
Additional Considerations

Each of the processes, methods, and algorithms described herein and/or depicted in the accompanying figures are configured to execute specific and specific computer instructions, one or more It may be embodied in code modules, executed by physical computing systems, hardware computer processors, application specific circuits, and/or electronic hardware, thereby being fully or partially automated. For example, a computing system can include a general purpose computer (eg, a server) or a special purpose computer, dedicated circuitry, etc. programmed with specific computer instructions. The code modules may be compiled and linked into an executable program, installed in a dynamically linked library, or written in an interpreted programming language. In some implementations, specific acts and methods may be performed by circuitry specific to a given function.

Moreover, implementation of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex to require either special-purpose hardware (utilizing appropriate specialized executable instructions) or single-use hardware. One or more physical computing devices may be required to perform the functionality, for example, due to the amount or complexity of the computations involved, or to provide results substantially in real time. For example, a motion picture or video contains many frames, each frame may have millions of pixels, and specifically programmed computer hardware can perform the desired image processing in a commercially reasonable amount of time. There is a need to process video data to serve a task or use.

Code modules or any type of data may be stored in hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical discs, volatile or nonvolatile storage devices, combinations of the same, and/or It may be stored on any type of non-transitory computer-readable medium such as physical computer storage, including equivalents. The methods and modules (or data) may also be transmitted as data signals (e.g., carrier waves or other analog or digital propagated signals) generated on various computer-readable transmission media, including wireless-based and wire/cable-based media. part) and may take various forms (eg, as part of a single or multiplexed analog signal, or as a plurality of discrete digital packets or frames). The results of any disclosed process or process step can be persistently or otherwise stored in any type of non-transitory, tangible computer storage device or communicated over a computer-readable transmission medium.

Any process, block, state, step, or functionality in a flow diagram described herein and/or depicted in an accompanying figure represents a specific function (e.g., logic or arithmetic) or It should be understood as potentially representing a code module, segment, or portion of code containing one or more executable instructions for implementing a step. Various processes, blocks, states, steps, or functionality may be combined, rearranged, added to, deleted from, modified, or modified from the illustrative embodiments provided herein. It can be modified from there otherwise. In some implementations, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and blocks, steps, or states associated therewith may be performed in any other suitable sequence, e.g., serially, in parallel. or in some other manner. Tasks or events may be added to or removed from the disclosed example implementations. Moreover, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be appreciated that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

The present processes, methods and systems may be implemented in a network (or distributed) computing environment. Networking environments include enterprise-wide computer networks, intranets, local area networks (LAN), wide area networks (WAN), personal area networks (PAN), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. . The network can be a wired or wireless network or any other type of communication network.

The system and method of the present disclosure each have several innovative aspects, none of which are solely responsible or required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. Various modifications of the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be adapted to other implementations without departing from the spirit or scope of this disclosure. can be applied. Accordingly, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the broadest scope consistent with the disclosure, principles and novel features disclosed herein. be.

Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Further, although features have been described above as acting in certain combinations and may also be originally claimed as such, one or more features from the claimed combination may in some cases be , may be deleted from the combination, and a claimed combination may cover subcombinations or variations of subcombinations. No single feature or group of features is required or mandatory in every implementation.

In particular, "can", "could", "might", "may", "e.g.", and equivalents , etc., as used herein generally refer to a feature, element, and /or are meant to convey that while the steps are included, other implementations do not include them. Thus, such conditional statements generally state that the features, elements and/or steps are required in any one or more implementations or that one or more implementations necessarily include logic, with or without author input or prompting, to determine whether these features, elements, and/or steps should be included or performed in any particular implementation; is not intended to imply The terms “comprising,” “including,” “having,” and equivalents are synonymous and are used generically in a non-limiting manner, Do not exclude subjective elements, features, acts, actions, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), thus, for example, when used to connect a list of elements, the term "or" Means one, some, or all of the elements in the list. Additionally, the articles "a," "an," and "the," as used in this application and the appended claims, unless specified otherwise, refer to "one or more" or "at least one." should be construed to mean "one".

As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" encompasses A, B, C, A and B, A and C, B and C, and A, B, and C is intended. Conjunctive sentences, such as the phrase "at least one of X, Y, and Z," generally refer to at least one of X, Y, or Z, unless specifically stated otherwise. understood differently in contexts such as those used to convey that a Thus, such a conjunction generally implies that an implementation requires that at least one of X, at least one of Y, and at least one of Z each be present. is not intended to be

Similarly, although operations may be depicted in the figures in a particular order, it is to be performed in the specific order in which such operations are shown or in a sequential order to achieve desirable results, or It should be appreciated that not all illustrated acts need be performed. Further, the drawings may graphically depict one or more exemplary processes in the form of flow charts. However, other acts not depicted may be incorporated within the illustrative methods and processes illustrated schematically. For example, one or more additional acts may be performed before, after, concurrently with, or between any of the illustrated acts. Additionally, the operations may be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, as the described program components and systems generally operate in a single unit. can be integrated together in multiple software products or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims (20)

A computerized method performed by a computing system, said computing system having one or more hardware computer processors and one or more non-transitory computer readable storage devices. said one or more non-transitory computer readable storage devices storing software instructions executable by said computing system to perform said computerized method; The modified method is
segmenting a digital map into a plurality of cells, each of said cells being associated with a defined area of said digital map and a corresponding area of a real-world environment;
determining a cell in which the wearable headset is positioned;
determining a cell quality score associated with the determined cell that can be used to locate a user within the determined cell;
determining a cell saturation indicator, the cell saturation indicator indicating whether the user is located within the determined cell for at least a threshold period of time;
determining a cell score indicative of the cell quality score and cell saturation score;
updating a user interface visible within the wearable headset to indicate a cell score for the determined cell. 2. The cell quality score and cell saturation indicator of claim 1, wherein the cell quality score and the cell saturation indicator are determined by a server remote to the wearable headset and transmitted to the wearable headset via an application programming interface (API) communication channel. The described computerized method. 3. The computerized method of claim 2, wherein the wearable headset initiates the API communication channel by transmitting credentials for direct access to map data from the remote server. 2. The computerized method of claim 1, wherein the cell quality score is between 0 and 1, with 0 indicating the lowest cell quality and 1 indicating the highest cell quality. 2. The computerized method of claim 1, wherein the cell score is the sum of the cell quality score and 0.5 and the maximum cell score is 1 if the cell saturation indicator is positive. 2. The computerized method of claim 1, wherein the cell score is the cell quality score if the cell saturation indicator is negative. 2. The computerized method of claim 1, wherein the user interface includes a minimap of a portion of the cell and its corresponding cell score. 8. The computerized method of claim 7, wherein the minimap is associated with a user input device location such that the minimap moves in conjunction with the user input device. 9. The computerized method of claim 8, wherein the minimap is displayed in front of the user input device. 8. The computerized method of claim 7, further comprising rotating the minimap to maintain the user's orientation in response to the user's movement. 2. The computerized method of claim 1, wherein the user interface includes a map-quality overlay in which cell score indicators overlay corresponding portions of the real-world environment. 2. The computerized method of claim 1, wherein the user interface includes at least one cell quality subscore determined based on images acquired from a particular viewing direction. 13. The computerized method of claim 12, wherein the at least one cell quality subscore includes a north viewing direction subscore, a south viewing direction subscore, a west viewing direction subscore, and an east viewing direction subscore. 14. The computerized method of claim 13, wherein the cell quality subscore is indicated within the determined cell. 15. The computerized method of claim 14, wherein the cell quality subscores are indicated as colored areas around the cell score indicators. 8. The computerized device of claim 7, wherein the cell scores are indicated using colors in the user interface, lower cell scores being a first color and higher cell scores being a second color. Method. 17. The computerized method of claim 16, wherein said first color is red and said second color is green. 8. The computerized method of claim 7, wherein the user interface indicates the determined cells using user icons. 2. The computerized method of claim 1, wherein the plurality of cells are in a grid pattern. A computerized method performed by a computing system, said computing system having one or more hardware computer processors and one or more non-transitory computer readable storage devices. wherein said one or more non-transitory computer readable storage devices store software instructions executable by a computing system to perform said computerized method; The method was
accessing map data associated with the environment of the wearable headset via an application programming interface configured to communicate with a map server;
displaying, via the wearable headset, a minimap indicative of the quality of the map data in each of a plurality of cells of a map;
acquiring images of the environment via one or more sensors of the wearable headset as a user moves around the environment;
determining updates to the quality of the map data based on captured images of the environment;
and updating the minimap to indicate updates to the quality of the map data.

Images