TW202318180A - Systems and methods for communicating model uncertainty to users - Google Patents

Abstract

The disclosed computer-implemented method may include (1) receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user, (2) determining, based on the information, a level of uncertainty associated with the real-time output, (3) modulating at least one attribute of feedback based on the level of uncertainty, and (4) presenting the feedback to the user substantially contemporaneous with the real-time output of the recognition model. Various other methods, systems, and computer-readable media are also disclosed.

Description

Systems and methods for communicating model uncertainty to users

This application relates to systems and methods for communicating model uncertainty to users. Cross-references to related applications

This application claims the benefit of U.S. Provisional Application No. 63/234,823, filed August 19, 2021, and U.S. Non-Provisional Application No. 17/575,676, filed January 14, 2022, the The disclosure is incorporated herein by reference in its entirety.

Users interact with computing systems in a variety of ways to provide input to or otherwise control actions performed by the computing system. A user can interact directly with a computing system using a simple physical user input device that provides a reliable discrete output (eg, a key or button press event) to the computing system. These simple physical user input devices are able to independently provide the user with actionable feedback on the user's behavior. For example, a user may provide text input to a computing system using a physical keyboard whose keys provide tactile resistance to movement at the point at which it produces discrete output. A physical user input device capable of independently providing immediate feedback to a user may enable a user to confidently provide input to a computing system without verifying that each of the user inputs was correctly recognized and responded to by the computing system. For example, physical keyboards can enable many users to quickly and reliably provide character input to a computing system without seeing which keys they are pressing and/or without verifying that individual characters are correctly received and processed by the computing system.

As an alternative to simple physical user input devices, computing systems may use virtual user input devices capable of detecting and reacting to certain user actions, such as virtual keyboards and/or recognition models. In some examples, a recognition model, such as a gesture recognizer, may output a probability or likelihood that a user performed or is performing an action. To use probabilities to respond to behaviors, computing systems may typically use thresholds to discretize the probabilities to determine the onset or occurrence of a behavior. For example, the computing system may use a threshold probability of 0.6 to distinguish the occurrence of an action from the non-occurrence of an action. Where the probability of an action occurring is extremely high (eg, 0.9), the computing system can be relatively certain that the action will occur. Likewise, where the probability is extremely low (eg, 0.1), the computing system can be relatively certain that the action did not occur. However, computing systems may be less certain about the probability of being within a range around a threshold (eg, within the range of 0.5 to 0.7), which can lead to false positive and false negative detections of system responses to user actions and occasional sexual mismatch.

An embodiment of the present application relates to a computer-implemented method comprising: receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user; Immediately outputting an associated uncertainty level; modulating at least one attribute of feedback based on the uncertainty level; and presenting the feedback to the user substantially simultaneously with the immediate output of the recognition model.

Another embodiment of the present application relates to a system comprising: at least one physical processor; and physical memory comprising computer-executable instructions that, when executed by the at least one physical processor, use The at least one physical processor: receives information associated with an immediate output of a recognition model adapted to identify at least one behavior of a user; determines an uncertainty level associated with the immediate output based on the information ; modulating at least one attribute of feedback based on the uncertainty level; and presenting the feedback to the user substantially simultaneously with the immediate output of the recognition model.

Yet another embodiment of the present application relates to a non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, causing the computing device to: receive information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user; determine an uncertainty level associated with the real-time output based on the information; Modulating at least one attribute of feedback based on the uncertainty level; and presenting the feedback to the user substantially simultaneously with the immediate output of the recognition model.

Users interact with computing systems in a variety of ways to provide input to or otherwise control actions performed by the computing system. A user can interact directly with a computing system using a simple physical user input device that provides a reliable discrete output (eg, a key or button press event) to the computing system. These simple physical user input devices are able to independently provide the user with actionable feedback on the user's behavior. For example, a user may provide text input to a computing system using a physical keyboard whose keys provide tactile resistance to movement at the point at which it produces discrete output. A physical user input device capable of independently providing immediate feedback to a user may enable a user to confidently provide input to a computing system without verifying that each of the user inputs was correctly recognized and responded to by the computing system. For example, physical keyboards can enable many users to quickly and reliably provide character input to a computing system without seeing which keys they are pressing and/or without verifying that individual characters are correctly received and processed by the computing system.

As an alternative to simple physical user input devices, computing systems may use virtual user input devices capable of detecting and reacting to certain user actions, such as virtual keyboards and/or recognition models. In some examples, a recognition model, such as a gesture recognizer, may output a probability or likelihood that a user performed or is performing an action. To use probabilities to respond to behaviors, computing systems may typically use thresholds to discretize the probabilities to determine the onset or occurrence of a behavior. For example, the computing system may use a threshold probability of 0.6 to distinguish the occurrence of an action from the non-occurrence of an action. Where the probability of an action occurring is extremely high (eg, 0.9), the computing system can be relatively certain that the action will occur. Likewise, where the probability is extremely low (eg, 0.1), the computing system can be relatively certain that the action did not occur. However, computing systems may be less certain about the probability of being within a range around a threshold (eg, within the range of 0.5 to 0.7), which can lead to false positive and false negative detections of system responses to user actions and occasional sexual mismatch.

The present disclosure generally relates to the use of various forms of actionable real-time feedback to communicate model uncertainty and related information to users. In some embodiments, the systems and methods described herein can use real-time haptic feedback (e.g., modulated vibrations) and/or other types of non-visual feedback to convey and recognize models (e.g., adapted to detect point And select the level of uncertainty associated with the output of the behavioral identification model). Embodiments of the present disclosure may enable a user to instantly understand when the computing system is most likely to respond correctly to the user's actions, which may allow the user to reliably provide input to the computing system with little or no hesitation system. Embodiments of the present disclosure may also enable a user to instantly understand when the computing system is most likely to react incorrectly to user behavior, which may allow the user to modify certain behaviors to be better recognized by the computing system. In some embodiments, the feedback presented to the user can indicate how the user can modify his behavior to be better recognized by the computing system. By communicating uncertainty information to the user via uncertainty modulation feedback, embodiments of the present disclosure may enable the user to better understand how the computing system will interpret the user's actions, which may reduce the risk of Counting the number of recognition errors made by the system and resulting in improved user interaction experience and performance.

Features from any of the specific examples described herein can be used in combination with each other according to the general principles described herein. These and other specific examples, features and advantages will be more fully understood upon reading the following detailed description in conjunction with the accompanying drawings and claims.

A detailed description of an exemplary system for communicating model uncertainty to a user is provided below with reference to FIG. 1 . The discussion corresponding to FIGS. 2-12 will provide a detailed description of the corresponding methods and data flows. Finally, with reference to FIGS. 13-21 , detailed descriptions of various augmented reality systems and components that may implement embodiments of the present disclosure will be provided below.

1 is a block diagram of an example system 100 for communicating model uncertainty to a user. As depicted in this figure, example system 100 may include one or more modules 102 for performing one or more tasks. As will be explained in more detail below, modules 102 may include a receiving module 104 that receives information associated with the immediate output of the recognition model. The example system 100 may also include a determination module 106 that determines an uncertainty level associated with the immediate output of the identification model based on information received by the receiving module 104 . The example system 100 may also include a modulation module 108 that modulates various properties of the feedback based on the uncertainty level. Additionally, the example system 100 may include a presentation module 110 that presents feedback to the user substantially simultaneously with the immediate output of the recognition model. In some embodiments, the example system 100 may also include a user input module 112 that executes or refrains from executing reactive user input operations in response to immediate outputs of the recognition model (eg, recognition events).

As further shown in FIG. 1 , example system 100 may also include one or more memory devices, such as memory 120 . Memory 120 may include or represent any type or form of volatile or non-volatile storage device or media capable of storing data and/or computer-readable instructions. In one example, the memory 120 can store, load and/or maintain one or more of the modules 102 . Examples of memory 120 include, but are not limited to, random access memory (RAM), read only memory (ROM), flash memory, hard disk drives (HDD), solid state drives (SSD), optical drives, flash drives, Take memory, a variation or combination of one or more of the aforementioned memories, or any other suitable storage memory.

As further shown in FIG. 1 , example system 100 may also include one or more physical processors, such as physical processor 130 . Physical processor 130 may include or represent any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions. In one example, physical processor 130 may access and/or modify one or more of modules 102 stored in memory 120 . Additionally or alternatively, entity processor 130 may execute one or more of modules 102 to facilitate communicating model uncertainty to a user. Examples of physical processor 130 include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs) implementing soft-core processors, application specific integrated circuits (ASICs), A portion of one or more of the aforementioned physical processors, a variation or combination of one or more of the aforementioned physical processors, or any other suitable physical processor.

As further shown in FIG. 1 , example system 100 may also include one or more recognition models, such as recognition model 140 . Recognition model 140 may represent or include any machine learning model, algorithm, heuristic, data, or combination thereof that can be used to identify user behavior (e.g., actions, cognitive states, intentions, etc.) and/or Recognize, detect, calculate, estimate, predict, flag, infer, and/or react to effects such as device, headset, or controller movement. Examples of recognition models 140 include, but are not limited to: pinch detection models (e.g., pinch detection model 142 ) adapted to recognize and/or respond to pinch gestures, adapted to recognize hand state and/or hand movement Manual tracking models that recognize and/or respond to (e.g., manual tracking model 144), models adapted to recognize and/or respond to user intent for interaction (e.g., intent interaction model 146), Models adapted to recognize and/or respond to transitions between cognitive states of the user (e.g., cognitive state model 148), gesture recognition models, eye tracking models, speech recognition models, body sensing models, body tracking Models, physiological state models adapted to recognize and/or respond to the user's physiological state, psychological state models adapted to recognize and/or respond to the user's psychological state, predictive of the user's A model intended for and responsive to encoding, a model adapted to perform inside-out tracking of a computing device, a portion of one or more of the foregoing models, a variation of one or more of the foregoing models, or combination, or any other suitable identification model. In some examples, the recognition model 140 may include, but is not limited to, decision trees (e.g., reinforced decision trees), neural networks (e.g., deep convolutional neural networks), deep learning models, support vector machines, linear classifiers, Non-linear classifiers, perceptrons, naive Bayesian classifiers, any other machine learning or classification techniques or algorithms, or any combination thereof.

In some examples, the systems described herein may use one or more of the recognition models 140 to recognize a user's pointing gestures, selections, or graphical elements displayed to the user via the system 100 (e.g., a graphical user interface). elements or elements rendered from a virtual reality or augmented reality environment) to recognize and/or respond to interactions. For example, the system described herein may use a pinch detection model 142 to detect when a user intends to make a selection, and may use a hand tracking model 144 to determine the graphical element the user is most likely to attempt to select.

In some examples, system 100 may include one or more sensors 150 for collecting real-time measurements indicative of user behavior. The recognition model 140 can use information derived from measurements collected from the sensors 150 to recognize user behavior in real time. Examples of sensor 150 include, but are not limited to, touch sensors, image sensors, proximity sensors, biometric sensors, inertial measurement units, biometric sensors, heart rate sensors, saturated oxygen sensors sensor, neuromuscular sensor, altimeter sensor, temperature sensor, bioimpedance sensor, pedometer sensor, optical sensor, sweat sensor, one or more of the foregoing sensors variations or combinations thereof, or any other type or form of sensing hardware or software.

FIG. 2 is a diagram of an exemplary data flow 200 for communicating uncertainty levels associated with identification model 140 to a user using one or more of the components of system 100 . In this example, one or more of the recognition models 140 may take input from one or more of the sensors 150 to generate one or more immediate outputs 202 indicative of and/or predicting one or more user actions. (for example, identifying events). As shown, the receiving module 104 and the user input module 112 can monitor and/or react to the real-time output 202 . In addition to or as an alternative to monitoring real-time output 202, receiving module 104 may also monitor some or all inputs to recognition model 140 (e.g., measurements from sensors 150), which may be used by recognition model 140 to generate real-time Output 202. In some examples, real-time output 202 may represent a time series (eg, a time series of probabilities of one or more user actions, such as probabilities 302 depicted in FIG. 3 ).

In some examples, recognition model 140 may be adapted to recognize, detect, calculate, estimate, predict, flag, infer, and/or react to one or more user behaviors, and immediate output 202 may represent or include an indication Any information about the occurrence or non-occurrence of one or more user actions. In some examples, immediate output 202 may include or represent a probability or likelihood that the user is performing a particular action (eg, a pinch or point gesture). Additionally or alternatively, recognition model 140 may be adapted to recognize a number of user behaviors, and immediate output 202 may include or represent a label or category of behavior that the user is likely to be performing and/or a trust associated with that label or category. level or score. In some examples, recognition model 140 may be adapted to estimate, predict, and/or infer various attributes of one or more user behaviors, and immediate output 202 may represent or include one or more estimated attributes of behavior (e.g., predicted position, orientation, or state) and/or a confidence level or score associated with the estimated property.

In some examples, user input module 112 may be adapted to perform user input operations and/or otherwise respond to associated user actions indicated by immediate output 202 . Examples of user input operations that may be performed by user input module 112 in response to immediate output 202 include, but are not limited to: updating the cursor's position based on user action indicated by immediate output 202 (e.g., a pointing gesture); Make selections based on user actions indicated by real-time output 202 (e.g., a pinch gesture); update the state of a virtual character or environment to reflect user actions indicated by real-time output 202; User input associated with indicated user action; process user input associated with user action indicated by immediate output 202; store user input associated with user action indicated by immediate output 202; trigger and an action or function associated with a user action indicated by a user input associated with the immediate output 202; passing the user input associated with the user action indicated by the instant output 202 to another function or application for further processing; and/or transmitting user input associated with user actions indicated by real-time output 202 to another computing system for further processing.

As shown in FIG. 2 , receiving module 104 may monitor real-time output 202 and/or any information associated with recognition model 140 and/or real-time output 202 , which may directly and/or indirectly indicate incorrect information about real-time output 202 . Interpretation of deterministic and/or immediate output 202 . For example, receiving module 104 may receive information associated with sensor 150 that may indicate the reliability of measurements by sensor 150 and/or information associated with immediate output 202 (eg, probability or level of trust). Decision module 106 may use the information gathered by receiving module 104 to determine or calculate one or more uncertainty levels 204 associated with immediate output 202 . The modulation module 108 may then use the uncertainty level 204 to modulate at least one attribute of the appropriate feedback to be presented by the presentation module 110 .

As mentioned above, the immediate output 202 of the recognition model 140 may include a time series indicating the probability of occurrence and/or non-occurrence of certain user actions. FIG. 3 is a graphical representation of an exemplary time series of probabilities 302 . Each of the probabilities 302 may represent a probability that the user is performing an action (eg, a pinch gesture) at the corresponding moment. In this example, the disclosed system can use a threshold value 304 (eg, 0.6) to distinguish between behavior occurrence and behavior non-occurrence. For example, the disclosed system may consider a probability greater than a threshold value 304 to be an action occurrence, and a probability less than the threshold value 304 to be an action non-occurrence. Where the behavior occurs with a high probability (eg, probability 306 ), the disclosed system can be relatively certain that the behavior occurred and can respond accordingly. Likewise, where the probability is extremely low (eg, probability 308 ), the disclosed system can be relatively certain that the action did not occur and can react accordingly. However, the disclosed system may be less certain of probabilities (eg, probabilities 310 and 312 ) within a range around threshold 304 and may react in ways that conflict with the user's intent. For example, the disclosed system can react to the chance 310 as if the user was performing the action, even though the user did not actually perform the action. Similarly, the disclosed system can react to the chance 312 as if the user performed the behavior even though the user did not actually perform the behavior. As will be explained in more detail below, the disclosed system can use various forms of feedback to communicate to the user a level of uncertainty associated with the probability 302, which can allow the user to adapt subsequent behavior such that the associated behavior The level of uncertainty in the future is reduced.

4 is a flowchart of an exemplary computer-implemented method 400 for communicating model uncertainty to a user. The steps shown in FIG. 4 may be performed by any suitable computer-executable code and/or computing system, including the system shown in FIG. 1 . In one example, each of the steps shown in FIG. 4 may represent an algorithm whose structure includes and/or is represented by a plurality of sub-steps, examples of which are provided in more detail below . In some examples, the steps of computer-implemented method 400 may be performed repeatedly. For example, the steps of computer-implemented method 400 may be performed once for each of the immediate outputs of the recognition model.

As shown in FIG. 4, at step 410, one or more of the systems described herein may receive information associated with the immediate output of a recognition model adapted to recognize at least one behavior of a user. For example, the receiving module 104 can receive some or all of the information associated with the real-time output 202, as described in connection with FIG. 2 .

The systems described herein can receive and/or monitor various types of information that can indicate uncertainties associated with the immediate output of an identification model. In one example, the identification model can output an indication of uncertainty (eg, probability, certainty level, reliability score, etc.) associated with its output, and the receiving module 104 can monitor this information in real time. In some examples, certain properties of the input to the identification model (eg, signal-to-noise ratio) may cause or affect uncertainties associated with the immediate output of the identification model, and the receiving module 104 may monitor such properties in real time. In at least one example, the disclosed system can use one or more thresholds to discretize and/or interpret the output of the identification model. For example, user input module 112 may use threshold value 304 to discretize and/or interpret probability 302 . Threshold values can influence and/or indicate a level of uncertainty associated with the immediate output of an identification model. For at least this reason, receiving module 104 may identify any thresholds for discretizing and/or interpreting the output of the identification model.

At step 420 , one or more of the systems described herein may determine a level of uncertainty associated with the immediate output based on the information received at step 410 . For example, determination module 106 may determine uncertainty level 204 based on information received by receiving module 104 , as shown in FIG. 2 .

The system described herein can determine the level of uncertainty associated with the immediate output of the identification model in a variety of ways. In some examples, the identification model can output a measure of uncertainty, and the disclosed system can determine the level of uncertainty associated with the immediate output from the identification model itself. For example, when the immediate output includes a probability, a certainty level, or a confidence score, the disclosed system can determine an uncertainty level based on the probability, the certainty level, or the confidence score. In some examples, if a probability indicates the likelihood that a user action is occurring, the disclosed system can generate an uncertainty level for the probability that is inversely proportional to the probability (i.e., a high A probability may be associated with a low uncertainty level, and a low probability may be associated with a high uncertainty level). Similarly, the disclosed system can generate an uncertainty level for an immediate output of a recognition model if the certainty level or confidence score associated with the immediate output indicates a likelihood that the immediate output is accurate, the uncertainty The certainty level is inversely proportional to the certainty level or confidence score (that is, a high certainty level or confidence score can be correlated with a low uncertainty level, and a low certainty level or confidence score can be correlated with associated with high uncertainty levels).

（1） In some examples, probabilities can reflect the likelihood that a user action is occurring or not occurring, and the disclosed systems can use one or more threshold probabilities (e.g., threshold 304 in FIG. is interpreted as the probability that the user action occurred and will be interpreted as the probability that the user action did not occur. In such examples, the disclosed systems can use thresholds to determine levels of uncertainty associated with probabilities. For example, the disclosed system can generate an uncertainty level for the probability that is inversely proportional to the distance between the probability and the threshold (i.e., the probability furthest from the threshold can be equal to A low uncertainty level is associated, and the probability closest to the threshold can be associated with a high uncertainty level). In at least one example, the disclosed system can determine an uncertainty level for probability, as shown in equation (1):

(1)

where Uncertainty is the level of uncertainty in the range between 0 (indicating low uncertainty) and 1 (indicating high uncertainty), where Probability is a single probability, and where Threshold is used to Probability is interpreted as the threshold for the occurrence or non-occurrence of an action. FIG. 5 depicts an exemplary uncertainty level 502 that can be derived using equation (1) and threshold 304 from FIG. 3 . As can be seen, the level of uncertainty associated with each probability may be inversely proportional to the distance between the probability and the threshold 304 . In this example, the probability associated with the highest uncertainty level is the probability closest to the threshold value 304 and the probability associated with the lowest uncertainty level is the probability furthest from the threshold value 304 .

In some instances, certain characteristics of the input to the identification model (eg, signal-to-noise ratio) can cause or affect uncertainties associated with the immediate output of the identification model. In such examples, the disclosed systems can generate or adjust the uncertainty level of the immediate output based at least in part on these characteristics and their contributions to the uncertainty.

At step 430 , one or more of the systems described herein may modulate at least one property of the feedback based on the uncertainty level determined at step 420 . For example, modulation module 108 may modulate at least one property of feedback 206 based on uncertainty level 204 associated with immediate output 202 .

The systems described herein can use various forms of sensory feedback to indicate and/or communicate the level of uncertainty associated with the immediate output of the recognition model and/or attempt to cause the user to improve the level of uncertainty associated with the immediate output of the recognition model. level of uncertainty. In one example, the disclosed system can use any suitable type or form of haptic feedback, such as vibration, force, pressure, traction, texture, and/or temperature, to convey information about the level of uncertainty. In some examples, the disclosed systems can use any suitable type or form of non-visual feedback, such as audio feedback, to convey information about the level of uncertainty. In at least one example, the disclosed system can use a form of visual feedback to convey information about the level of uncertainty.

The system described herein can modulate various properties of the feedback to convey information about the level of uncertainty. For example, the disclosed system can adjust feedback amplitude, feedback frequency, feedback duration, feedback type, feedback spatialization, feedback location, feedback intensity, feedback effect, feedback intensity, changes in one or more of the foregoing feedback Or combination, or any other perceivable attribute of the feedback.

The system described herein can modulate the properties of the feedback in a variety of ways based on the level of uncertainty associated with the immediate output of the identification model. In some examples, the disclosed systems can modulate the attributes of the feedback by setting, tuning, tuning, or changing the attributes and/or the perceptibility of the attributes to substantially reflect or otherwise communicate the immediate output associated with the recognition model and/or reflect or otherwise communicate changes in the uncertainty level associated with the immediate output of the identification model.

In some examples, the disclosed systems can communicate the level of uncertainty using secondary, tertiary, quaternary, or quintuple feedback channels. For example, where the visual feedback channel is primarily used to provide feedback to the user, the disclosed system can use an auditory feedback channel or a tactile feedback channel to communicate the level of uncertainty. In some examples, the disclosed systems can select feedback that can be subtle and/or less diffuse than other forms of primary feedback communicated to the user via the primary feedback channel.

In some examples, the disclosed systems can associate one or more types or forms of feedback with one or more types or forms of model uncertainty. For example, the disclosed systems can associate a single type or form of feedback (eg, a single type of vibration or a form of vibration) with uncertainty about the occurrence and non-occurrence of an identified user action. In such examples, the disclosed systems can modulate one or more properties of the feedback to convey the level of uncertainty. Alternatively, the disclosed system can associate a first type or form of feedback with uncertainty about the occurrence of the identified behavior, and can associate a second type or form of feedback with the uncertainty about the non-occurrence of the behavior Associated. In these examples, the disclosed systems may modulate one or more properties of a first type of feedback to convey a level of uncertainty associated with the occurrence of behavior, and/or may modulate one or more of a second type of feedback. Multiple attributes to convey the level of uncertainty associated with the non-occurrence of the behavior.

In some examples, the disclosed system can provide instructions to the user on how to reduce the uncertainty associated with the immediate output of the recognition model before presenting any feedback to the user. Where there are multiple methods by which a user can reduce the level of uncertainty and/or multiple reasons for a low level of uncertainty, the disclosed system can use commands and different types of feedback for each method Correlation so that the user can quickly know the appropriate method to apply when feedback is received.

In some examples, the disclosed system can modulate one or more properties of the feedback to deliver a method for reducing the level of uncertainty in the immediate output of the identification model. For example, if the user's hand must remain within the view of the head-mounted camera system to be tracked, as soon as the user's hand begins to deviate from the view of the head-mounted camera system to remind the user to move their hand The disclosed system can modulate one or more properties of the haptic feedback presented via the wrist-worn device by returning the portion to a more ideal position for tracking. In at least one example, the disclosed system can adjust the position of the haptic feedback to indicate the direction in which the user should move their hand to return their hand to a more ideal position for tracking.

At step 440, one or more of the systems described herein may present feedback to the user substantially simultaneously with the immediate output of the recognition model. For example, the presentation module 110 can present the feedback 206 to the user substantially simultaneously with one of the real-time outputs 202 .

The system described herein can present feedback to the user in a variety of ways simultaneously with the immediate output of the recognition model. In some examples, the disclosed systems can present feedback to the user substantially simultaneously with the execution of user input operations in response to the immediate output of the recognition model. If the immediate output of the recognition model indicates that the user input operation will not be performed, the disclosed system can present feedback to the user substantially simultaneously with the determination that the immediate output indicates that the user input operation will not be performed.

FIGS. 6-11 illustrate various exemplary ways in which the disclosed system may modulate the magnitude of the feedback to convey a level of uncertainty 502 that may be correlated with an indication of the occurrence and/or non-occurrence of a user action. Probability is related. These examples on amplitude modulation and probability are provided for ease of understanding. It should be noted that the disclosed system can modulate any of the feedback properties described above in the same or similar manner to convey the level of uncertainty associated with any type or form of identification model output.

FIG. 6 illustrates an exemplary proportional relationship between an exemplary amplitude level 602 and an uncertainty level 502 . In this example, the disclosed system ranges from outputting a low magnitude feedback in response to a probability associated with a low uncertainty level (e.g., the probability furthest away from the threshold 304) to a response to a high uncertainty level. The probability associated with the sex level (eg, the probability of being closest to the threshold value 304 ) outputs a high magnitude feedback. In this example, the user may receive the most discriminative feedback when the disclosed system is encountering the most uncertain identification model output.

FIG. 7 illustrates an exemplary relationship between an exemplary amplitude level 702 and uncertainty level 502 . In this example, the disclosed system can modulate the amplitude level 702 differently depending on whether chance is to be interpreted as the occurrence or non-occurrence of user action. The scope of the disclosed system can be self-output low in response to probabilities associated with low uncertainty levels (e.g., probabilities furthest away from threshold 304) when probabilities can be interpreted as user actions not occurring. The magnitude feedback is to output a high magnitude feedback in response to the probability associated with the high uncertainty level (eg, the probability of being closest to the threshold 304 ). On the other hand, when chance can be interpreted as a user action occurring, the disclosed system can output high magnitude feedback regardless of the level of uncertainty associated with the chance. In this example, the user may receive the most discernible feedback when the user is most likely to attempt to perform the user action.

FIGS. 8-11 illustrate various exemplary ways in which the disclosed system may modulate the magnitude of the feedback based on whether the encountered uncertainty level is above or below an uncertainty threshold 802 . FIG. 9 illustrates an exemplary relationship between an exemplary amplitude level 902 and an uncertainty level 502 . In this example, the disclosed system may modulate the amplitude level 902 differently depending on whether a probability is associated with an uncertainty level above or below the threshold value 802 . When the encountered probability is associated with an uncertainty level below a threshold 802, the disclosed system may output no feedback. On the other hand, when the encountered probability is associated with a level of uncertainty above the threshold 802, the disclosed system can output a high magnitude feedback regardless of the level of uncertainty associated with the probability. In this example, the user can receive discriminative feedback when the disclosed system is encountering the most uncertain identification model output.

FIG. 10 illustrates an exemplary relationship between an exemplary amplitude level 1002 and an uncertainty level 502 . In this example, the disclosed system may modulate the amplitude level differently depending on whether the probability is above the threshold 304 and/or whether the probability is associated with an uncertainty level above or below the threshold 802 Standard 1002. When the encountered probability is below threshold 304 and associated with an uncertainty level below threshold 802 , the disclosed system may output no feedback. On the other hand, the range of the disclosed system may be responsive to lower Outputting a lower magnitude feedback in response to a higher probability to output a higher magnitude feedback in response to a higher probability. In this example, the user may receive the most discernible feedback when the user is most likely to attempt to perform the user action.

FIG. 11 illustrates an exemplary proportional relationship between an exemplary amplitude level 1102 and an uncertainty level 502 . In this example, the disclosed system may modulate the amplitude level 1102 differently depending on whether a probability is associated with an uncertainty level above or below the threshold value 802 . When the encountered probability is associated with an uncertainty level below a threshold 802, the disclosed system may output no feedback. On the other hand, when the encountered probabilities are associated with uncertainty levels above the threshold 802, the range of the disclosed system may be responsive to probabilities associated with low uncertainty levels (e.g. , the probability furthest from the threshold value 304) and output a low magnitude feedback to output a high magnitude feedback in response to the probability associated with a high uncertainty level (eg, the probability closest to the threshold value 304). In this example, the user may receive the most discriminative feedback when the disclosed system is encountering the most uncertain identification model output, but the user may receive the most discriminative feedback when the disclosed system is encountering the most certain identification model output. Don't be bothered by feedback.

12 is a flowchart of an exemplary computer-implemented method 1200 for communicating probability-based uncertainty levels to a user while responding to user behavior. The steps shown in FIG. 12 may be performed by any suitable computer-executable code and/or computing system, including the system depicted in FIG. 1 . In one example, each of the steps shown in FIG. 12 may represent an algorithm whose structure includes and/or is represented by a plurality of sub-steps, examples of which are provided in more detail below. . Some of the steps shown in FIG. 12 are similar to the steps shown in FIG. 4 , therefore, the discussion of the steps shown in FIG. 4 may also apply to the steps shown in FIG. 12 .

As shown in FIG. 12, at step 1210, one or more of the systems described herein may receive from the recognition model the probability of occurrence of a user action. For example, the receiving module 104 may receive one of the probabilities 306-312 shown in FIG. 3 . At step 1220 , one or more of the systems described herein may then determine an uncertainty level based on the probabilities received at step 1210 . In one example, the decision module 106 may use equation (1) as described above to determine the uncertainty level of the probability received at step 1210 . For example, decision module 106 may determine a relatively lower uncertainty level for probabilities 306 and 308 and/or a relatively higher uncertainty level for probabilities 310 and 312 .

At step 1230 , one or more of the systems described herein may then modulate at least one property of the feedback based on the uncertainty level determined at step 1220 . For example, modulation module 108 may modulate at least one property of the haptic feedback to convey the level of uncertainty determined at step 1220 (e.g., relatively low uncertainty for the probability that lower magnitude vibrations may convey 306 and 308 sex level, or the relatively high uncertainty level of the transmittable probabilities 310 and 312 of higher amplitude vibrations). At step 1240, one or more of the systems described herein may determine whether the probability received at step 1210 is above a predetermined threshold. For example, the determination module 106 can determine whether the probability of receiving at step 1210 is above or below the threshold 304 . If the probability of receipt at step 1210 is determined to be above a threshold (e.g., as likely for random rates 306 and 310 to occur), execution of method 1200 may continue at step 1250, where the disclosed system may respond, As if the behavior was detected by performing a user input action associated with that behavior. On the other hand, if at step 1210 the probability of receipt is determined to be below the threshold (e.g., as likely random rates 308 and 312 occur), execution of method 1200 may continue at step 1260, where the disclosed system may React as if the associated behavior was not detected by avoiding user input associated with the behavior. Finally, at step 1270 , one or more of the systems described herein may present the feedback modulated at step 1230 to the user substantially simultaneously with the execution of step 1250 or step 1260 .

As described above, the disclosed system can use modulated haptics and/or other forms of user feedback to communicate the level of uncertainty of possible changes in predictive models that predict, among other things, user position, controller Model of location and/or user input. In some examples, the pinch detection model may output a value indicating the probability that the user is performing a pinch gesture. The disclosed system can react as if the user is performing a pinch gesture when the output probability is in the upper range, and/or can react as if the user is not performing a pinch gesture when the output probability is in the lower range . When the output probability is in the middle range, the disclosed system may be less certain or uncertain whether the user is performing a pinch gesture. In at least these uncertain situations, the disclosed system can present to the user haptic feedback that has been modulated to indicate the uncertainty/level of uncertainty of the system. In another example, the hand tracking model can estimate the location of the user's hands, and can output a value indicative of the certainty or uncertainty of any estimated hand location. As the user moves their hand, the disclosed system can communicate the certainty or uncertainty of the estimated hand position to the user through modulated haptic feedback. As explained above, by providing transparency about model uncertainty, the disclosed systems can reduce user frustration and/or can allow users to adapt their behavior in a manner that results in improved model predictions and user experience.

Instance Concrete example

Example 1: A computer-implemented method for communicating model uncertainty may include: (1) receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user ; (2) determining a level of uncertainty associated with the immediate output based on the information; (3) modulating at least one property of the feedback based on the level of uncertainty; and (4) in the identification model Presenting the feedback to the user substantially simultaneously with the immediate output.

Embodiment 2: The computer-implemented method of Embodiment 1, wherein: (1) the information associated with the real-time output of the recognition model includes a probability that the user performs the behavior; (2) the computer-implemented method further Including at least one of the following: (a) performing a user input operation when the probability of the user performing the action is above a predetermined threshold; and/or (b) performing a user input operation when the user performs the action Avoid performing the user input operation when the probability is below the predetermined threshold; and (3) the uncertainty level associated with the immediate output may be based on a value between the probability and the predetermined threshold judged by distance. In some examples, the uncertainty level can be inversely proportional to the distance.

Embodiment 3: The computer-implemented method of any one of Embodiments 1-2, wherein: (1) the information associated with the real-time output of the recognition model includes a probability that the user performs the behavior; and ( 2) The attribute of the feedback can be tuned to have a perceptibility level proportional to the probability of the user performing the behavior.

Embodiment 4: The computer-implemented method of any one of Embodiments 1 to 3, wherein the property of the feedback is modulated to have a perceptibility level proportional to the uncertainty level, the non- A certainty level is associated with this immediate output.

Embodiment 5: The computer-implemented method of any one of Embodiments 1 to 4, wherein: (1) the recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; (2) The information includes a probability that the user performs the pinch gesture; (3) the computer-implemented method further includes performing a user input operation when the probability that the user performs the pinch gesture is above a predetermined threshold; and (4) When the probability of the user performing the pinch gesture is higher than the predetermined threshold value, the feedback may be presented to the user while performing the user input operation.

Embodiment 6: The computer-implemented method of any one of Embodiments 1 to 5, wherein: (1) the recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; (2) The information includes a probability that the user performs the pinch gesture; (3) the computer-implemented method further includes refraining from performing a user input operation when the probability that the user performs the pinch gesture is below a predetermined threshold; And (4) while determining that the probability of the user performing the pinch gesture is lower than the predetermined threshold, the feedback may be presented to the user.

Embodiment 7: The computer-implemented method of any one of Embodiments 1-6, wherein: (1) the recognition model may include a manual tracking model adapted to output (a) for the user position or orientation information of one or more portions of the hand, and (b) a confidence level for the position or orientation information; and (2) the uncertainty level associated with the real-time output can be Judgment based on this trust level. In some examples, the uncertainty level can be inversely proportional to the confidence level.

Embodiment 8: The computer-implemented method of any one of Embodiments 1-7, wherein: (1) the feedback can be haptic feedback; and (2) at least one attribute of the haptic feedback can be based on the the level of uncertainty.

Embodiment 9: The computer-implemented method of any one of Embodiments 1-8, wherein: (1) the feedback can be a vibration; and (2) at least one attribute of the vibration can be based on an the level of uncertainty.

Embodiment 10: The computer-implemented method of any one of Embodiments 1 to 9, wherein modulating the property of the feedback based on the uncertainty level comprises modulating one or more of: (1) the an amplitude of the feedback to communicate the level of uncertainty to the user; (2) a frequency of the feedback to communicate the level of uncertainty to the user; (3) a duration of the feedback time to communicate the level of uncertainty to the user; (4) a type of feedback to communicate the level of uncertainty to the user; and/or (5) one of the feedback spatialized to communicate the uncertainty level to the user.

Embodiment 11: The computer-implemented method of any one of Embodiments 1-10, wherein the feedback indicates a means for reducing the uncertainty level of the immediate output.

Embodiment 12: The computer-implemented method of any one of Embodiments 1 to 11, further comprising: (1) receiving additional information associated with an additional real-time output of the recognition model; (2) based on the additional information, determining an additional uncertainty level associated with the additional immediate output; and (3) modulating the feedback presented to the user based on the additional uncertainty level.

Embodiment 13: A system may include: (1) at least one physical processor; and (2) physical memory including computer-executable instructions that, when executed by the physical processor, cause the physical The processor: (a) receives information associated with an immediate output of a recognition model adapted to identify at least one behavior of a user; (b) determines based on the information an an uncertainty level; (c) modulating at least one attribute of feedback based on the uncertainty level; and (d) presenting the feedback to the user substantially simultaneously with the immediate output of the recognition model .

Embodiment 14: The system of Embodiment 13, wherein: (1) the information associated with the real-time output of the recognition model includes a probability that the user performs the action; (2) the computer-executable instructions are in When executed by the physical processor, the physical processor further causes the physical processor to: (a) perform a user input operation when the probability of the user performing the action is above a predetermined threshold; and/or (b) when the refraining from performing the user input operation when the probability of the user performing the action is below the predetermined threshold; and (3) the uncertainty level associated with the immediate output may be based on It is judged by a distance between the limit values. In some examples, the uncertainty level can be inversely proportional to the distance.

Embodiment 15: The system of any of Embodiments 13 to 14, wherein: (1) the information associated with the real-time output of the recognition model includes a probability that the user performs the behavior; and (2) The attribute of the feedback can be tuned to have a perceptibility level proportional to the probability of the user performing the behavior.

Embodiment 16: The system of any of Embodiments 13 to 15, wherein the property of the feedback is modulated to have a level of perceptibility proportional to the level of uncertainty, the uncertainty A level is associated with the immediate output.

Embodiment 17: The system of any of Embodiments 13-16, wherein: (1) the recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; (2) the information Including the probability that the user performs the pinch gesture; (3) When the computer-executable instructions are executed by the physical processor, the physical processor further makes the probability that the user performs the pinch gesture higher than a predetermined and (4) when the probability that the user performs the pinch gesture is higher than the predetermined threshold, the feedback may be presented while performing the user input operation to the user.

Embodiment 18: The system of any of Embodiments 13 to 17, wherein: (1) the recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; (2) the information Including the probability that the user performs the pinch gesture; (3) the computer-executable instructions, when executed by the physical processor, further cause the physical processor to lower the probability of the user performing the pinch gesture below a predetermined and (4) while determining that the probability of the user performing the pinch gesture is lower than the predetermined threshold, the feedback may be presented to the user.

Embodiment 19: The system of any of Embodiments 13 to 18, wherein: (1) the recognition model may include a manual tracking model adapted to output (a) for the user's hand position or orientation information for one or more portions of the portion, and (b) a level of confidence in the position or orientation information; and (2) the level of uncertainty associated with the immediate output may be based on the Judgment based on trust level. In some examples, the uncertainty level can be inversely proportional to the confidence level.

Embodiment 20: A non-transitory computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: (1) receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user; (2) determining an uncertainty level associated with the real-time output based on the information ; (3) modulating at least one attribute of feedback based on the uncertainty level; and (4) presenting the feedback to the user substantially simultaneously with the immediate output of the recognition model.

Embodiments of the present disclosure may include or be implemented in combination with various types of artificial reality systems. Artificial reality is a form of reality that has been modified in some way before being presented to the user, which may include, for example, virtual reality, augmented reality, mixed reality, hybrid reality, or some combination and/or derivative thereof things. Artificial reality content may include fully computer-generated content or computer-generated content combined with captured (eg, real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereoscopic video that creates a three-dimensional (3D) effect on the viewer ). Additionally, in some embodiments, AR can also be used with applications that are used, for example, to generate content in AR and/or otherwise used in AR (e.g., to perform activities in AR) , products, accessories, services or a combination thereof.

The artificial reality system can be implemented in various form factors and configurations. Some artificial reality systems can be designed to work without a near-eye display (NED). Other artificial reality systems may include NEDs that also provide visibility into the real world (such as augmented reality system 1300 in FIG. 13 ) or visually immerse the user in an artificial reality (such as in FIG. 14 In the virtual reality system 1400). While some augmented reality devices may be self-contained systems, other augmented reality devices may communicate and/or coordinate with external devices to provide a user with an augmented reality experience. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by a user, devices worn by one or more other users, and/or any other suitable external system.

Turning to FIG. 13 , an augmented reality system 1300 may include a glasses device 1302 having a frame 1310 configured to hold a left display device 1315 (A) and a right display device 1315 (B) in front of a user's eyes. Left display device 1315(A) and right display device 1315(B) may function together or independently to present an image or series of images to the user. Although augmented reality system 1300 includes two displays, embodiments of the present disclosure can be implemented in augmented reality systems with a single NED or more than two NEDs.

In some embodiments, augmented reality system 1300 may include one or more sensors, such as sensor 1340 . Sensor 1340 may generate measurement signals in response to motion of augmented reality system 1300 and may be located on substantially any portion of frame 1310 . Sensor 1340 may represent one or more of a variety of different sensing mechanisms, such as position sensors, inertial measurement units (IMUs), depth camera assemblies, structured light emitters, and/or or detectors, or any combination thereof. In some embodiments, the augmented reality system 1300 may or may not include the sensor 1340 or may include more than one sensor. In embodiments where sensor 1340 includes an IMU, the IMU can generate calibration data based on measurement signals from sensor 1340 . Examples of sensors 1340 may include, but are not limited to, accelerometers, gyroscopes, magnetometers, other suitable types of sensors to detect motion, sensors for error correction of IMUs, or some combination thereof.

In some examples, augmented reality system 1300 may also include a microphone array with a plurality of acoustic transducers 1320 (A)- 1320 (J), collectively referred to as acoustic transducers 1320 . Acoustic energy transducer 1320 may represent a transducer that detects changes in air pressure induced by sound waves. Each acoustic transducer 1320 may be configured to detect sound and convert the detected sound to an electronic format (eg, analog or digital format). The microphone array in FIG. 13 may include, for example, ten acoustic transducers: 1320(A) and 1320(B), which may be designed to be placed inside the corresponding ears of the user; acoustic transducers 1320(C), 1320(D), 1320(E), 1320(F), 1320(G), and 1320(H), which may be positioned at various locations on frame 1310; and/or acoustic transducers 1320(1) and 1320 (J), which can be positioned on the corresponding neck strap 1305.

In some embodiments, one or more of the acoustic energy transducers 1320(A)-1320(J) may function as an output transducer (eg, a speaker). For example, acoustic transducers 1320(A) and/or 1320(B) may be earphones or any other suitable type of headphones or speakers.

The configuration of the acoustic transducer 1320 of the microphone array can vary. Although the augmented reality system 1300 is shown in FIG. 13 as having ten acoustic transducers 1320 , the number of acoustic transducers 1320 may be greater or less than ten. In some embodiments, using a higher number of acoustic transducers 1320 can increase the amount of audio information collected and/or the sensitivity and accuracy of the audio information. In contrast, using a lower number of acoustic transducers 1320 reduces the computational power required by the associated controller 1350 to process the collected audio information. In addition, the positions of the acoustic transducers 1320 of the microphone array may be different. For example, the location of the acoustic transducers 1320 may include a defined location on the user, defined coordinates on the frame 1310, an orientation associated with each acoustic transducer 1320, or some combination thereof.

Acoustic transducers 1320(A) and 1320(B) may be positioned on different parts of the user's ear, such as behind the pinna, behind the tragus, and/or within the pinna or ear socket. Alternatively, in addition to the acoustic transducer 1320 inside the ear canal, there may also be an additional acoustic transducer 1320 on or around the ear. Positioning the acoustic transducer 1320 in close proximity to the user's ear canal may enable the microphone array to gather information about how sound reaches the ear canal. By positioning at least two of the acoustic transducers 1320 on either side of the user's head (e.g., as binaural microphones), the augmented reality device 1300 can simulate binaural hearing and capture the user's head Surrounding 3D stereo sound field. In some embodiments, acoustic energy transducers 1320(A) and 1320(B) can be connected to augmented reality system 1300 via wired connection 1330, and in other embodiments, acoustic energy transducers 1320(A) and 1320 (B) Can be connected to the augmented reality system 1300 via a wireless connection (eg, Bluetooth connection). In still other embodiments, acoustic energy transducers 1320(A) and 1320(B) may not be used in conjunction with augmented reality system 1300 at all.

The acoustic transducers 1320 on the frame 1310 can be positioned in a number of different ways, including along the length of the temples, across bridges, above or below the left display device 1315(A) and right display device 1315(B), or in some other way. a combination. Acoustic transducer 1320 may be oriented such that the microphone array is capable of detecting sound in a wide range of directions around the user wearing augmented reality system 1300 . In some embodiments, an optimization procedure may be performed during manufacture of the augmented reality system 1300 to determine the relative positioning of the acoustic transducers 1320 in the microphone array.

In some examples, augmented reality system 1300 may include or be connected to an external device (eg, a companion device), such as neckband 1305 . Neckband 1305 generally represents any type or form of paired device. Accordingly, the following discussion of the neckband 1305 is also applicable to various other paired devices, such as charging cases, smart watches, smartphones, wristbands, other wearable devices, handheld controllers, tablets, laptops, other external computing devices, etc.

As shown, the neckband 1305 can be coupled to the eyewear device 1302 via one or more connectors. Connectors may be wired or wireless, and may include electrical and/or non-electrical (eg, structural) components. In some cases, glasses device 1302 and neckband 1305 may operate independently without any wired or wireless connection between the two. Although FIG. 13 depicts components of the eyewear device 1302 and neckband 1305 in example locations on the eyewear device 1302 and neckband 1305, such components may be located elsewhere and/or distributed in different ways across the eyewear device 1302 and/or or neck strap 1305 on. In some embodiments, components of the glasses device 1302 and neckband 1305 may be located on one or more additional peripheral devices paired with the glasses device 1302, neckband 1305, or some combination thereof.

Pairing an external device such as the neckband 1305 with the augmented reality glasses device may enable the glasses device to achieve the form factor of a pair of glasses while still providing sufficient battery and computing power for expansion capabilities. Some or all of the battery power, computing resources, and/or additional features of the augmented reality system 1300 may be provided by the paired device or shared between the paired device and the glasses device, thus reducing the weight, heat distribution, and form factor while still retaining desired functionality. For example, the neck strap 1305 can allow components that would otherwise be included on the eyewear device to be included in the neck strap 1305 because the user can bear more weight on their shoulders than they would on their head weight load. The neckband 1305 may also have a larger surface area on which to spread and dissipate heat to the surrounding environment. Thus, the neckband 1305 may allow for greater battery capacity and computing power than might otherwise exist on a standalone eyewear device. Since the weight carried in the neckband 1305 may be less intrusive to the user than the weight carried in the eyewear device 1302, the user can afford to wear a lighter eyewear device and bear the burden of carrying or wearing a paired device. The length of time is greater than the length of time a user would endure wearing a heavier standalone eyewear device, thereby enabling the user to more fully incorporate the artificial reality environment into their daily activities.

Neckband 1305 may be communicatively coupled with eyewear device 1302 and/or other devices. These other devices may provide certain functionality to the augmented reality system 1300 (eg, tracking, positioning, depth mapping, processing, storage, etc.). In the particular example of FIG. 13, the neckband 1305 may include two acoustic transducers (e.g., 1320(I) and 1320(J)) that are part of a microphone array (or may form their own microphone sub-array) . Neckband 1305 may also include controller 1325 and power supply 1335 .

Acoustic transducers 1320(I) and 1320(J) of neckband 1305 may be configured to detect sound and convert the detected sound to an electronic format (analog or digital). In the specific example of FIG. 13, acoustic transducers 1320(I) and 1320(J) may be positioned on neckband 1305, thereby increasing the number of acoustic transducers 1320(I) and 1320(J) on the neckband. The distance from other acoustic transducers 1320 positioned on the eyewear device 1302 . In some cases, increasing the distance between the acoustic transducers 1320 of the microphone arrays can improve the accuracy of beamforming performed via the microphone arrays. For example, if sound is detected by acoustic transducers 1320(C) and 1320(D) and the distance between acoustic transducers 1320(C) and 1320(D) is greater than, for example, acoustic transducer 1320 (D) and 1320(E), the determined source location of the detected sound can be more accurate than if the sound was detected by acoustic transducers 1320(D) and 1320(E).

Controller 1325 of neckband 1305 may process information generated by sensors on neckband 1305 and/or augmented reality system 1300 . For example, the controller 1325 may process information from the microphone array describing the sound detected by the microphone array. For each detected sound, the controller 1325 may perform a direction of arrival (DOA) estimation to estimate a direction from which the detected sound arrives at the microphone array. When sound is detected by the microphone array, the controller 1325 can populate the audio dataset with information. In embodiments where augmented reality system 1300 includes an inertial measurement unit, controller 1325 may calculate all inertial and spatial calculations from an IMU located on eyewear device 1302 . The connectors can communicate information between the augmented reality system 1300 and the neckband 1305 and between the augmented reality system 1300 and the controller 1325 . This information may be in the form of optical data, electrical data, wireless data or any other form of transmittable data. Moving the processing of information generated by the augmented reality system 1300 to the neckband 1305 can reduce weight and heat in the eyewear device 1302, thereby making the eyewear device more comfortable for the user.

A power supply 1335 in the neckband 1305 may provide power to the eyewear device 1302 and/or the neckband 1305 . Power source 1335 may include, but is not limited to, lithium ion batteries, lithium polymer batteries, lithium primary batteries, alkaline batteries, or any other form of power storage. In some cases, power source 1335 may be a wired power source. Including the power supply 1335 on the neckband 1305 instead of the eyewear device 1302 can help to better distribute the weight and heat generated by the power supply 1335 .

As mentioned, instead of blending artificial reality with actual reality, some artificial reality systems may essentially replace one or more of the user's sensory perception of the real world with a virtual experience. An example of a system of this type is a head mounted display system, such as virtual reality system 1400 in FIG. 14, which primarily or completely covers the user's field of view. The virtual reality system 1400 may include a front rigid body 1402 and a band 1404 shaped to fit around a user's head. The virtual reality system 1400 may also include output audio converters 1406(A) and 1406(B). Additionally, although not shown in FIG. 14 , front rigid body 1402 may include one or more electronic components including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking transmitters sensor or detector and/or any other suitable device or system for generating an artificial reality experience.

An artificial reality system may include various types of visual feedback mechanisms. For example, display devices in augmented reality system 1300 and/or virtual reality system 1400 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, micro LED displays, organic LED ( OLED) displays, digital light projection (DLP) microdisplays, liquid crystal on silicon (LCoS) microdisplays, and/or any other suitable type of display screen. These artificial reality systems may include a single display screen for both eyes or may provide a display screen for each eye, which may allow additional flexibility for zoom adjustment or for correcting the user's refractive error. Some of these artificial reality systems may also include an optical subsystem with one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, tunable liquid lenses, etc.) Wait for the lens to view the display screen. These optical subsystems can be used for a variety of purposes, including collimating light (for example, making objects appear at a greater distance than they actually are), amplifying light (for example, making objects appear larger than they really are), and / or relay light (relay light to eg the eyes of the viewer). These optical subsystems can be used in non-pupil forming architectures (such as single lens configurations that directly collimate light but produce so-called pincushion distortion) and/or pupil-forming architectures (such as producing so-called barrel distortion to eliminate the pincushion distortion). shape distortion of the multi-lens configuration).

Some of the artificial reality systems described herein may also include one or more projection systems in addition to or instead of using a display screen. For example, display devices in augmented reality system 1300 and/or virtual reality system 1400 may include micro LED projectors that project light (using, for example, waveguides) into display devices such as those that allow Clear combiner lens through which ambient light passes. The display device can refract the projected light toward the user's pupil and can enable the user to view both the artificial reality content and the real world at the same time. Display devices may use any of a variety of different optical components for this purpose, including waveguide components (e.g., holographic, planar, diffractive, polarizing, and/or reflective waveguide elements), light manipulating surfaces, and elements (such as diffractive, reflective and refractive elements and gratings), coupling elements, etc. The artificial reality system may also be configured with any other suitable type or form of image projection system, such as a retinal projector used in a virtual retinal display.

The artificial reality systems described herein may also include various types of computer vision components and subsystems. For example, augmented reality system 1300 and/or virtual reality system 1400 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, flight Temporal depth sensors, single beam or swept laser range finders, 3D lidar sensors and/or any other suitable type or form of optical sensors. An artificial reality system may process data from one or more of these sensors to identify a user's location, map the real world, provide the user with context about the real world environment, and/or perform a variety of other functions.

The artificial reality systems described herein may also include one or more input and/or output audio converters. Output audio transducers may include voice coil speakers, ribbon speakers, electrostatic speakers, piezoelectric speakers, bone conduction transducers, cartilage conduction transducers, tragus vibration transducers, and/or any other suitable type or form of audio transducers. Similarly, input audio transducers may include condenser microphones, dynamic microphones, ribbon microphones, and/or any other type or form of input transducers. In some embodiments, a single converter can be used for both audio input and audio output.

In some embodiments, the artificial reality systems described herein can also include haptic (i.e., sense of touch) feedback systems that can be incorporated into headgear, gloves, one-piece suits, handheld controllers , environmental fixtures (eg, chairs, floor mats, etc.) and/or any other type of device or system. Haptic feedback systems can provide various types of skin feedback including vibration, force, traction, texture and/or temperature. Haptic feedback systems can also provide various types of kinesthetic feedback, such as motion and compliance. Haptic feedback can be implemented using motors, piezoelectric actuators, fluid systems, and/or various other types of feedback mechanisms. The haptic feedback system can be implemented independently of, within, and/or in conjunction with other VR devices.

By providing tactile sensations, auditory content, and/or visual content, an artificial reality system can create an entire virtual experience or enhance a user's real-world experience in a variety of situations and environments. For example, an artificial reality system can assist or extend a user's perception, memory, or cognition within a specific environment. Some systems may enhance a user's interaction with others in the real world or enable more immersive interactions with others in a virtual world. Artificial reality systems may also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government organizations, military organizations, commercial enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.) and/or for accessibility purposes (e.g. as hearing aids, visual aids, etc.). Embodiments disclosed herein may enable or enhance a user's artificial reality experience in one or more of these contexts and environments and/or in other contexts and environments.

Some augmented reality systems may use a technique known as "simultaneous location and mapping" (SLAM) to map the environment of the user and/or the device. SLAM mapping and location recognition techniques can involve a variety of hardware and software tools that can generate or update a map of an environment while tracking a user's location within the mapped environment. SLAM can use many different types of sensors to generate a map and determine the user's location within the map.

SLAM technology can, for example, implement optical sensors to determine the location of a user. Radios including WiFi, Bluetooth, Global Positioning System (GPS), cellular, or other communication devices can also be used to determine a user's location relative to a radio transceiver or group of transceivers (for example, a WiFi router or a group of GPS satellites) . Acoustic sensors such as microphone arrays or 2D or 3D sonar sensors can also be used to determine the user's position within the environment. Augmented reality devices and virtual reality devices such as systems 1300 and 1400 of FIGS. 13 and 14 , respectively, can incorporate any or all of these types of sensors to perform SLAM operations, such as generating and continuously updating using A map of the reader's current environment. In at least some of the embodiments described herein, the SLAM data generated by these sensors may be referred to as "environmental data" and may be indicative of the user's current environment. This data can be stored in local or remote data storage (eg, cloud data storage) and can be provided to the user's AR/VR device on demand.

As mentioned, the augmented reality systems 1300 and 1400 can be used with various other types of devices to provide a more convincing augmented reality experience. These devices can be haptic interfaces with transducers that provide haptic feedback and/or collect haptic information about the user's interaction with the environment. The artificial reality systems disclosed herein may include various types of tactile interfaces that detect or transmit various types of tactile information, including tactile feedback (e.g., feedback detected by a user via nerves in the skin, It may also be referred to as cutaneous feedback) and/or kinesthetic feedback (eg, feedback detected by the user via receptors located in muscles, joints and/or tendons).

Haptic feedback may be provided by an interface positioned within the user's environment (eg, chair, table, floor, etc.) and/or an interface on an item (eg, glove, wristband, etc.) that may be worn or carried by the user. As an example, FIG. 15 illustrates a vibrohaptic system 1500 in the form of a wearable glove (haptic device 1510 ) and wristband (haptic device 1520 ). Haptic device 1510 and haptic device 1520 are shown as examples of wearable devices including flexible, wearable textile material 1530 shaped and configured to fit against a user's hand and wrist, respectively. position. The present disclosure also includes vibrotactile systems that can be shaped and configured for positioning against other body parts such as fingers, arms, head, torso, feet, or legs. By way of example and not limitation, vibrotactile systems according to various embodiments of the present disclosure may also take the form of gloves, headbands, armbands, cuffs, hoods, socks, shirts, or pants, among other possibilities. In some instances, the term "textile" may include any flexible, wearable material, including wovens, non-wovens, leather, woven cloth, flexible polymeric materials, composite materials, and the like.

One or more vibro-tactile devices 1540 may be positioned at least partially within one or more corresponding recesses formed in textile material 1530 of vibro-tactile system 1500 . The vibro-tactile device 1540 may be positioned in a location that provides a vibratory sensation (eg, tactile feedback) to a user of the vibro-tactile system 1500 . For example, vibrotactile device 1540 may be positioned against a user's finger, thumb, or wrist, as shown in FIG. 15 . In some examples, the vibro-tactile device 1540 may be flexible enough to conform to or bend with the corresponding body part of the user.

A power source 1550 (eg, a battery) for applying a voltage to the vibro-tactile device 1540 for activating the vibro-tactile device may be electrically coupled to the vibro-tactile device 1540 , such as via conductive wiring 1552 . In some examples, each of vibratory haptic devices 1540 may be independently electrically coupled to power source 1550 for individual activation. In some embodiments, processor 1560 is operatively coupled to power source 1550 and configured (eg, programmed) to control activation of vibro-tactile device 1540 .

The vibrotactile system 1500 can be implemented in a variety of ways. In some examples, vibrotactile system 1500 may be a stand-alone system with integrated subsystems and components for operation independently of other devices and systems. As another example, vibrotactile system 1500 may be configured for interaction with another device or system 1570 . For example, vibrotactile system 1500 may include a communication interface 1580 for receiving signals and/or sending signals to another device or system 1570 in some examples. Another device or system 1570 can be a mobile device, a game console, an artificial reality (e.g., virtual reality, augmented reality, mixed reality) device, a personal computer, a tablet computer, a network device (e.g., a modem) , routers, etc.), handheld controllers, etc. Communication interface 1580 may enable communication between vibro-haptic system 1500 and another device or system 1570 via a wireless (eg, Wi-Fi, Bluetooth, cellular, radio, etc.) link or a wired link. If present, communication interface 1580 may communicate with processor 1560 to provide a signal to processor 1560 to activate or deactivate one or more of vibro-tactile devices 1540 .

Vibrotactile system 1500 may optionally include other subsystems and components, such as touch sensitive pads 1590, pressure sensors, motion sensors, position sensors, lighting elements, and/or user interface elements (e.g., switch on /disconnect buttons, vibration control elements, etc.). During use, the vibrotactile device 1540 can be configured to be activated for a number of different reasons, such as in response to user interaction with a user interface element, a signal from a motion or position sensor, from a touch-sensitive backing A signal from the pad 1590, a signal from a pressure sensor, a signal from another device or system 1570, etc.

Although power supply 1550, processor 1560, and communication interface 1580 are depicted in FIG. 15 as being positioned within haptic device 1520, the disclosure is not so limited. For example, one or more of power supply 1550, processor 1560, or communication interface 1580 may be positioned within haptic device 1510 or within another wearable textile.

Haptic wearables such as those shown in and described in connection with FIG. 15 can be implemented in many types of artificial reality systems and environments. 16 shows an example artificial reality environment 1600 that includes a head-mounted virtual reality display and two haptic devices (i.e., gloves), and in other embodiments, any number of these and other components and/or Any combination of these components and other components may be included in an artificial reality system. For example, in some embodiments there may be multiple head-mounted displays, each with an associated haptic device, where each head-mounted display and each haptic device is connected to the same console, portable computing device, or other computing system communication.

Head-mounted display 1602 generally represents any type or form of virtual reality system, such as virtual reality system 1400 in FIG. 14 . Haptic device 1604 generally represents any type or form of wearable device worn by a user of an artificial reality system that provides haptic feedback to the user to give the user a sense that he or she is physically connected to a virtual object. In some embodiments, haptic device 1604 can provide haptic feedback by applying vibration, motion, and/or force to the user. For example, the haptic device 1604 can limit or amplify the movement of the user. As a particular example, the haptic device 1604 can constrain the forward movement of the user's hand to give the user the perception that his or her hand is in physical contact with the virtual wall. In this particular example, one or more actuators within the haptic device can achieve physical movement restriction by pumping fluid into an inflatable bladder of the haptic device. In some examples, the user may also use the haptic device 1604 to send action requests to the console. Examples of action requests include, but are not limited to, requests to start an application and/or end an application and/or to perform a specific action within an application.

While the haptic interface can be used with a virtual reality system, as shown in FIG. 16 , the haptic interface can also be used with an augmented reality system, as shown in FIG. 17 . FIG. 17 is a perspective view of user 1710 interacting with augmented reality system 1700 . In this example, a user 1710 may wear a pair of augmented reality glasses 1720 that may have one or more displays 1722 and pair with a haptic device 1730 . In this example, haptic device 1730 may be a wristband including a plurality of strap elements 1732 and a tensioning mechanism 1734 connecting strap elements 1732 to each other.

One or more of strap elements 1732 may include any type or form of actuator suitable for providing tactile feedback. For example, one or more of strap elements 1732 may be configured to provide one or more of various types of skin feedback, including vibration, force, traction, texture, and/or temperature. To provide such feedback, strap element 1732 may include one or more of various types of actuators. In one example, each of strap elements 1732 may include vibratory haptics (eg, vibrotactile actuators) configured to vibrate collectively or independently to provide the user with various types of One or more of the tactile sensations. Alternatively, only a single strap element or a subset of strap elements may include vibratory tactile sensors.

Haptic devices 1510, 1520, 1604, and 1730 may include any suitable number and/or type of haptic transducers, sensors, and/or feedback mechanisms. For example, haptic devices 1510, 1520, 1604, and 1730 may include one or more mechanical, piezoelectric, and/or fluidic transducers. Haptic devices 1510, 1520, 1604, and 1730 may also include various combinations of different types and forms of transducers that work together or independently to enhance the user's artificial reality experience. In one example, each of the strap elements 1732 of the haptic device 1730 may include a vibrotactile sensor (e.g., a vibrotactile actuator) configured to vibrate collectively or independently for use or provide one or more of various types of tactile sensations.

In some embodiments, the systems described herein may also include an eye-tracking subsystem designed to identify and track various characteristics of a user's eyes, such as the user's gaze direction. The phrase "eye tracking" may refer in some instances to a process by which the position, orientation and/or movement of an eye is measured, detected, sensed, determined and/or monitored. The disclosed system can measure eye position, orientation, and/or motion in a number of different ways, including through the use of various optical-based eye-tracking techniques, ultrasound-based eye-tracking techniques, and the like. The eye-tracking subsystem can be configured in many different ways and can include many different eye-tracking hardware components or other computer vision components. For example, an eye-tracking subsystem can include a variety of different optical sensors, such as two-dimensional (2D) or 3D cameras, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors sensor and/or any other suitable type or form of optical sensor. In this example, the processing subsystem may process data from one or more of these sensors to measure, detect, determine, and/or otherwise monitor the position, orientation, and/or sports.

FIG. 18 is a diagram of an exemplary system 1800 incorporating an eye tracking subsystem capable of tracking a user's eyes. As depicted in FIG. 18 , system 1800 may include light source 1802 , optics subsystem 1804 , eye-tracking subsystem 1806 , and/or control subsystem 1808 . In some examples, light source 1802 may generate light for an image (eg, to be presented to a viewer's eye 1801 ). Light source 1802 may represent any of a variety of suitable devices. For example, light source 1802 may include a two-dimensional projector (e.g., an LCoS display), a scanning source (e.g., a scanning laser), or other device (e.g., LCD, LED display, OLED display, active matrix OLED display (AMOLED), A transparent OLED display (TOLED), a waveguide, or some other display capable of generating light for presenting an image to a viewer). In some examples, an image may represent a virtual image, which may refer to an optical image formed by the apparent divergence of light rays from a point in space, as opposed to an image formed by the actual divergence of light rays.

In some embodiments, optical subsystem 1804 can receive light generated by light source 1802 and generate convergent light 1820 including an image based on the received light. In some examples, optics subsystem 1804 may include any number of lenses (e.g., Fresnel lenses, convex lenses, concave lenses), apertures, filters, mirrors, lenses, and/or other optical components that may be associated with actuation device and/or other device combinations. In particular, actuators and/or other devices may translate and/or rotate one or more of the optical components to alter one or more aspects of the focused light 1820 . Additionally, various mechanical couplings may be used to maintain the relative spacing and/or orientation of the optical components in any suitable combination.

In one embodiment, the eye tracking subsystem 1806 can generate tracking information indicative of the gaze angle of the viewer's eyes 1801 . In this embodiment, control subsystem 1808 can control an aspect of optics subsystem 1804 (eg, the angle of incidence of convergent light 1820 ) based at least in part on the tracking information. Additionally, in some examples, the control subsystem 1808 may store and utilize historical tracking information (e.g., a history of tracking information for a given duration such as the previous second or fraction thereof) to anticipate the gaze angle of the eye 1801 ( For example, the angle between the visual axis of the eye 1801 and the anatomical axis). In some embodiments, eye tracking subsystem 1806 may detect radiation originating from a portion of eye 1801 (eg, cornea, iris, pupil, or the like) to determine the current gaze angle of eye 1801 . In other examples, the eye tracking subsystem 1806 may employ a wavefront sensor to track the current location of the pupil.

Any number of techniques may be used to track eyes 1801. Some techniques may involve illuminating the eye 1801 with infrared light, and measuring reflection with at least one optical sensor tuned to be sensitive to infrared light. Information about how infrared light is reflected from the eye 1801 may be analyzed to determine the position, orientation, and/or motion of one or more features of the eye, such as the cornea, pupil, iris, and/or retinal blood vessels.

In some examples, radiation captured by sensors of eye-tracking subsystem 1806 may be digitized (ie, converted into electronic signals). Additionally, the sensor may transmit a digital representation of this electronic signal to one or more processors (eg, a processor associated with a device including eye-tracking subsystem 1806). Eye tracking subsystem 1806 may include any of a variety of sensors in a variety of different configurations. For example, eye tracking subsystem 1806 may include an infrared detector that responds to infrared radiation. Infrared detectors may be thermal detectors, photon detectors, and/or any other suitable type of detector. Thermal detectors may include detectors that respond to the thermal effect of incident infrared radiation.

In some examples, one or more processors may process digital representations generated by sensors of eye tracking subsystem 1806 to track the movement of eyes 1801 . In another example, the processors can track the movement of the eyes 1801 by executing algorithms represented by computer-executable instructions stored on non-transitory memory. In some instances, on-chip logic (eg, application specific integrated circuits or ASICs) may be used to perform at least a portion of such algorithms. As mentioned, the eye tracking subsystem 1806 can be programmed to track the movement of the eye 1801 using the output of the sensors. In some embodiments, eye tracking subsystem 1806 can analyze digital representations generated by sensors to extract eye rotation information from changes in reflection. In one embodiment, the eye tracking subsystem 1806 can use the corneal reflection or glint (also known as the Purkinje image) and/or the center of the pupil 1822 of the eye as features to track over time.

In some embodiments, the eye-tracking subsystem 1806 may use the center of the pupil 1822 of the eye and infrared or near-infrared uncollimated light to generate corneal reflections. In these embodiments, the eye-tracking subsystem 1806 may calculate the gaze direction of the eye 1801 using the vector between the center of the eye's pupil 1822 and the corneal reflection. In some embodiments, the disclosed system can perform a calibration procedure for the individual (using, for example, supervised or unsupervised techniques) prior to tracking the user's eyes. For example, the calibration procedure may include directing the user to look at one or more points displayed on the display while the eye-tracking system records values corresponding to each gaze position associated with each point.

In some embodiments, the eye-tracking subsystem 1806 can use two types of infrared and/or near-infrared (also known as active light) eye-tracking technology: bright pupil and dark pupil eye tracking, which can be based on the relative The parts of the optical components used are distinguished. If the illumination is coaxial with the optical path, the eye 1801 can act as a back reflector when light reflects off the retina, thereby creating a bright pupil effect similar to the red eye effect in photography. If the illumination originates from an optical path shift, the pupil 1822 of the eye may appear dark because retroreflections from the retina are directed away from the sensor. In some embodiments, bright pupil tracking can result in greater iris/pupil contrast, thereby allowing for more robust eye tracking due to iris pigmentation, and can provide reduced interference (e.g., caused by eyelashes and other occluding features) interference). Bright pupil tracking also allows tracking in lighting conditions ranging from total darkness to very bright environments.

In some embodiments, control subsystem 1808 may control light source 1802 and/or optical subsystem 1804 to reduce optical aberrations (e.g., chromatic aberrations and/or monochromatic aberrations) of images that may be caused or affected by eye 1801 . In some examples, as mentioned above, control subsystem 1808 may use tracking information from eye-tracking subsystem 1806 to perform such control. For example, in controlling the light source 1802, the control subsystem 1808 may alter the light produced by the light source 1802 (e.g., by means of image rendering) to modify (e.g., pre-distort) the image such that the image caused by the eye 1801 Aberrations are reduced.

The disclosed system can track both the position and relative size of the pupil (due to, for example, pupil dilation and/or constriction). In some examples, eye-tracking devices and components (eg, sensors and/or sources) used to detect and/or track pupils may be different (or calibrated differently) for different types of eyes. For example, the frequency range of the sensor may be different (or calibrated separately) for eyes of different colors and/or different pupil types, sizes and/or the like. As such, the various eye tracking components (eg, infrared sources and/or sensors) described herein may need to be calibrated for each individual user and/or eye.

The disclosed system can track both eyes with or without ophthalmic corrections such as provided by contact lenses worn by the user. In some embodiments, ophthalmic corrective elements (eg, adjustable lenses) can be incorporated directly into the artificial reality systems described herein. In some instances, the color of the user's eyes may require modification of the corresponding eye tracking algorithm. For example, it may be desirable to modify the eye tracking algorithm based at least in part on the different color contrast between brown eyes and, for example, blue eyes.

19 is a more detailed illustration of various aspects of the eye tracking subsystem depicted in FIG. 18 . As shown in this figure, eye tracking subsystem 1900 may include at least one source 1904 and at least one sensor 1906 . Source 1904 generally represents any type or form of element capable of emitting radiation. In one example, source 1904 can generate visible light, infrared light, and/or near infrared light radiation. In some examples, source 1904 may radiate uncollimated infrared light and/or near-infrared light portions of the electromagnetic spectrum toward user's eye 1902 . Source 1904 may utilize a variety of sampling rates and speeds. For example, the disclosed system may use a source with a higher sampling rate in order to capture the gaze eye movement of the user's eye 1902 and/or correctly measure the saccade dynamics of the user's eye 1902 . As mentioned above, any type or form of eye-tracking technology may be used to track the user's eyes 1902, including optical-based eye-tracking technology, ultrasound-based eye-tracking technology, and the like.

Sensor 1906 generally represents any type or form of element capable of detecting radiation, such as radiation reflected from a user's eye 1902 . Examples of sensor 1906 include, but are not limited to, charge coupled devices (CCDs), photodiode arrays, complementary metal oxide semiconductor (CMOS) based sensor devices, and/or the like. In one example, sensor 1906 may represent a sensor having predetermined parameters including, but not limited to, dynamic resolution range, linearity, and/or other sensors selected and/or specifically designed for eye tracking. characteristic.

As detailed above, eye tracking subsystem 1900 may generate one or more flashes of light. As detailed above, glint 1903 may represent the reflection of radiation (eg, infrared light radiation from an infrared light source such as source 1904 ) off the structure of the user's eye. In various embodiments, the flash 1903 and/or the user's pupils may be tracked using an eye-tracking algorithm executed by a processor (inside or outside the artificial reality device). For example, an artificial reality device may include a processor and/or memory device for performing eye tracking locally, and/or for sending and receiving A transceiver for the data necessary to perform eye-tracking on the device).

FIG. 19 shows an example image 1905 captured by an eye-tracking subsystem, such as eye-tracking subsystem 1900 . In this example, image 1905 may include both the user's pupil 1908 and a flash of light 1910 near the pupil. In some examples, pupils 1908 and/or glint 1910 may be identified using artificial intelligence-based algorithms, such as computer vision-based algorithms. In one embodiment, image 1905 may represent a single frame in a series of frames that may be analyzed over time in order to track user's eyes 1902 . In addition, pupil 1908 and/or flash 1910 may be tracked over a period of time to determine the user's gaze.

In one example, the eye tracking subsystem 1900 can be configured to identify and measure a user's interpupillary distance (IPD). In some embodiments, the eye-tracking subsystem 1900 can measure and/or calculate the user's IPD while the user is wearing the artificial reality system. In these embodiments, the eye-tracking subsystem 1900 can detect the location of the user's eyes and can use this information to calculate the user's IPD.

As mentioned, eye tracking systems or subsystems disclosed herein can track a user's eye position and/or eye movement in a variety of ways. In one example, one or more light sources and/or optical sensors may capture images of the user's eyes. The eye-tracking subsystem can then use the captured information to determine the user's interpupillary distance, interocular distance, and/or the 3D position of each eye (e.g., for distortion adjustment purposes), including twist and rotation (i.e., roll , pitch and yaw) and/or the gaze direction for each eye. In one example, infrared light may be emitted by the eye-tracking subsystem and reflected from each eye. The reflected light may be received or detected by an optical sensor and analyzed to extract eye rotation data from changes in infrared light reflected from each eye.

The eye tracking subsystem may use any of a number of different methods to track the user's eyes. For example, a light source (eg, an infrared light emitting diode) may emit a pattern of dots onto each eye of the user. The eye-tracking subsystem can then detect (eg, via optical sensors coupled to the artificial reality system) and analyze the reflection of the dot pattern from each eye of the user to identify the location of each pupil of the user. Thus, the eye-tracking subsystem can track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) for each eye and at least a subset of the tracked quantities can be estimated from a combination of the user's two eyes Gaze point (ie, the 3D part or position in the virtual scene that the user is looking at) and/or IPD.

In some cases, the distance between the user's pupil and the display may change as the user's eyes move to look in different directions. The varying distance between the pupil and the display when the viewing direction changes can be referred to as "pupil wandering" and can contribute to the distortion perceived by the user because light is focused at different locations when the distance between the pupil and the display changes caused by. Therefore, measuring distortion at different eye positions and pupillary distances relative to the display and generating distortion corrections for different positions and distances may allow for tracking the 3D position of the user's eyes and applying the corresponding Distortion correction of the 3D position of each of the subject's eyes to mitigate distortion caused by pupil wander. Thus, knowing the 3D position of each of the user's eyes may allow distortion caused by changes in the distance between the pupil of the eye and the display to be mitigated by applying a distortion correction for each 3D eye position. Furthermore, as mentioned above, knowledge of the location of each of the user's eyes may also enable the eye-tracking subsystem to make automatic adjustments to the user's IPD.

In some embodiments, the display subsystem can include various additional subsystems that can work in conjunction with the eye-tracking subsystem described herein. For example, the display subsystem may include a zoom subsystem, a scene rendering module, and/or a vergence processing module. The zoom subsystem enables the left and right display elements to change the focal length of the display device. In one embodiment, the zoom subsystem can physically change the distance between the display and the optics through which the zoom subsystem is viewed by moving the display, the optics, or both. Additionally, moving or translating the two lenses relative to each other can also be used to change the focal length of the display. Thus, the zoom subsystem may include actuators or motors that move the display and/or optics to change the distance between the two. This zoom subsystem can be separate from the display subsystem or integrated into the display subsystem. The zoom subsystem may also be integrated into or separate from its actuation subsystem and/or the eye tracking subsystem described herein.

In one example, the display subsystem may include a vergence processing module configured to determine a depth of convergence of a user's gaze based on gaze points and/or estimated intersection points of gaze lines determined by the eye tracking subsystem. Vergence may refer to simultaneous movement or rotation of two eyes in opposite directions to maintain monocular vision, which may be performed naturally and automatically by the human eye. Thus, the part where the user's eyes turn is the part where the user is looking at, and typically also the part where the user's eyes are focused. For example, the convergence processing module may triangulate gaze lines to estimate distances or depths from the user associated with intersection points of gaze lines. The depth associated with the intersection of the gaze lines can then be used as an approximation of the adjustment distance, which can identify the distance from the user at which the user's eyes are pointing. Thus, the vergence distance allows a determination of where the user's eyes should focus and at what depth from the user's eyes at which the eyes should focus, thereby providing information (such as the plane of the object or focal point) for rendering adjustments to the virtual scene .

The vergence processing module may coordinate with the eye tracking subsystem described herein to make adjustments to the display subsystem to account for the user's depth of convergence. When the user focuses on something at a distance, the user's pupils may be slightly further apart than when the user focuses on something nearby. The eye-tracking subsystem can obtain information about the user's vergence or depth of focus and can adjust the display subsystem to move closer together when the user's eyes are focused or close to something nearby, and when the user's eyes are focused or close More apart when something is at a distance.

Eye-tracking information generated by the eye-tracking subsystem described above may also be used, for example, to modify various aspects of how different computer-generated images are presented. For example, the display subsystem can be configured to modify at least one aspect of how the computer-generated image is presented based on information generated by the eye-tracking subsystem. For example, a computer-generated image can be modified based on the user's eye movement so that if the user is looking, the computer-generated image can move upwards on the screen. Similarly, if the user is looking sideways or down, the computer generated image can move sideways or down on the screen. If the user's eyes are closed, the computer-generated image can be paused or removed from the display and resumed after the user's eyes are opened again.

The eye-tracking subsystem described above can be incorporated into one or more of the various immersive systems described herein in a variety of ways. For example, one or more of the various components of system 1800 and/or eye tracking subsystem 1900 may be incorporated into augmented reality system 1300 in FIG. 13 and/or virtual reality system 1400 in FIG. 14 to enable such systems to perform various eye-tracking tasks, including one or more of the eye-tracking operations described herein.

FIG. 20A illustrates an exemplary human-machine interface (also referred to herein as an EMG control interface) configured to be worn on a user's lower arm or wrist as a wearable system 2000 . In this example, wearable system 2000 may include sixteen neuromuscular sensors 2010 (e.g., EMG sensors) arranged circumferentially around elastic band 2020, wherein inner surface 2030 is configured to contact the user's skin. However, any suitable number of neuromuscular sensors may be used. The number and configuration of neuromuscular sensors may depend on the particular application for which the wearable device is used. For example, a wearable armband or wristband can be used to generate control information for controlling an augmented reality system, a robot, controlling a vehicle, scrolling text, controlling an avatar, or any other suitable control task. As shown, the sensors can be coupled together using flexible electronics incorporated into the wireless device. 20B shows a cross-sectional view through one of the sensors of the wearable device shown in FIG. 20A. In some embodiments, hardware signal processing circuitry can optionally be used to process the output of one or more of the sensing components (eg, to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the output of the sensing element may be performed in software. Thus, signal processing of signals sampled by sensors may be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the techniques described herein are not limited in this respect. A non-limiting example of a signal processing chain to process recorded data from the sensor 2010 is discussed in more detail below with reference to FIGS. 21A and 21B .

21A and 21B show illustrative schematic diagrams of internal components of a wearable system with an EMG sensor. As shown, the wearable system can include a wearable portion 2110 ( FIG. 21A ) and an adapter portion 2120 ( FIG. 21B ) that communicates with the wearable portion 2110 (eg, via Bluetooth or another suitable wireless communication technology). As shown in FIG. 21A, wearable portion 2110 may include skin contact electrodes 2111, examples of which are described in connection with FIGS. 20A and 20B. The output of the skin contact electrodes 2111 can be provided to an analog front end 2130, which can be configured to perform analog processing (eg, amplification, noise reduction, filtering, etc.) on the recorded signal. The processed analog signal may then be provided to an analog-to-digital converter 2132, which may convert the analog signal to a digital signal that may be processed by one or more computer processors. An example of a computer processor that may be used in accordance with some embodiments is the microcontroller (MCU) 2134 depicted in Figure 21A. As shown, MCU 2134 may also include inputs from other sensors (eg, IMU sensor 2140 ) and power and battery module 2142 . The output of the processing performed by the MCU 2134 may be provided to the antenna 2150 for transmission to the adapter portion 2120 shown in FIG. 21B.

Adapter portion 2120 may include antenna 2152 that may be configured to communicate with antenna 2150 included as part of wearable portion 2110 . Communication between antennas 2150 and 2152 may occur using any suitable wireless technology and protocol, non-limiting examples of which include radio frequency signaling and Bluetooth. As shown, signals received by antenna 2152 of adapter portion 2120 may be provided to a host computer for further processing, display, and/or for enabling control of one or more specific physical or virtual objects.

Although the examples provided with reference to FIGS. 20A-20B and 21A-21B are discussed in the context of an interface with an EMG sensor, the techniques described herein for reducing electromagnetic interference can also be implemented with other types of In the wearable interface of sensors, including but not limited to myocardiography (MMG) sensors, sound myography (SMG) sensors and electrical impedance tomography (EIT) sensors. The techniques described herein for reducing electromagnetic interference can also be implemented in wearable interfaces that communicate with a computer host via wires and cables (eg, USB cables, fiber optic cables, etc.).

As detailed above, computing devices and systems described and/or illustrated herein broadly represent any type or form of computer-readable instructions capable of executing such instructions as those contained within the modules described herein. computing device or system. In their most basic configuration, these computing devices may each include at least one memory device and at least one physical processor.

In some instances, the term "memory device" generally refers to any type or form of volatile or non-volatile storage device or media capable of storing data and/or computer-readable instructions. In one example, a memory device can store, load and/or maintain one or more of the modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read only memory (ROM), flash memory, hard disk drives (HDD), solid state drives (SSD), optical drives, flash drives, Take memory, a variation or combination of one or more of the aforementioned memory devices, or any other suitable storage memory.

In some instances, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions. In one example, the physical processor can access and/or modify one or more modules stored in the aforementioned memory device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), the aforementioned A portion of one or more of the physical processors, a variation or combination of one or more of the aforementioned physical processors, or any other suitable physical processor.

Although illustrated as separate elements, modules described and/or illustrated herein may represent portions of a single module or application. Additionally, in some embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, one or more of the modules described and/or illustrated herein may represent storage on one or more of the computing devices or systems described and/or illustrated herein and configured to The mod to execute on. One or more of these modules may also represent all or part of one or more special purpose computers configured to perform one or more tasks.

Additionally, one or more of the modules described herein may transform data, physical devices, and/or representations of physical devices from one form to another. For example, one or more of the modules listed herein can receive the real-time output of the identification model to be transformed, convert the real-time output of the recognition model into an uncertainty level, and output the result of the transformation to a feedback system For subsequent communication with the user, and/or using the result of the transformation to modulate at least one attribute for presenting the user with appropriate and/or appropriate feedback. Additionally or alternatively, one or more of the modules enumerated herein may, by executing on, storing data on, and/or otherwise interacting with the computing device, offload the processor, volatilize volatile memory, non-volatile memory, and/or any other portion of a physical computing device from one form to another.

In some embodiments, the term "computer-readable medium" generally refers to any form of device, carrier or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission media, such as carrier waves; and non-transitory media, such as magnetic storage media (for example, hard drives, tape drives, and floppy disks), optical storage media (for example, optical discs (CD), digital video disc (DVD) and Blu-ray (BLU-RAY) disc), electronic storage media (such as solid-state drives and flash media) and other distribution systems.

Program parameters and sequence of steps described and/or illustrated herein are provided as examples only and may be varied as desired. For example, although steps illustrated and/or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps beyond those disclosed.

The foregoing description has been provided to enable those of ordinary skill in the art to best utilize various aspects of the illustrative embodiments disclosed herein. This illustrative description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the disclosure. The specific examples disclosed herein are to be considered in all respects as illustrative and not restrictive. In determining the scope of the disclosure, reference should be made to the appended claims and their equivalents.

Unless otherwise indicated, the terms "connected to" and "coupled to" (and their derivatives) as used in this specification and claims are to be interpreted as allowing direct and indirect (that is, via other elements or components) Connect the two. In addition, the term "one (a or an)" as used in this specification and claims shall be deemed to mean "at least one of them". Finally, for ease of use, the terms "comprising" and "having" (and their derivatives) as used in this specification and claims are interchangeable with the word "comprising" and have the same meanings.

100: system 102:Module 104: Receiving module 106: Judgment module 108:Modulation module 110: Presentation module 112: User input module 120: memory 130: entity processor 140: Identification Model 142:Pinch detection model 144:Manual tracking model 146: Intent Interaction Model 148: Cognitive state models 150: sensor 150(1): Sensor 150(N): sensor 200: data flow 202: Immediate output 204: Uncertainty level 206: Giving Back 302: Probability 304:Threshold value 306: Probability 308: Probability 310: Probability 312: Probability 400: computer implementation method 410: Step 420: Step 430: step 440: step 502: Uncertainty level 602: amplitude level 702: amplitude level 802: Uncertainty Threshold/Threshold 902: amplitude level 1002: amplitude level 1102: amplitude level 1200: Computer-implemented methods/methods 1210: step 1220: step 1230: step 1240: step 1250: step 1260: step 1270: step 1300: Augmented reality system/augmented reality device/system/artificial reality system 1302: glasses device 1305: neck strap 1310: frame 1315(A): left display device/display device 1315(B): Right Display Device/Display Device 1320(A): Acoustic energy converter 1320(B): Acoustic Energy Converter 1320(C): Acoustic Energy Converter 1320(D): Acoustic Energy Converter 1320(E): Acoustic energy converter 1320(F): Acoustic Energy Converter 1320(G): Acoustic Energy Converter 1320(H): Acoustic Energy Converter 1320(I): Acoustic energy converter 1320(J): Acoustic energy converter 1325: controller 1330: wired connection 1335: Power 1340: sensor 1350: controller 1400: Virtual Reality Systems/Systems/Artificial Reality Systems 1402: Front rigid body 1404: with 1406(A): Output Audio Converter 1406(B): Output Audio Converter 1500: Vibration tactile system 1510: Haptic devices 1520: Haptic devices 1530: Wearable Textile Materials/Textile Materials 1540: Vibration Tactile Device 1550: power supply 1552: Conductive Wiring 1560: Processor 1570: Another device or system 1580: communication interface 1590: Touch Sensitive Liners 1600: Artificial Reality Environment 1602: Head Mounted Display 1604: Haptic devices 1700: Augmented Reality System 1710: user 1720: Augmented Reality Glasses 1722:Display 1730: Haptic devices 1732: with components 1734: tensioning mechanism 1800: system 1801: eyes 1802: light source 1804: Optical Subsystems 1806: Eye Tracking Subsystem 1808: Control Subsystem 1820: converging light 1822: Pupil 1900: Eye Tracking Subsystem 1902: eyes 1903: Flash 1904: source 1905: Image 1906: Sensor 1908: pupil 1910: Flash 2000: Wearable systems 2010: Neuromuscular Sensors/Sensors 2020: Elastic Bands 2030: inner surface 2110: Wearable part 2111: Skin contact electrodes 2120: Adapter part 2130: Analog front end 2132: Analog to Digital Converter 2134: Microcontroller (MCU) 2140:IMU sensor 2142: Power and battery modules 2150: Antenna 2152: Antenna

The accompanying drawings depict several illustrative embodiments and are a part of this specification. Together with the description below, these drawings show and explain various principles of the disclosure.

[FIG. 1] is a block diagram of an example system for communicating model uncertainty to a user.

[FIG. 2] is a flowchart of an exemplary data flow for communicating model uncertainty to a user.

[ FIG. 3 ] is a diagram of an exemplary time series of probabilities.

[ FIG. 4 ] is a flowchart of an exemplary method for communicating model uncertainty to a user.

[ Fig. 5 ] is a diagram illustrating an exemplary relationship between an uncertainty level and a probability.

[FIG. 6] is a diagram of an exemplary uncertainty modulation feedback.

[FIG. 7] is another diagram of exemplary uncertainty modulation feedback.

[FIG. 8] is a graph of exemplary uncertainty thresholds.

[ FIG. 9 ] is a diagram illustrating exemplary uncertainty-based feedback.

[ FIG. 10 ] is another diagram illustrating exemplary uncertainty-based feedback.

[ FIG. 11 ] is another diagram illustrating exemplary uncertainty-based feedback.

[ FIG. 12 ] is a flowchart of another exemplary method for communicating model uncertainty to a user.

[ FIG. 13 ] is a diagram of exemplary augmented reality glasses that may be used in conjunction with embodiments of the present disclosure.

[ FIG. 14 ] is an illustration of an exemplary virtual reality headset that may be used in conjunction with embodiments of the present disclosure.

[ FIG. 15 ] is a diagram of an exemplary haptic device that may be used in conjunction with embodiments of the present disclosure.

[ FIG. 16 ] is a diagram of an exemplary virtual reality environment according to embodiments of the present disclosure.

[ FIG. 17 ] Is a diagram of an exemplary augmented reality environment according to embodiments of the present disclosure.

[ FIG. 18 ] is a diagram of an exemplary system incorporating an eye tracking subsystem capable of tracking a user's eyes.

[ FIG. 19 ] is a more detailed illustration of various aspects of the eye tracking subsystem depicted in FIG. 18 .

[ FIG. 20A ] and [ FIG. 20B ] are illustrations of an exemplary human-machine interface configured to be worn on a user's lower arm or wrist.

[ FIG. 21A ] and [ FIG. 21B ] are diagrams with illustrative schematic diagrams of the internal components of the wearable system.

Throughout the drawings, like reference numbers and descriptions indicate similar, but not necessarily identical, elements. While the illustrative embodiments described herein are susceptible to various modifications and alternative forms, certain embodiments have been shown by way of example in the drawings and will be described herein in detail. However, the illustrative embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

400: computer implementation method

410: Step

420: Step

430: step

440: step

Claims (20)

A computer-implemented method comprising: receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user; determining a level of uncertainty associated with the immediate output based on the information; modulating at least one attribute of feedback based on the uncertainty level; and The feedback is presented to the user substantially simultaneously with the immediate output of the recognition model. The computer implementation method of claim 1, wherein: the information associated with the real-time output of the recognition model includes a probability of the user performing the at least one behavior; The computer-implemented method further includes at least one of the following: performing a user input operation when the probability of the user performing the at least one action is above a predetermined threshold; or refraining from performing the user input operation when the probability of the user performing the at least one action is below the predetermined threshold; and The uncertainty level associated with the immediate output is determined based on the distance between the probability and the predetermined threshold, the uncertainty level being inversely proportional to the distance. The computer implementation method of claim 1, wherein: the information associated with the real-time output of the recognition model includes a probability of the user performing the at least one action; and The at least one attribute of the feedback is tuned to have a perceptibility level proportional to the probability that the user performs the at least one behavior. The computer-implemented method of claim 1, wherein the at least one attribute of the feedback is modulated to have a perceptibility level proportional to the level of uncertainty associated with the immediate output . The computer implementation method of claim 1, wherein: The recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; The information includes the probability that the user performed the pinch gesture; The computer-implemented method further includes performing a user input operation when the probability that the user performs the pinch gesture is above a predetermined threshold; and When the probability of the user performing the pinch gesture is higher than the predetermined threshold, the feedback is presented to the user while performing the user input operation. The computer implementation method of claim 1, wherein: The recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; The information includes the probability that the user performed the pinch gesture; The computer-implemented method further includes refraining from performing a user input operation when the probability that the user performs the pinch gesture is below a predetermined threshold; and The feedback is presented to the user while determining that the probability of the user performing the pinch gesture is below the predetermined threshold. The computer implementation method of claim 1, wherein: The identification model includes a manual tracking model adapted to output: location or orientation information for one or more parts of the user's hand; and the level of reliance on the location or orientation information; and The uncertainty level associated with the immediate output is determined based on the confidence level, the uncertainty level being inversely proportional to the confidence level. The computer implementation method of claim 1, wherein: the feedback is tactile feedback; and At least one property of the haptic feedback is based on the uncertainty level associated with the immediate output. The computer implementation method of claim 1, wherein: the feedback system vibrates; and At least one property of the vibration is based on the uncertainty level associated with the immediate output. The computer-implemented method of claim 1, wherein modulating the at least one attribute of the feedback based on the uncertainty level includes modulating one or more of the following: the magnitude of the feedback to communicate the level of uncertainty to the user; the frequency of the feedback to communicate the level of uncertainty to the user; the duration of the feedback to communicate the level of uncertainty to the user; the type of feedback to communicate the level of uncertainty to the user; and The feedback is spatialized to communicate the level of uncertainty to the user. The computer-implemented method of claim 1, wherein the feedback indicates a means for reducing the uncertainty level of the immediate output. The computer-implemented method of claim 1, which further includes: receiving additional information associated with the additional real-time output of the identification model; determining an additional level of uncertainty associated with the additional immediate output based on the additional information; and The at least one property of the feedback is modulated based on the additional uncertainty level. A system comprising: at least one physical processor; and Physical memory containing computer-executable instructions that, when executed by the at least one physical processor, cause the at least one physical processor to: receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user; determining a level of uncertainty associated with the immediate output based on the information; modulating at least one attribute of feedback based on the uncertainty level; and The feedback is presented to the user substantially simultaneously with the immediate output of the recognition model. The system of claim 13, wherein: the information associated with the real-time output of the recognition model includes a probability of the user performing the at least one behavior; The computer-executable instructions, when executed by the at least one physical processor, further cause the at least one physical processor to: performing a user input operation when the probability of the user performing the at least one action is above a predetermined threshold; or refraining from performing the user input operation when the probability of the user performing the at least one action is below the predetermined threshold; and The uncertainty level associated with the immediate output is determined based on the distance between the probability and the predetermined threshold, the uncertainty level being inversely proportional to the distance. The system of claim 13, wherein: the information associated with the real-time output of the recognition model includes a probability of the user performing the at least one action; and The at least one attribute of the feedback is tuned to have a perceptibility level proportional to the probability that the user performs the at least one behavior. The system of claim 13, wherein the at least one attribute of the feedback is modulated to have a perceptibility level proportional to the level of uncertainty associated with the immediate output. The system of claim 13, wherein: The recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; The information includes a probability that the user performed the pinch gesture; The computer-executable instructions, when executed by the at least one physical processor, further cause the at least one physical processor to perform a user input operation when the probability that the user performs the pinch gesture is above a predetermined threshold; and When the probability of the user performing the pinch gesture is higher than a predetermined threshold, the feedback is presented to the user while performing the user input operation. The system of claim 13, wherein: The recognition model includes a pinch recognition model adapted to output a probability that the user performs a pinch gesture; The information includes the probability that the user performed the pinch gesture; The computer-executable instructions, when executed by the at least one physical processor, further cause the at least one physical processor to refrain from performing a user input operation when the probability that the user performs the pinch gesture is below a predetermined threshold; and The feedback is presented to the user while determining that the probability of the user performing the pinch gesture is below the predetermined threshold. The system of claim 13, wherein: The identification model includes a manual tracking model adapted to output: location or orientation information for one or more parts of the user's hand; and the level of reliance on the location or orientation information; and The uncertainty level associated with the immediate output is determined based on the confidence level, the uncertainty level being inversely proportional to the confidence level. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user; determining a level of uncertainty associated with the immediate output based on the information; modulating at least one attribute of feedback based on the uncertainty level; and The feedback is presented to the user substantially simultaneously with the immediate output of the recognition model.

Images