Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/012—Head tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user

Definitions

• the present invention refers to a computer-implemented method according to claim 1, a system according to claim 13, a non-transitory computer-readable medium according to claim 14 and a software product according to claim 15.
• Advantageous embodiments may include features of depending claims.
• a computer-implemented method comprises receiving information associated with a real-time output of a recognition model, the recognition model being adapted to recognize at least one behavior of a user.
• the method further comprises determining, based on the information, a level of uncertainty associated with the real-time output. At least one attribute of feedback is modulated based on the level of uncertainty and the feedback is presented to the user substantially contemporaneous with the real-time output of the recognition model.
• the information associated with the real-time output of the recognition model may comprise a probability of the user performing the behavior.
• the computer-implemented method may further comprise at least one of performing a user-input operation when the probability of the user performing the behavior is above a predetermined threshold, or refraining from performing the user-input operation when the probability of the user performing the behavior is below the predetermined threshold.
• the level of uncertainty associated with the real-time output may be determined based on a distance between the probability and the predetermined threshold, the level of uncertainty being inversely proportional to the distance.
• the information associated with the real-time output of the recognition model may comprise a probability of the user performing the behavior and the attribute of the feedback may be modulated to have a level of perceivability proportional to the probability of the user performing the behavior.
• the attribute of the feedback may be modulated to have a level of perceivability proportional to the level of uncertainty associated with the real- time output.
• the recognition model may comprise a pinch recognition model adapted to output probabilities of the user performing a pinch gesture.
• the information may comprise a probability of the user performing the pinch gesture.
• the computer-implemented method may further comprise performing a user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold and the feedback may be presented to the user contemporaneous with performing the user-input operation when the probability of the user performing the pinch gesture is above the predetermined threshold.
• the recognition model may comprise a pinch recognition model adapted to output probabilities of the user performing a pinch gesture.
• the information may comprise a probability of the user performing the pinch gesture.
• the computer-implemented method may further comprise refraining from performing a userinput operation when the probability of the user performing the pinch gesture is below a predetermined threshold and the feedback may be presented to the user contemporaneous with determining that the probability of the user performing the pinch gesture is below the predetermined threshold.
• the recognition model may comprise a handtracking model adapted to output position or orientation information for one or more portions of the user's hand and a confidence level of the position or orientation information.
• the level of uncertainty associated with the real-time output may be determined based on the confidence level, the level of uncertainty being inversely proportional to the confidence level.
• the feedback may be haptic feedback and at least one attribute of the haptic feedback may be based on the level of uncertainty associated with the real-time output.
• the feedback may be a vibration and at least one attribute of the vibration may be based on the level of uncertainty associated with the realtime output.
• modulating the attribute of the feedback based on the level of uncertainty may comprise modulating one or more of an amplitude of the feedback to convey the level of uncertainty to the user, a frequency of the feedback to convey the level of uncertainty to the user, a duration of the feedback to convey the level of uncertainty to the user, a pattern of the feedback to convey the level of uncertainty to the user and a spatialization of the feedback to convey the level of uncertainty to the user.
• the feedback may indicate a method for reducing the level of uncertainty of the real-time output.
• the computer-implemented method may further comprise receiving additional information associated with an additional real-time output of the recognition model, determining, based on the additional information, an additional level of uncertainty associated with the additional real-time output and modulating the attribute of the feedback based on the additional level of uncertainty.
• a system comprises at least one physical processor and physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to carry out any one of the methods described above or to receive information associated with a real-time output of a recognition model, the recognition model being adapted to recognize at least one behavior of a user, determine, based on the information, a level of uncertainty associated with the realtime output, modulate at least one attribute of feedback based on the level of uncertainty and present the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• the information associated with the real-time output of the recognition model may comprise a probability of the user performing the behavior and the computer-executable instructions, when executed by the physical processor, further cause the physical processor to perform a user-input operation when the probability of the user performing the behavior is above a predetermined threshold or to refrain from performing the user-input operation when the probability of the user performing the behavior is below the predetermined threshold, wherein the level of uncertainty associated with the real-time output may be determined based on a distance between the probability and the predetermined threshold, the level of uncertainty being inversely proportional to the distance.
• the information associated with the real-time output of the recognition model may comprise a probability of the user performing the behavior and the attribute of the feedback may be modulated to have a level of perceivability proportional to the probability of the user performing the behavior.
• the attribute of the feedback may be modulated to have a level of perceivability proportional to the level of uncertainty associated with the real-time output.
• the recognition model may comprise a pinch recognition model adapted to output probabilities of the user performing a pinch gesture and the information may comprise a probability of the user performing the pinch gesture.
• the computer-executable instructions when executed by the physical processor, may further cause the physical processor to perform a user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold and the feedback may be presented to the user contemporaneous with performing the user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold.
• the recognition model may comprise a pinch recognition model adapted to output probabilities of the user performing a pinch gesture and the information may comprise a probability of the user performing the pinch gesture.
• the computer-executable instructions when executed by the physical processor, may further cause the physical processor to refrain from performing a user-input operation when the probability of the user performing the pinch gesture is below a predetermined threshold and the feedback may be presented to the user contemporaneous with determining that the probability of the user performing the pinch gesture is below the predetermined threshold.
• the recognition model may comprise a hand-tracking model adapted to output position or orientation information for one or more portions of the user's hand and a confidence level of the position or orientation information.
• the level of uncertainty associated with the real-time output may be determined based on the confidence level, the level of uncertainty being inversely proportional to the confidence level.
• a non-transitory computer-readable medium comprises one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to carry out any one of the methods described above or to receive information associated with a real-time output of a recognition model, the recognition model being adapted to recognize at least one behavior of a user, determine, based on the information, a level of uncertainty associated with the real-time output, modulate at least one attribute of feedback based on the level of uncertainty and present the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• a software product comprises one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to carry out any one of the methods described above or to receive information associated with a real-time output of a recognition model, the recognition model being adapted to recognize at least one behavior of a user, determine, based on the information, a level of uncertainty associated with the real-time output, modulate at least one attribute of feedback based on the level of uncertainty and present the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• FIG. 1 is a block diagram of an example system for communicating model uncertainty to users.
• FIG. 2 is a flow diagram of an exemplary data flow for communicating model uncertainty to users.
• FIG. 3 is a diagram of an exemplary time series of probabilities.
• FIG. 4 is a flow diagram of an exemplary method for communicating model uncertainty to users.
• FIG. 5 is a diagram of an exemplary relationship between levels of uncertainty and probabilities.
• FIG. 6 is a diagram of exemplary uncertainty-modulated feedback.
• FIG. 7 is another diagram of exemplary uncertainty-modulated feedback.
• FIG. 8 is a diagram of exemplary uncertainty thresholds.
• FIG. 9 is a diagram of exemplary uncertainty-based feedback.
• FIG. 10 is another diagram of exemplary uncertainty-based feedback.
• FIG. 11 is another diagram of exemplary uncertainty-based feedback.
• FIG. 12 is a flow diagram of another exemplary method for communicating model uncertainty to users.
• FIG. 13 is an illustration of exemplary augmented-reality glasses that may be used in connection with embodiments of this disclosure.
• FIG. 14 is an illustration of an exemplary virtual-reality headset that may be used in connection with embodiments of this disclosure.
• FIG. 15 is an illustration of exemplary haptic devices that may be used in connection with embodiments of this disclosure.
• FIG. 16 is an illustration of an exemplary virtual-reality environment according to embodiments of this disclosure.
• FIG. 17 is an illustration of an exemplary augmented-reality environment according to embodiments of this disclosure.
• FIG. 18 an illustration of an exemplary system that incorporates an eyetracking subsystem capable of tracking a user's eye(s).
• FIG. 19 is a more detailed illustration of various aspects of the eye-tracking subsystem illustrated in FIG. 18.
• FIGS. 20A and 20B are illustrations of an exemplary human-machine interface configured to be worn around a user's lower arm or wrist.
• FIGS. 21A and 21B are illustrations of an exemplary schematic diagram with internal components of a wearable system.
• Users interact with computing systems in a variety of ways to provide input to, or otherwise control, actions performed by the computing systems.
• Users may interact directly with computing systems using simple physical user-input devices that provide reliable discretized outputs (e.g., key or button press events) to the computing systems.
• These simple physical user-input devices may be capable of independently providing users with actionable feedback to the users' behaviors.
• users may provide text input to computing systems using physical keyboards whose keys provide tactile resistance to movement at the points at which they generate discrete outputs.
• Physical user-input devices that are capable of independently providing a user with immediate feedback may enable the user to confidently provide input to a computing system without needing to verify that each of the user's inputs are properly recognized and responded to by the computing system.
• physical keyboards may enable many users to provide character input quickly and reliably to a computing system without looking at the keys they are pressing and/or without verifying that individual characters are correctly received and processed by the computing system.
• computing systems may use virtual user-input devices such as virtual keyboards and/or recognition models capable of detecting and reacting to certain user behaviors.
• recognition models such as gesture recognizers, may output a probability or likelihood that a user performed or is performing a behavior.
• a computing system may discretize the probability, typically using a threshold, to determine the onset or occurrence of the behavior. For example, a computing system may use a threshold probability of 0.6 to distinguish occurrences of a behavior from nonoccurrences of the behavior. In cases where the probability of an occurrence of a behavior is very high (e.g., 0.9), the computing system may be relatively certain that the behavior occurred.
• the computing system may be relatively certain that the behavior did not occur. However, the computing system may be less certain of probabilities that range around the threshold (e.g., probabilities in the range of 0.5-0.7), which may lead to false-positive and false-negative detections and sporadic mismatches of system responses to users' behaviors.
• the present disclosure is generally directed to communicating model uncertainty and related information to users using various forms of actionable real-time feedback.
• the systems and methods described herein may use realtime haptic feedback (e.g., modulated vibrations) and/or other types of nonvisual feedback to communicate levels of uncertainty associated with the outputs of recognition models (e.g., recognition models adapted to detect point and select behaviors).
• recognition models e.g., recognition models adapted to detect point and select behaviors.
• Embodiments of the present disclosure may enable a user to understand, in real time, when a computing system is most likely to react correctly to the user's behaviors, which may allow the user to confidently provide input to the computing system with less or no hesitation.
• Embodiments of the present disclosure may also enable a user to understand, in real time, when the computing system is most likely to react incorrectly to the user's behaviors, which may cause the user to modify certain behaviors to be better recognized by the computing system.
• feedback presented to the user may indicate how the user can modify their behaviors to be better recognized by the computing system.
• embodiments of the present disclosure may enable a user to better understand how a computing system will interpret the user's behaviors, which may reduce the number of recognition errors made by the computing system and lead to improved user-interaction experiences and performance.
• FIG. 1 is a block diagram of an example system 100 for communicating model uncertainty to users.
• example system 100 may include one or more modules 102 for performing one or more tasks.
• modules 102 may include a receiving module 104 that receives information associated with real-time outputs of a recognition model.
• Example system 100 may also include a determining module 106 that determines, based on information received by receiving module 104, levels of uncertainty associated with the real-time outputs of the recognition model.
• Example system 100 may also include a modulating module 108 that modulates various attributes of feedback based on levels of uncertainty.
• example system 100 may include a presenting module 110 that presents feedback to users substantially contemporaneous with the real-time outputs of the recognition model.
• example system 100 may also include a user-input module 112 that performs, or refrains from performing, reactive user-input operations in response to real-time outputs (e.g., recognition events) of the recognition model.
• example system 100 may also include one or more memory devices, such as memory 120.
• Memory 120 may include or represent any type or form of volatile or non-volatile storage device or medium capable of storing data and/or computer-readable instructions.
• memory 120 may store, load, and/or maintain one or more of modules 102.
• Examples of memory 120 include, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory.
• example system 100 may also include one or more physical processors, such as physical processor 130.
• Physical processor 130 may include or represent any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions.
• physical processor 130 may access and/or modify one or more of modules 102 stored in memory 120. Additionally or alternatively, physical processor 130 may execute one or more of modules 102 to facilitate the communication of model uncertainties to users.
• Examples of physical processor 130 include, without limitation, microprocessors, microcontrollers, central processing units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.
• CPUs central processing units
• FPGAs Field-Programmable Gate Arrays
• ASICs Application-Specific Integrated Circuits
• example system 100 may also include one or more recognition models, such as recognition model(s) 140.
• Recognition model(s) 140 may represent or include any machine-learning model, algorithm, heuristic, data, or combination thereof, that may be used to recognize, detect, calculate, estimate, predict, label, infer, and/or react to user behaviors (e.g., actions, cognitive states, intentions, etc.) and/or effects caused by user behaviors (e.g., device, headset, or controller movements).
• user behaviors e.g., actions, cognitive states, intentions, etc.
• effects caused by user behaviors e.g., device, headset, or controller movements.
• recognition model(s) 140 include, without limitation, pinch-detection models (e.g., a pinchdetection model 142) adapted to recognize and/or react to pinch gestures, hand-tracking models (e.g., a hand-tracking model 144) adapted to recognize and/or react to hand states and/or hand movement, models adapted to recognize and/or react to users' intentions to interact (e.g., an intent-to-interact model 146), models adapted to recognize and/or react to transitions between users' cognitive states (e.g., a cognitive-state model 148), gesturerecognition models, eye-tracking models, speech-recognition models, body-sensing models, body-tracking models, physiological-state models adapted to recognize and/or react to users' physiological states, psychological-state models adapted to recognize and/or react to users' psychological states, models for predicting and reacting to users' intents to encode, models adapted to perform inside-out tracking of computing devices, portions of one or more of
• recognition model(s) 140 may include, without limitation, decision trees (e.g., boosting decision trees), neural networks (e.g., a deep convolutional neural network), deep-learning models, support vector machines, linear classifiers, non-linear classifiers, perceptrons, naive Bayes classifiers, any other machinelearning or classification techniques or algorithms, or any combination thereof.
• decision trees e.g., boosting decision trees
• neural networks e.g., a deep convolutional neural network
• deep-learning models e.g., support vector machines, linear classifiers, non-linear classifiers, perceptrons, naive Bayes classifiers, any other machinelearning or classification techniques or algorithms, or any combination thereof.
• the systems described herein may use one or more of recognition model(s) 140 to recognize and/or react to a user's pointing gestures, selections, or interactions relating to graphical elements displayed to the user via system 100 (e.g., an element of a graphical user interface or an element rendered from a virtual-reality or augmented-reality environment).
• recognition model(s) 140 may use pinchdetection model 142 to detect the moment at which a user intends to make a selection and may use hand-tracking model 144 to determine the graphical element the user is most likely attempting to select.
• system 100 may include one or more sensors 150 for gathering real-time measurements indicative of user behaviors.
• Recognition model(s) 140 may use information derived from the measurements gathered from sensors 150 to recognize user behaviors in real time.
• sensors 150 include, without limitation, touch sensors, image sensors, proximity sensors, biometric sensors, inertial measurement units, biosensors, heart-rate sensors, saturated-oxygen sensors, neuromuscular sensors, altimeter sensors, temperature sensors, bioimpedance sensors, pedometer sensors, optical sensors, sweat sensors, variations or combinations of one or more of the same, or any other type or form of sensing hardware or software.
• FIG. 2 is a diagram of an exemplary data flow 200 for communicating levels of uncertainty associated with recognition model(s) 140 to users using one or more of the elements of system 100.
• one or more of recognition model(s) 140 may take input from one or more of sensors 150 to generate one or more real-time output(s) 202 (e.g., recognition events) indicative and/or predictive of one or more user behaviors.
• receiving module 104 and user-input module 112 may monitor and/or react to real-time output(s) 202.
• receiving module 104 may monitor some or all of the inputs to recognition model(s) 140 (e.g., measurements of sensor(s) 150) that may be used by recognition model(s) 140 to generate real-time output(s) 202.
• real-time output(s) 202 may represent a time series (e.g., a time series of probabilities of one or more user behaviors, such as probabilities 302 illustrated in FIG. 3).
• recognition model(s) 140 may be adapted to recognize, detect, calculate, estimate, predict, label, infer, and/or react to one or more user behaviors, and real-time output(s) 202 may represent or include any information indicative of an occurrence or a nonoccurrence of the one or more user behaviors.
• realtime output(s) 202 may include or represent a probability or likelihood that a user is performing a particular behavior (e.g., a pinch or pointing gesture).
• recognition model(s) 140 may be adapted to recognize many user behaviors, and real-time output(s) 202 may include or represent a label or classification of a behavior that the user is likely to be performing and/or a confidence level or score associated with the label or classification.
• recognition model(s) 140 may be adapted to estimate, predict, and/or infer various attributes of one or more user behaviors, and real-time output(s) 202 may represent or include one or more estimated attributes of the behavior (e.g., a predicted position, orientation, or state) and/or a confidence level or score associated with the estimated attributes.
• user-input module 112 may be adapted to perform userinput operations and/or otherwise respond to associated user behaviors indicated by realtime output(s) 202.
• Examples of user-input operations that may be performed by user-input module 112 in response to real-time output(s) 202 include, without limitation, updating a cursor's position based on a user behavior indicated by real-time output(s) 202 (e.g., a pointing gesture), making a selection based on a user behavior indicated by real-time output(s) 202 (e.g., a pinch gesture), updating a state of a virtual character or environment to reflect a user behavior indicated by real-time output(s) 202, displaying user input associated with a user behavior indicated by real-time output(s) 202 via a graphical user interface element, processing user input associated with a user behavior indicated by real-time output(s) 202, storing user input associated with a user behavior indicated by real-time output(s) 202,
• receiving module 104 may monitor real-time output(s) 202 and/or any information associated with recognition model(s) 140 and/or real-time output(s) 202 that may be directly and/or indirectly indicative of uncertainties relating to realtime output(s) 202 and/or the interpretations of real-time output(s) 202.
• receiving module 104 may receive information associated with sensors 150 that may be indicative of the reliability of the measurements of sensors 150 and/or information associated with real-time output(s) 202 (e.g., a probability or confidence level).
• Determining module 106 may use information gathered by receiving module 104 to determine or calculate one or more uncertainty level(s) 204 associated with real-time output(s) 202.
• Modulating module 108 may then use uncertainty level(s) 204 to modulate at least one attribute of appropriate feedback to be presented by presenting module 110.
• real-time output(s) 202 of recognition model(s) 140 may include a time series of probabilities indicative of occurrences and/or nonoccurrences of certain user behaviors.
• FIG. 3 is an illustration of an exemplary time series of probabilities 302.
• Each of probabilities 302 may represent a probability that a user is performing a behavior (e.g., a pinch gesture) at a corresponding moment in time.
• the disclosed systems may use a threshold 304 (e.g., 0.6) to distinguish occurrences of the behavior from nonoccurrences of the behavior.
• the disclosed systems may treat probabilities greater than threshold 304 as occurrences of the behavior and probabilities less than threshold 304 as nonoccurrences of the behavior.
• the disclosed systems may be relatively certain that the behavior occurred and may react accordingly.
• a probability is very low (e.g., probability 308)
• the disclosed systems may be relatively certain that the behavior did not occur and may react accordingly.
• the disclosed systems may be less certain of probabilities that range around threshold 304 (e.g., probabilities 310 and 312) and may react in a way that conflicts with a user's intentions. For example, the disclosed systems may react to probability 310 as if the user is performing the behavior even if the user is not actually performing the behavior.
• the disclosed systems may react to probability 312 as if the user is not performing the behavior even if the user is performing the behavior.
• the disclosed systems may use various forms of feedback to convey levels of uncertainty associated with probabilities 302 to the user, which may cause the user to adapt subsequent behavior such that future levels of uncertainty associated with the behavior are reduced.
• FIG. 4 is a flow diagram of an exemplary computer-implemented method 400 for communicating model uncertainty to users.
• the steps shown in FIG. 4 may be performed by any suitable computer-executable code and/or computing system, including the system(s) illustrated in FIG. 1.
• each of the steps shown in FIG. 4 may represent an algorithm whose structure includes and/or is represented by multiple sub-steps, examples of which will be provided in greater detail below.
• the steps of computer-implemented method 400 may be performed repeatedly.
• the steps of computer-implemented method 400 may be performed once for each of a recognition model's real-time outputs.
• one or more of the systems described herein may receive information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user.
• receiving module 104 may receive some or all of the information associated with real-time output(s) 202, as described in connection with FIG. 2.
• the systems described herein may receive and/or monitor various types of information that may be indicative of an uncertainty associated with a real-time output of a recognition model.
• a recognition model may output indications of the uncertainties associated with its outputs (e.g., probabilities, certainty levels, confidence scores, etc.), and receiving module 104 may monitor this information in real time.
• certain characteristics e.g., a signal-to-noise ratio
• the disclosed systems may use one or more thresholds to discretize and/or interpret a recognition model's outputs.
• user-input module 112 may use threshold 304 to discretize and/or interpret probabilities 302.
• a threshold may influence and/or be indicative of a level of uncertainty associated with a recognition model's real-time outputs. For at least this reason, receiving module 104 may identify any thresholds used to discretize and/or interpret a recognition model's outputs.
• one or more of the systems described herein may determine, based on the information received at step 410, a level of uncertainty associated with the realtime output. For example, determining module 106 may determine uncertainty level(s) 204 based on information received by receiving module 104, as shown in FIG. 2.
• a recognition model may output metrics of uncertainty
• the disclosed systems may determine a level of uncertainty associated with a real-time output from the recognition model itself. For example, when a real-time output includes a probability, a certainty level, or a confidence score, the disclosed systems may determine a level of uncertainty based on the probability, the certainty level, or the confidence score.
• the disclosed systems may generate an uncertainty level for the probability that is inversely proportional to the probability (i.e., high probabilities may be associated with low uncertainty levels and low probabilities may be associated with high uncertainty levels).
• the disclosed systems may generate an uncertainty level for the realtime output that is inversely proportional to the certainty level or the confidence score (i.e., high certainty levels or confidence scores may be associated with low uncertainty levels and low certainty levels or confidence scores may be associated with high uncertainty levels).
• probabilities may reflect a likelihood that a user behavior is or is not occurring, and the disclosed systems may use one or more threshold probabilities (e.g., threshold 304 in FIG. 3) to differentiate probabilities that will be interpreted as occurrences of the user behavior from probabilities that will be interpreted as nonoccurrences of the user behavior.
• the disclosed systems may use the threshold to determine a level of uncertainty associated with a probability. For example, the disclosed systems may generate a level of uncertainty for a probability that is inversely proportional to a distance between the probability and the threshold (i.e., probabilities furthest from the threshold may be associated with low uncertainty levels and probabilities closest to the threshold may be associated with high uncertainty levels).
• the disclosed systems may determine an uncertainty level for a probability as shown in Equation (1):
• FIG. 5 illustrates exemplary uncertainty levels 502, which may be derived using Equation (1) and threshold 304 from FIG. 3.
• the uncertainty level associated with each probability may be inversely proportional to the distance between the probability and threshold 304.
• the probabilities associated with the highest uncertainty levels are those closest to threshold 304, and the probabilities associated with the lowest uncertainty levels are those furthest from threshold 304.
• certain characteristics e.g., a signal-to-noise ratio
• the disclosed systems may generate or adjust a level of uncertainty for a real-time output based at least in part on such characteristics and their contributions to uncertainty.
• one or more of the systems described herein may modulate at least one attribute of feedback based on the level of uncertainty determined at step 420.
• modulating module 108 may modulate at least one attribute of feedback 206 based on uncertainty level(s) 204 associated with real-time output(s) 202.
• the systems described herein may use various forms of sensory feedback to indicate and/or convey levels of uncertainty associated with the real-time outputs of recognition models and/or to attempt to cause users to improve levels of uncertainty associated with the real-time outputs of recognition models.
• the disclosed systems may use any suitable type or form of haptic feedback such as vibrations, forces, pressures, tractions, textures, and/or temperatures to communicate information about levels of uncertainty.
• the disclosed systems may use any suitable type or form of nonvisual feedback, such as audio feedback, to communicate information about levels of uncertainty.
• the disclosed systems may use a form of visual feedback to communicate information about levels of uncertainty.
• the systems described herein may modulate various attributes of feedback to communicate information about levels of uncertainty.
• the disclosed systems may modulate feedback amplitudes, feedback frequencies, feedback durations, feedback patterns, feedback spatializations, feedback locations, feedback strengths, feedback effects, feedback intensities, variations or combinations of one or more of the same, or any other perceptible attributes of feedback.
• the systems described herein may modulate attributes of feedback based on the uncertainty levels associated with a recognition model's real-time outputs in a variety of ways.
• the disclosed systems may modulate an attribute of feedback by setting, adjusting, tuning, or varying the attribute and/or the perceivability of the attribute to substantially reflect or otherwise convey an uncertainty level associated with a recognition model's real-time output and/or to reflect or otherwise convey changes in the uncertainty levels associated with the recognition model's real-time outputs.
• the disclosed systems may use secondary, tertiary, quaternary, or quinary feedback channels to communicate uncertainty levels. For example, in cases where a visual feedback channel is primarily used to provide feedback to a user, the disclosed systems may use an auditory feedback channel or a haptic feedback channel to communicate uncertainty levels. In some examples, the disclosed systems may select feedback that may be subtle and/or less distracting than other forms of primary feedback communicated to users via primary feedback channels.
• the disclosed systems may associate one or more types or forms of feedback with one or more types or forms of model uncertainty.
• the disclosed systems may associate a single type or form of feedback (e.g., a single type of vibration or a form of vibration) with uncertainties related to recognized occurrences and nonoccurrences of a user behavior.
• the disclosed systems may modulate one or more attributes of the feedback to convey levels of uncertainty.
• the disclosed systems may associate a first type or form of feedback with uncertainties related to recognized occurrences of the behavior and may associate a second type or form of feedback with uncertainties related to nonoccurrences of the behavior.
• the disclosed systems may modulate one or more attributes of the first type of feedback to convey levels of uncertainty related to occurrences of the behavior and/or may modulate one or more attributes of the second type of feedback to convey levels of uncertainty related to nonoccurrences of the behavior.
• the disclosed systems may provide users with instructions on how to reduce uncertainties associated with the real-time outputs of a recognition model before presenting any feedback to the users.
• the disclosed systems may associate instructions for each method with a different type of feedback such that the users may quickly know the appropriate method to apply when feedback is received.
• the disclosed systems may modulate one or more attributes of feedback to convey methods for reducing levels of uncertainty of the real-time outputs of a recognition model. For example, if a user's hand must be kept within view of a head-mounted camera system to be tracked, the disclosed systems may modulate one or more attributes of haptic feedback presented via a wrist-worn device whenever the user's hand begins to drift from view of the head-mounted camera system to remind the user to return their hand to a more ideal position for tracking. In at least one example, the disclosed systems may modulate the location of the haptic feedback to indicate the direction the user should move their hand to return it to the more ideal position for tracking.
• one or more of the systems described herein may present the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• presenting module 110 may present feedback 206 to a user substantially contemporaneous with one of real-time output(s) 202.
• the systems described herein may present feedback to a user contemporaneous with the real-time outputs of a recognition model in a variety of ways.
• the disclosed systems may present the feedback to a user substantially contemporaneous with the performance of a user-input operation that is made in response to a real-time output of the recognition model. If a real-time output of the recognition model indicates that a user-input operation will not be performed, the disclosed systems may present the feedback to the user substantially contemporaneous with the determination that the real-time output indicates that the user-input operation will not be performed.
• FIGS. 6-11 illustrate various exemplary ways in which the disclosed systems may modulate feedback amplitude to communicate uncertainty levels 502, which may be associated with probabilities indicative of occurrences and/or nonoccurrences of a user behavior. These examples are provided in terms of amplitude modulation and probabilities for ease of understanding. It should be noted that the disclosed systems may modulate any of the feedback attributes described above in the same or a similar manner to communicate uncertainty levels associated with any type or form of recognition-model output.
• FIG. 6 illustrates an exemplary proportional relationship between exemplary amplitude levels 602 and uncertainty levels 502.
• the disclosed systems may range from outputting low amplitude feedback in response to probabilities associated with low uncertainty levels (e.g., probabilities furthest from threshold 304) to outputting high amplitude feedback in response to probabilities associated with high uncertainty levels (e.g., probabilities closest to threshold 304).
• a user may receive the most noticeable feedback when the disclosed systems are encountering the most uncertain recognition-model outputs.
• FIG. 7 illustrates an exemplary relationship between exemplary amplitude levels 702 and uncertainty levels 502.
• the disclosed systems may modulate amplitude levels 702 differently depending on whether probabilities will be interpreted as occurrences or nonoccurrences of a user behavior.
• the disclosed systems may range from outputting low amplitude feedback in response to probabilities associated with low uncertainty levels (e.g., probabilities furthest from threshold 304) to outputting high amplitude feedback in response to probabilities associated with high uncertainty levels (e.g., probabilities closest to threshold 304).
• the disclosed systems may output high amplitude feedback regardless of the uncertainty level associated with the probabilities.
• a user may receive the most noticeable feedback when the user is most likely to be attempting to perform the user behavior.
• FIGS. 8-11 illustrate various exemplary ways in which the disclosed systems may modulate feedback amplitude based on whether an encountered uncertainty level is above or below an uncertainty threshold 802.
• FIG. 9 illustrates an exemplary relationship between exemplary amplitude levels 902 and uncertainty levels 502.
• the disclosed systems may modulate amplitude levels 902 differently depending on whether probabilities are associated with uncertainty levels above or below threshold 802.
• the disclosed systems may output no feedback.
• the disclosed systems may output high amplitude feedback regardless of the uncertainty level associated with the probability.
• a user may receive noticeable feedback when the disclosed systems are encountering the most uncertain recognition-model outputs.
• FIG. 10 illustrates an exemplary relationship between exemplary amplitude levels 1002 and uncertainty levels 502.
• the disclosed systems may modulate amplitude levels 1002 differently depending on whether probabilities are above threshold 304 and/or whether the probabilities are associated with uncertainty levels above or below threshold 802.
• the disclosed systems may output no feedback.
• the disclosed systems may range from outputting lower amplitude feedback in response to lower probabilities to outputting higher amplitude feedback in response to higher probabilities.
• a user may receive the most noticeable feedback when the user is most likely to be attempting to perform the user behavior.
• FIG. 11 illustrates an exemplary proportional relationship between exemplary amplitude levels 1102 and uncertainty levels 502.
• the disclosed systems may modulate amplitude levels 1102 differently depending on whether probabilities are associated with uncertainty levels above or below threshold 802.
• the disclosed systems may output no feedback.
• the disclosed systems may range from outputting low amplitude feedback in response to probabilities associated with low uncertainty levels (e.g., probabilities furthest from threshold 304) to outputting high amplitude feedback in response to probabilities associated with high uncertainty levels (e.g., probabilities closest to threshold 304).
• a user may receive the most noticeable feedback when the disclosed systems are encountering the most uncertain recognition-model outputs but may not be bothered with feedback when the disclosed systems are encountering the most certain recognition-model outputs.
• FIG. 12 is a flow diagram of an exemplary computer-implemented method 1200 for communicating probability-based uncertainty levels to users contemporaneous with reacting to user behaviors.
• the steps shown in FIG. 12 may be performed by any suitable computer-executable code and/or computing system, including the system(s) illustrated in FIG. 1.
• each of the steps shown in FIG. 12 may represent an algorithm whose structure includes and/or is represented by multiple sub-steps, examples of which will be provided in greater detail below.
• Some of the steps shown in FIG. 12 are similar to the steps shown in FIG. 4, therefore, the discussion of the steps shown in FIG. 4 may also apply to the steps shown in FIG. 12.
• one or more of the systems described herein may receive, from a recognition model, a probability of an occurrence of a user behavior.
• receiving module 104 may receive one of probabilities 306-312 illustrated in FIG. 3.
• one or more of the systems described herein may next determine a level of uncertainty based on the probability received at step 1210.
• determining module 106 may determine an uncertainty level of the probability received at step 1210 using Equation (1) as described above. For example, determining module 106 may determine relatively lower uncertainty levels for probabilities 306 and 308 and/or relatively higher uncertainty levels for probabilities 310 and 312.
• one or more of the systems described herein may then modulate at least one attribute of feedback based on the level of uncertainty determined at step 1220.
• modulating module 108 may modulate at least one attribute of haptic feedback to convey the level of uncertainty determined at step 1220 (e.g., lower amplitude vibrations may convey the relatively lower uncertainty levels of probabilities 306 and 308 or higher amplitude vibrations may convey the relatively higher uncertainty levels of probabilities 310 and 312).
• one or more of the systems described herein may determine if a probability received at step 1210 is above a predetermined threshold. For example, determining module 106 may determine if a probability received at step 1210 is above or below threshold 304.
• step 1210 If a probability received at step 1210 is determined to be above the threshold (e.g., as may occur with probabilities 306 and 310), execution of method 1200 may continue at step 1250 where the disclosed systems may react as if the behavior was detected by performing a user-input operation associated with the behavior. On the other hand, if the probability received at step 1210 is determined to be below the threshold (e.g., as may occur with probabilities 308 and 312), execution of method 1200 may continue at step 1260 where the disclosed systems may react as if the associated behavior was not detected by refraining from performing the user-input operation associated with the behavior. Finally, at step 1270 one or more of the systems described herein may present the feedback modulated at step 1230 to the user substantially contemporaneous with the performance of step 1250 or step 1260.
• a pinch detection model may output a value indicating the probability that a user is performing a pinch gesture.
• the disclosed systems may react as if the user is performing the pinch gesture when an outputted probability is within an upper range and/or may react as if the user is not performing the pinch gesture when an outputted probability is within a lower range.
• the disclosed systems may be less certain or uncertain of whether the user is performing the pinch gesture.
• the disclose systems may present haptic feedback to the user that has been modulated to indicate the system's uncertainty/level of uncertainty.
• a hand tracking model may estimate the position of a user's hand and may output a value indicating the certainty or uncertainty of any estimated hand positions.
• the disclosed systems may communicate the certainty or uncertainty of the estimated hand positions to the user via modulated haptic feedback.
• the disclosed systems may reduce user frustration and/or may cause users to adapt their behaviors in ways that lead to improved model predictions and user experiences.
• a computer-implemented method for communicating model uncertainty may include (1) receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user, (2) determining, based on the information, a level of uncertainty associated with the real-time output, (3) modulating at least one attribute of feedback based on the level of uncertainty, and (4) presenting the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• Example 2 The computer-implemented method of Example 1 where (1) the information associated with the real-time output of the recognition model includes a probability of the user performing the behavior, (2) the computer-implemented method further includes at least one of (a) performing a user-input operation when the probability of the user performing the behavior is above a predetermined threshold and/or (b) refraining from performing the user-input operation when the probability of the user performing the behavior is below the predetermined threshold, and (3) the level of uncertainty associated with the real-time output may be determined based on a distance between the probability and the predetermined threshold. In some examples, the level of uncertainty may be inversely proportional to the distance.
• Example 3 The computer-implemented method of any of Examples 1-2 where (1) the information associated with the real-time output of the recognition model includes a probability of the user performing the behavior and (2) the attribute of the feedback may be modulated to have a level of perceivability proportional to the probability of the user performing the behavior.
• Example 4 The computer-implemented method of any of Examples 1-3 where the attribute of the feedback may be modulated to have a level of perceivability proportional to the level of uncertainty associated with the real-time output.
• Example 5 The computer-implemented method of any of Examples 1-4 where (1) the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture, (2) the information includes a probability of the user performing the pinch gesture, (3) the computer-implemented method further includes performing a user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold, and (4) the feedback may be presented to the user contemporaneous with performing the user-input operation when the probability of the user performing the pinch gesture is above the predetermined threshold.
• the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture
• the information includes a probability of the user performing the pinch gesture
• the computer-implemented method further includes performing a user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold
• the feedback may be presented to the user contemporaneous with performing the user-input operation when the probability of the user performing the pinch gesture is above the predetermined threshold.
• Example 6 The computer-implemented method of any of Examples 1-5 where (1) the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture, (2) the information includes a probability of the user performing the pinch gesture, (3) the computer-implemented method further includes refraining from performing a user-input operation when the probability of the user performing the pinch gesture is below a predetermined threshold, and (4) the feedback may be presented to the user contemporaneous with determining that the probability of the user performing the pinch gesture is below the predetermined threshold.
• the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture
• the information includes a probability of the user performing the pinch gesture
• the computer-implemented method further includes refraining from performing a user-input operation when the probability of the user performing the pinch gesture is below a predetermined threshold
• the feedback may be presented to the user contemporaneous with determining that the probability of the user performing the pinch gesture is below the predetermined threshold.
• Example 7 The computer-implemented method of any of Examples 1-6 where (1) the recognition model may include a hand-tracking model adapted to output (a) position or orientation information for one or more portions of the user's hand and (b) a confidence level of the position or orientation information and (2) the level of uncertainty associated with the real-time output may be determined based on the confidence level. In some examples, the level of uncertainty may be inversely proportional to the confidence level.
• Example 8 The computer-implemented method of any of Examples 1-7 where (1) the feedback may be haptic feedback and (2) at least one attribute of the haptic feedback may be based on the level of uncertainty associated with the real-time output.
• Example 9 The computer-implemented method of any of Examples 1-8 where (1) the feedback may be a vibration and (2) at least one attribute of the vibration may be based on the level of uncertainty associated with the real-time output.
• Example 10 The computer-implemented method of any of Examples 1-9 where modulating the attribute of the feedback based on the level of uncertainty includes modulating one or more of (1) an amplitude of the feedback to convey the level of uncertainty to the user, (2) a frequency of the feedback to convey the level of uncertainty to the user, (3) a duration of the feedback to convey the level of uncertainty to the user, (4) a pattern of the feedback to convey the level of uncertainty to the user, and/or (5) a spatialization of the feedback to convey the level of uncertainty to the user.
• Example 11 The computer-implemented method of any of Examples 1-10 where the feedback indicates a method for reducing the level of uncertainty of the real-time output.
• Example 12 The computer-implemented method of any of Examples 1-11 further including (1) receiving additional information associated with an additional real-time output of the recognition model, (2) determining, based on the additional information, an additional level of uncertainty associated with the additional real-time output, and (3) modulating the feedback being presented to the user based on the additional level of uncertainty.
• a system may include (1) at least one physical processor and (2) physical memory including computer-executable instructions that, when executed by the physical processor, cause the physical processor to (a) receive information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user, (b) determine, based on the information, a level of uncertainty associated with the real-time output, (c) modulate at least one attribute of feedback based on the level of uncertainty, and (d) present the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• Example 14 The system of Example 13 where (1) the information associated with the real-time output of the recognition model includes a probability of the user performing the behavior, (2) the computer-executable instructions, when executed by the physical processor, further cause the physical processor to (a) perform a user-input operation when the probability of the user performingthe behavior is above a predetermined threshold and/or (b) refrain from performing the user-input operation when the probability of the user performing the behavior is below the predetermined threshold, and (3) the level of uncertainty associated with the real-time output may be determined based on a distance between the probability and the predetermined threshold. In some examples, the level of uncertainty may be inversely proportional to the distance.
• Example 15 The system of any of Examples 13-14 where (1) the information associated with the real-time output of the recognition model includes a probability of the user performing the behavior and (2) the attribute of the feedback may be modulated to have a level of perceivability proportional to the probability of the user performing the behavior.
• Example 16 The system of any of Examples 13-15 where the attribute of the feedback may be modulated to have a level of perceivability proportional to the level of uncertainty associated with the real-time output.
• Example 17 The system of any of Examples 13-16 where (1) the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture, (2) the information includes a probability of the user performing the pinch gesture, (3) the computer-executable instructions, when executed by the physical processor, further cause the physical processor to perform a user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold, and (4) the feedback may be presented to the user contemporaneous with performing the userinput operation when the probability of the user performing the pinch gesture is above the predetermined threshold.
• the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture
• the information includes a probability of the user performing the pinch gesture
• the computer-executable instructions when executed by the physical processor, further cause the physical processor to perform a user-input operation when the probability of the user performing the pinch gesture is above a predetermined threshold
• the feedback may be presented to the user contemporaneous with performing the userinput operation when the probability of the user performing the pinch gesture
• Example 18 The system of any of Examples 13-17 where (1) the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture, (2) the information includes a probability of the user performing the pinch gesture, (3) the computer-executable instructions, when executed by the physical processor, further cause the physical processor to refrain from performing a user-input operation when the probability of the user performing the pinch gesture is below a predetermined threshold, and (4) the feedback may be presented to the user contemporaneous with determining that the probability of the user performing the pinch gesture is below the predetermined threshold.
• the recognition model includes a pinch recognition model adapted to output probabilities of the user performing a pinch gesture
• the information includes a probability of the user performing the pinch gesture
• the computer-executable instructions when executed by the physical processor, further cause the physical processor to refrain from performing a user-input operation when the probability of the user performing the pinch gesture is below a predetermined threshold
• the feedback may be presented to the user contemporaneous with determining that the probability of the user performing the pinch gesture is below the pre
• Example 19 The system of any of Examples 13-18 where (1) the recognition model may include a hand-tracking model adapted to output (a) position or orientation information for one or more portions of the user's hand and (b) a confidence level of the position or orientation information and (2) the level of uncertainty associated with the realtime output may be determined based on the confidence level. In some examples, the level of uncertainty may be inversely proportional to the confidence level.
• a non-transitory computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to (1) receive information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user, (2) determine, based on the information, a level of uncertainty associated with the real-time output, (3) modulate at least one attribute of feedback based on the level of uncertainty, and (4) present the feedback to the user substantially contemporaneous with the real-time output of the recognition model.
• Embodiments of the present disclosure may include or be implemented in conjunction with various types of artificial-reality systems.
• Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, for example, a virtual reality, an augmented reality, a mixed reality, a hybrid reality, or some combination and/or derivative thereof.
• Artificial-reality content may include completely computer-generated content or computer-generated content combined with captured (e.g., real-world) content.
• the artificial-reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional (3D) effect to the viewer).
• artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, for example, create content in an artificial reality and/or are otherwise used in (e.g., to perform activities in) an artificial reality.
• Artificial-reality systems may be implemented in a variety of different form factors and configurations. Some artificial-reality systems may be designed to work without near-eye displays (NEDs). Other artificial-reality systems may include an NED that also provides visibility into the real world (such as, e.g., augmented-reality system 1300 in FIG. 13) or that visually immerses a user in an artificial reality (such as, e.g., virtual-reality system 1400 in FIG. 14). While some artificial-reality devices may be self-contained systems, other artificialreality devices may communicate and/or coordinate with external devices to provide an artificial-reality experience to a user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by a user, devices worn by one or more other users, and/or any other suitable external system.
• augmented-reality system 1300 may include an eyewear device 1302 with a frame 1310 configured to hold a left display device 1315(A) and a right display device 1315(B) in front of a user's eyes.
• Display devices 1315(A) and 1315(B) may act together or independently to present an image or series of images to a user.
• augmented-reality system 1300 includes two displays, embodiments of this disclosure may be implemented in augmented-reality systems with a single NED or more than two NEDs.
• augmented-reality system 1300 may include one or more sensors, such as sensor 1340.
• Sensor 1340 may generate measurement signals in response to motion of augmented-reality system 1300 and may be located on substantially any portion of frame 1310.
• Sensor 1340 may represent one or more of a variety of different sensing mechanisms, such as a position sensor, an inertial measurement unit (IMU), a depth camera assembly, a structured light emitter and/or detector, or any combination thereof.
• augmented-reality system 1300 may or may not include sensor 1340 or may include more than one sensor.
• the IMU may generate calibration data based on measurement signals from sensor 1340.
• Examples of sensor 1340 may include, without limitation, accelerometers, gyroscopes, magnetometers, other suitable types of sensors that detect motion, sensors used for error correction of the IMU, or some combination thereof.
• augmented-reality system 1300 may also include a microphone array with a plurality of acoustic transducers 1320(A)-1320(J), referred to collectively as acoustic transducers 1320.
• Acoustic transducers 1320 may represent transducers that detect air pressure variations induced by sound waves.
• Each acoustic transducer 1320 may be configured to detect sound and convert the detected sound into an electronic format (e.g., an analog or digital format).
• 13 may include, for example, ten acoustic transducers: 1320(A) and 1320(B), which may be designed to be placed inside a corresponding ear of the user, acoustic transducers 1320(C), 1320(D), 1320(E), 1320(F), 1320(G), and 1320(H), which may be positioned at various locations on frame 1310, and/or acoustic transducers 1320(1) and 1320(J), which may be positioned on a corresponding neckband 1305.
• acoustic transducers 1320(A)-(J) may be used as output transducers (e.g., speakers).
• acoustic transducers 1320(A) and/or 1320(B) may be earbuds or any other suitable type of headphone or speaker.
• the configuration of acoustic transducers 1320 of the microphone array may vary. While augmented-reality system 1300 is shown in FIG. 13 as having ten acoustic transducers 1320, the number of acoustic transducers 1320 may be greater or less than ten. In some embodiments, using higher numbers of acoustic transducers 1320 may increase the amount of audio information collected and/or the sensitivity and accuracy of the audio information. In contrast, using a lower number of acoustic transducers 1320 may decrease 1 the computing power required by an associated controller 1350 to process the collected audio information. In addition, the position of each acoustic transducer 1320 of the microphone array may vary. For example, the position of an acoustic transducer 1320 may include a defined position on the user, a defined coordinate on frame 1310, an orientation associated with each acoustic transducer 1320, or some combination thereof.
• Acoustic transducers 1320(A) and 1320(B) may be positioned on different parts of the user's ear, such as behind the pinna, behind the tragus, and/or within the auricle or fossa. Or, there may be additional acoustic transducers 1320 on or surrounding the ear in addition to acoustic transducers 1320 inside the ear canal. Having an acoustic transducer 1320 positioned next to an ear canal of a user may enable the microphone array to collect information on how sounds arrive at the ear canal.
• augmented- reality device 1300 may simulate binaural hearing and capture a 3D stereo sound field around about a user's head.
• acoustic transducers 1320(A) and 1320(B) may be connected to augmented-reality system 1300 via a wired connection 1330, and in other embodiments acoustic transducers 1320(A) and 1320(B) may be connected to augmented- reality system 1300 via a wireless connection (e.g., a BLUETOOTH connection).
• acoustic transducers 1320(A) and 1320(B) may not be used at all in conjunction with augmented-reality system 1300.
• Acoustic transducers 1320 on frame 1310 may be positioned in a variety of different ways, including along the length of the temples, across the bridge, above or below display devices 1315(A) and 1315(B), or some combination thereof. Acoustic transducers 1320 may also be oriented such that the microphone array is able to detect sounds in a wide range of directions surroundingthe user wearingthe augmented-reality system 1300. In some embodiments, an optimization process may be performed during manufacturing of augmented-reality system 1300 to determine relative positioning of each acoustic transducer 1320 in the microphone array.
• augmented-reality system 1300 may include or be connected to an external device (e.g., a paired device), such as neckband 1305.
• an external device e.g., a paired device
• Neckband 1305 generally represents any type or form of paired device.
• the following discussion of neckband 1305 may also apply to various other paired devices, such as charging cases, smart watches, smart phones, wrist bands, other wearable devices, hand-held controllers, tablet computers, laptop computers, other external compute devices, etc.
• neckband 1305 may be coupled to eyewear device 1302 via one or more connectors.
• the connectors may be wired or wireless and may include electrical and/or non-electrical (e.g., structural) components.
• eyewear device 1302 and neckband 1305 may operate independently without any wired or wireless connection between them.
• FIG. 13 illustrates the components of eyewear device 1302 and neckband 1305 in example locations on eyewear device 1302 and neckband 1305, the components may be located elsewhere and/or distributed differently on eyewear device 1302 and/or neckband 1305.
• the components of eyewear device 1302 and neckband 1305 may be located on one or more additional peripheral devices paired with eyewear device 1302, neckband 1305, or some combination thereof.
• Pairing external devices such as neckband 1305, with augmented-reality eyewear devices may enable the eyewear devices to achieve the form factor of a pair of glasses while still providing sufficient battery and computation power for expanded capabilities.
• Some or all of the battery power, computational resources, and/or additional features of augmented-reality system 1300 may be provided by a paired device or shared between a paired device and an eyewear device, thus reducing the weight, heat profile, and form factor of the eyewear device overall while still retaining desired functionality.
• neckband 1305 may allow components that would otherwise be included on an eyewear device to be included in neckband 1305 since users may tolerate a heavier weight load on their shoulders than they would tolerate on their heads.
• Neckband 1305 may also have a larger surface area over which to diffuse and disperse heat to the ambient environment. Thus, neckband 1305 may allow for greater battery and computation capacity than might otherwise have been possible on a stand-alone eyewear device. Since weight carried in neckband 1305 may be less invasive to a user than weight carried in eyewear device 1302, a user may tolerate wearing a lighter eyewear device and carrying or wearing the paired device for greater lengths of time than a user would tolerate wearing a heavy standalone eyewear device, thereby enabling users to more fully incorporate artificial-reality environments into their day-to-day activities.
• Neckband 1305 may be communicatively coupled with eyewear device 1302 and/or to other devices. These other devices may provide certain functions (e.g., tracking, localizing, depth mapping, processing, storage, etc.) to augmented-reality system 1300.
• neckband 1305 may include two acoustic transducers (e.g., 1320(1) and 1320(1)) that are part of the microphone array (or potentially form their own microphone subarray).
• Neckband 1305 may also include a controller 1325 and a power source 1335.
• Acoustic transducers 1320(1) and 1320(1) of neckband 1305 may be configured to detect sound and convert the detected sound into an electronic format (analog or digital).
• acoustic transducers 1320(1) and 1320(1) may be positioned on neckband 1305, thereby increasing the distance between the neckband acoustic transducers 1320(1) and 1320(1) and other acoustic transducers 1320 positioned on eyewear device 1302.
• increasing the distance between acoustic transducers 1320 of the microphone array may improve the accuracy of beamforming performed via the microphone array.
• the determined source location of the detected sound may be more accurate than if the sound had been detected by acoustic transducers 1320(D) and 1320(E).
• Controller 1325 of neckband 1305 may process information generated by the sensors on neckband 1305 and/or augmented-reality system 1300. For example, controller 1325 may process information from the microphone array that describes sounds detected by the microphone array. For each detected sound, controller 1325 may perform a direction-of-arrival (DOA) estimation to estimate a direction from which the detected sound arrived at the microphone array. As the microphone array detects sounds, controller 1325 may populate an audio data set with the information. In embodiments in which augmented- reality system 1300 includes an inertial measurement unit, controller 1325 may compute all inertial and spatial calculations from the IMU located on eyewear device 1302.
• DOA direction-of-arrival
• a connector may convey information between augmented-reality system 1300 and neckband 1305 and between augmented-reality system 1300 and controller 1325.
• the information may be in the form of optical data, electrical data, wireless data, or any other transmittable data form. Moving the processing of information generated by augmented-reality system 1300 to neckband 1305 may reduce weight and heat in eyewear device 1302, making it more comfortable to the user.
• Power source 1335 in neckband 1305 may provide power to eyewear device 1302 and/or to neckband 1305.
• Power source 1335 may include, without limitation, lithium ion batteries, lithium-polymer batteries, primary lithium batteries, alkaline batteries, or any other form of power storage.
• power source 1335 may be a wired power source. Including power source 1335 on neckband 1305 instead of on eyewear device 1302 may help better distribute the weight and heat generated by power source 1335.
• some artificial-reality systems may, instead of blending an artificial reality with actual reality, substantially replace one or more of a user's sensory perceptions of the real world with a virtual experience.
• a head-worn display system such as virtual-reality system 1400 in FIG. 14, that mostly or completely covers a user's field of view.
• Virtual-reality system 1400 may include a front rigid body 1402 and a band 1404 shaped to fit around a user's head.
• Virtual-reality system 1400 may also include output audio transducers 1406(A) and 1406(B).
• front rigid body 1402 may include one or more electronic elements, including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking emitters or detectors, and/or any other suitable device or system for creating an artificial-reality experience.
• IMUs inertial measurement units
• Artificial-reality systems may include a variety of types of visual feedback mechanisms.
• display devices in augmented-reality system 1300 and/or virtual- reality system 1400 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, microLED displays, organic LED (OLED) displays, digital light project (DLP) micro-displays, liquid crystal on silicon (LCoS) micro-displays, and/or any other suitable type of display screen.
• LCDs liquid crystal displays
• LED light emitting diode
• microLED organic LED
• DLP digital light project
• LCD liquid crystal on silicon
• These artificial-reality systems may include a single display screen for both eyes or may provide a display screen for each eye, which may allow for additional flexibility for varifocal adjustments or for correcting a user's refractive error.
• Some of these artificial-reality systems may also include optical subsystems having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, adjustable liquid lenses, etc.) through which a user may view a display screen.
• optical subsystems may serve a variety of purposes, including to collimate (e.g., make an object appear at a greater distance than its physical distance), to magnify (e.g., make an object appear larger than its actual size), and/or to relay (to, e.g., the viewer's eyes) light.
• optical subsystems may be used in a non- pupil-forming architecture (such as a single lens configuration that directly collimates light but results in so-called pincushion distortion) and/or a pupil-forming architecture (such as a multi-lens configuration that produces so-called barrel distortion to nullify pincushion distortion).
• a non- pupil-forming architecture such as a single lens configuration that directly collimates light but results in so-called pincushion distortion
• a pupil-forming architecture such as a multi-lens configuration that produces so-called barrel distortion to nullify pincushion distortion
• some of the artificial-reality systems described herein may include one or more projection systems.
• display devices in augmented-reality system 1300 and/or virtual-reality system 1400 may include micro-LED projectors that project light (using, e.g., a waveguide) into display devices, such as clear combiner lenses that allow ambient light to pass through.
• the display devices may refract the projected light toward a user's pupil and may enable a user to simultaneously view both artificial-reality content and the real world.
• the display devices may accomplish this using any of a variety of different optical components, including waveguide components (e.g., holographic, planar, diffractive, polarized, and/or reflective waveguide elements), light-manipulation surfaces and elements (such as diffractive, reflective, and refractive elements and gratings), coupling elements, etc.
• waveguide components e.g., holographic, planar, diffractive, polarized, and/or reflective waveguide elements
• light-manipulation surfaces and elements such as diffractive, reflective, and refractive elements and gratings
• coupling elements etc.
• Artificial-reality systems may also be configured with any other suitable type or form of image projection system, such as retinal projectors used in virtual retina displays.
• augmented-reality system 1300 and/or virtual-reality system 1400 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, time-of- flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor.
• An artificial-reality system may process data from one or more of these sensors to identify a location of a user, to map the real world, to provide a user with context about real-world surroundings, and/or to perform a variety of other functions.
• the artificial-reality systems described herein may also include one or more input and/or output audio transducers.
• Output audio transducers may include voice coil speakers, ribbon speakers, electrostatic speakers, piezoelectric speakers, bone conduction transducers, cartilage conduction transducers, tragus-vibration transducers, and/or any other suitable type or form of audio transducer.
• input audio transducers may include condenser microphones, dynamic microphones, ribbon microphones, and/or any other type or form of input transducer.
• a single transducer may be used for both audio input and audio output.
• the artificial-reality systems described herein may also include tactile (i.e., haptic) feedback systems, which may be incorporated into headwear, gloves, body suits, handheld controllers, environmental devices (e.g., chairs, floormats, etc.), and/or any other type of device or system.
• Haptic feedback systems may provide various types of cutaneous feedback, including vibration, force, traction, texture, and/or temperature.
• Haptic feedback systems may also provide various types of kinesthetic feedback, such as motion and compliance.
• Haptic feedback may be implemented using motors, piezoelectric actuators, fluidic systems, and/or a variety of other types of feedback mechanisms.
• Haptic feedback systems may be implemented independent of other artificial-reality devices, within other artificial-reality devices, and/or in conjunction with other artificial-reality devices.
• artificial-reality systems may create an entire virtual experience or enhance a user's real- world experience in a variety of contexts and environments. For instance, artificial-reality systems may assist or extend a user's perception, memory, or cognition within a particular environment. Some systems may enhance a user's interactions with other people in the real world or may enable more immersive interactions with other people in a virtual world.
• Artificial-reality systems may also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government organizations, military organizations, business enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.), and/or for accessibility purposes (e.g., as hearing aids, visual aids, etc.).
• the embodiments disclosed herein may enable or enhance a user's artificial-reality experience in one or more of these contexts and environments and/or in other contexts and environments.
• Some augmented-reality systems may map a user's and/or device's environment using techniques referred to as "simultaneous location and mapping" (SLAM).
• SLAM mapping and location identifying techniques may involve a variety of hardware and software tools that can create or update a map of an environment while simultaneously keeping track of a user's location within the mapped environment.
• SLAM may use many different types of sensors to create a map and determine a user's position within the map.
• SLAM techniques may, for example, implement optical sensors to determine a user's location. Radios including WiFi, BLUETOOTH, global positioning system (GPS), cellular or other communication devices may be also used to determine a user's location relative to a radio transceiver or group of transceivers (e.g., a WiFi router or group of GPS satellites). Acoustic sensors such as microphone arrays or 2D or 3D sonar sensors may also be used to determine a user's location within an environment. Augmented-reality and virtual-reality devices (such as systems 1300 and 1400 of FIGS. 13 and 14, respectively) may incorporate any or all of these types of sensors to perform SLAM operations such as creating and continually updating maps of the user's current environment.
• Radios including WiFi, BLUETOOTH, global positioning system (GPS), cellular or other communication devices may be also used to determine a user's location relative to a radio transceiver or group of transceivers (e.g., a WiFi router or group of GPS satellites).
• SLAM data generated by these sensors may be referred to as "environmental data" and may indicate a user's current environment.
• This data may be stored in a local or remote data store (e.g., a cloud data store) and may be provided to a user's AR/VR device on demand.
• artificial-reality systems 1300 and 1400 may be used with a variety of other types of devices to provide a more compelling artificial-reality experience. These devices may be haptic interfaces with transducers that provide haptic feedback and/or that collect haptic information about a user's interaction with an environment.
• the artificialreality systems disclosed herein may include various types of haptic interfaces that detect or convey various types of haptic information, including tactile feedback (e.g., feedback that a user detects via nerves in the skin, which may also be referred to as cutaneous feedback) and/or kinesthetic feedback (e.g., feedback that a user detects via receptors located in muscles, joints, and/or tendons).
• Haptic feedback may be provided by interfaces positioned within a user's environment (e.g., chairs, tables, floors, etc.) and/or interfaces on articles that may be worn or carried by a user (e.g., gloves, wristbands, etc.).
• FIG. 15 illustrates a vibrotactile system 1500 in the form of a wearable glove (haptic device 1510) and wristband (haptic device 1520).
• Haptic device 1510 and haptic device 1520 are shown as examples of wearable devices that include a flexible, wearable textile material 1530 that is shaped and configured for positioning against a user's hand and wrist, respectively.
• vibrotactile systems that may be shaped and configured for positioning against other human body parts, such as a finger, an arm, a head, a torso, a foot, or a leg.
• vibrotactile systems may also be in the form of a glove, a headband, an armband, a sleeve, a head covering, a sock, a shirt, or pants, among other possibilities.
• the term "textile” may include any flexible, wearable material, including woven fabric, non-woven fabric, leather, cloth, a flexible polymer material, composite materials, etc.
• One or more vibrotactile devices 1540 may be positioned at least partially within one or more corresponding pockets formed in textile material 1530 of vibrotactile system 1500. Vibrotactile devices 1540 may be positioned in locations to provide a vibrating sensation (e.g., haptic feedback) to a user of vibrotactile system 1500. For example, vibrotactile devices 1540 may be positioned against the user's finger(s), thumb, or wrist, as shown in FIG. 15. Vibrotactile devices 1540 may, in some examples, be sufficiently flexible to conform to or bend with the user's corresponding body part(s).
• a power source 1550 (e.g., a battery) for applying a voltage to the vibrotactile devices 1540 for activation thereof may be electrically coupled to vibrotactile devices 1540, such as via conductive wiring 1552.
• each of vibrotactile devices 1540 may be independently electrically coupled to power source 1550 for individual activation.
• a processor 1560 may be operatively coupled to power source 1550 and configured (e.g., programmed) to control activation of vibrotactile devices 1540.
• Vibrotactile system 1500 may be implemented in a variety of ways.
• vibrotactile system 1500 may be a standalone system with integral subsystems and components for operation independent of other devices and systems.
• vibrotactile system 1500 may be configured for interaction with another device or system 1570.
• vibrotactile system 1500 may, in some examples, include a communications interface 1580 for receiving and/or sending signals to the other device or system 1570.
• the other device or system 1570 may be a mobile device, a gaming console, an artificial-reality (e.g., virtual-reality, augmented-reality, mixed-reality) device, a personal computer, a tablet computer, a network device (e.g., a modem, a router, etc.), a handheld controller, etc.
• Communications interface 1580 may enable communications between vibrotactile system 1500 and the other device or system 1570 via a wireless (e.g., Wi-Fi, BLUETOOTH, cellular, radio, etc.) link or a wired link. If present, communications interface 1580 may be in communication with processor 1560, such as to provide a signal to processor 1560 to activate or deactivate one or more of the vibrotactile devices 1540.
• Vibrotactile system 1500 may optionally include other subsystems and components, such as touch-sensitive pads 1590, pressure sensors, motion sensors, position sensors, lighting elements, and/or user interface elements (e.g., an on/off button, a vibration control element, etc.).
• vibrotactile devices 1540 may be configured to be activated for a variety of different reasons, such as in response to the user's interaction with user interface elements, a signal from the motion or position sensors, a signal from the touch- sensitive pads 1590, a signal from the pressure sensors, a signal from the other device or system 1570, etc.
• power source 1550, processor 1560, and communications interface 1580 are illustrated in FIG. 15 as being positioned in haptic device 1520, the present disclosure is not so limited.
• one or more of power source 1550, processor 1560, or communications interface 1580 may be positioned within haptic device 1510 or within another wearable textile.
• FIG. 16 shows an example artificial-reality environment 1600 including one head-mounted virtual-reality display and two haptic devices (i.e., gloves), and in other embodiments any number and/or combination of these components and other components may be included in an artificial-reality system.
• haptic devices i.e., gloves
• Head-mounted display 1602 generally represents any type or form of virtual-reality system, such as virtual-reality system 1400 in FIG. 14.
• Haptic device 1604 generally represents any type or form of wearable device, worn by a user of an artificial-reality system, that provides haptic feedback to the user to give the user the perception that he or she is physically engaging with a virtual object.
• haptic device 1604 may provide haptic feedback by applying vibration, motion, and/or force to the user. For example, haptic device 1604 may limit or augment a user's movement.
• haptic device 1604 may limit a user's hand from moving forward so that the user has the perception that his or her hand has come in physical contact with a virtual wall.
• one or more actuators within the haptic device may achieve the physical-movement restriction by pumping fluid into an inflatable bladder of the haptic device.
• a user may also use haptic device 1604 to send action requests to a console. Examples of action requests include, without limitation, requests to start an application and/or end the application and/or requests to perform a particular action within the application.
• FIG. 17 is a perspective view of a user 1710 interacting with an augmented-reality system 1700.
• user 1710 may wear a pair of augmented-reality glasses 1720 that may have one or more displays 1722 and that are paired with a haptic device 1730.
• haptic device 1730 may be a wristband that includes a plurality of band elements 1732 and a tensioning mechanism 1734 that connects band elements 1732 to one another.
• band elements 1732 may include any type or form of actuator suitable for providing haptic feedback.
• one or more of band elements 1732 may be configured to provide one or more of various types of cutaneous feedback, including vibration, force, traction, texture, and/or temperature.
• band elements 1732 may include one or more of various types of actuators.
• each of band elements 1732 may include a vibrotactor (e.g., a vibrotactile actuator) configured to vibrate in unison or independently to provide one or more of various types of haptic sensations to a user.
• a vibrotactor e.g., a vibrotactile actuator
• only a single band element or a subset of band elements may include vibrotactors.
• Haptic devices 1510, 1520, 1604, and 1730 may include any suitable number and/or type of haptic transducer, sensor, and/or feedback mechanism.
• haptic devices 1510, 1520, 1604, and 1730 may include one or more mechanical transducers, piezoelectric transducers, and/or fluidic transducers.
• Haptic devices 1510, 1520, 1604, and 1730 may also include various combinations of different types and forms of transducers that work together or independently to enhance a user's artificial-reality experience.
• each of band elements 1732 of haptic device 1730 may include a vibrotactor (e.g., a vibrotactile actuator) configured to vibrate in unison or independently to provide one or more of various types of haptic sensations to a user.
• a vibrotactor e.g., a vibrotactile actuator
• the systems described herein may also include an eye-tracking subsystem designed to identify and track various characteristics of a user's eye(s), such as the user's gaze direction.
• eye tracking may, in some examples, refer to a process by which the position, orientation, and/or motion of an eye is measured, detected, sensed, determined, and/or monitored.
• the disclosed systems may measure the position, orientation, and/or motion of an eye in a variety of different ways, including through the use of various optical-based eye-tracking techniques, ultrasound-based eye-tracking techniques, etc.
• An eye-tracking subsystem may be configured in a number of different ways and may include a variety of different eye-tracking hardware components or other computervision components.
• an eye-tracking subsystem may include a variety of different optical sensors, such as two-dimensional (2D) or 3D cameras, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor.
• a processing subsystem may process data from one or more of these sensors to measure, detect, determine, and/or otherwise monitor the position, orientation, and/or motion of the user's eye(s).
• FIG. 18 is an illustration of an exemplary system 1800 that incorporates an eye-tracking subsystem capable of tracking a user's eye(s).
• system 1800 may include a light source 1802, an optical subsystem 1804, an eye-tracking subsystem 1806, and/or a control subsystem 1808.
• light source 1802 may generate light for an image (e.g., to be presented to an eye 1801 of the viewer).
• Light source 1802 may represent any of a variety of suitable devices.
• light source 1802 can include a two-dimensional projector (e.g., a LCoS display), a scanning source (e.g., a scanning laser), or other device (e.g., an LCD, an LED display, an OLED display, an active-matrix OLED display (AMOLED), a transparent OLED display (TOLED), a waveguide, or some other display capable of generating light for presenting an image to the viewer).
• the image may represent a virtual image, which may refer to an optical image formed from the apparent divergence of light rays from a point in space, as opposed to an image formed from the light ray's actual divergence.
• optical subsystem 1804 may receive the light generated by light source 1802 and generate, based on the received light, converging light 1820 that includes the image.
• optical subsystem 1804 may include any number of lenses (e.g., Fresnel lenses, convex lenses, concave lenses), apertures, filters, mirrors, prisms, and/or other optical components, possibly in combination with actuators and/or other devices.
• the actuators and/or other devices may translate and/or rotate one or more of the optical components to alter one or more aspects of converging light 1820.
• various mechanical couplings may serve to maintain the relative spacing and/or the orientation of the optical components in any suitable combination.
• eye-tracking subsystem 1806 may generate tracking information indicating a gaze angle of an eye 1801 of the viewer.
• control subsystem 1808 may control aspects of optical subsystem 1804 (e.g., the angle of incidence of converging light 1820) based at least in part on this tracking information. Additionally, in some examples, control subsystem 1808 may store and utilize historical tracking information (e.g., a history of the tracking information over a given duration, such as the previous second or fraction thereof) to anticipate the gaze angle of eye 1801 (e.g., an angle between the visual axis and the anatomical axis of eye 1801).
• historical tracking information e.g., a history of the tracking information over a given duration, such as the previous second or fraction thereof
• eye-tracking subsystem 1806 may detect radiation emanating from some portion of eye 1801 (e.g., the cornea, the iris, the pupil, or the like) to determine the current gaze angle of eye 1801. In other examples, eye-tracking subsystem 1806 may employ a wavefront sensor to track the current location of the pupil.
• some portion of eye 1801 e.g., the cornea, the iris, the pupil, or the like
• eye-tracking subsystem 1806 may employ a wavefront sensor to track the current location of the pupil.
• Any number of techniques can be used to track eye 1801. Some techniques may involve illuminating eye 1801 with infrared light and measuring reflections with at least one optical sensor that is tuned to be sensitive to the infrared light. Information about how the infrared light is reflected from eye 1801 may be analyzed to determine the position(s), orientation(s), and/or motion(s) of one or more eye feature(s), such as the cornea, pupil, iris, and/or retinal blood vessels.
• the radiation captured by a sensor of eye-tracking subsystem 1806 may be digitized (i.e., converted to an electronic signal). Further, the sensor may transmit a digital representation of this electronic signal to one or more processors (for example, processors associated with a device including eye-tracking subsystem 1806).
• Eyetracking subsystem 1806 may include any of a variety of sensors in a variety of different configurations.
• eye-tracking subsystem 1806 may include an infrared detector that reacts to infrared radiation.
• the infrared detector may be a thermal detector, a photonic detector, and/or any other suitable type of detector.
• Thermal detectors may include detectors that react to thermal effects of the incident infrared radiation.
• one or more processors may process the digital representation generated by the sensor(s) of eye-tracking subsystem 1806 to track the movement of eye 1801.
• these processors may track the movements of eye 1801 by executing algorithms represented by computer-executable instructions stored on non-transitory memory.
• on-chip logic e.g., an application-specific integrated circuit or ASIC
• eye-tracking subsystem 1806 may be programmed to use an output of the sensor(s) to track movement of eye 1801.
• eye-tracking subsystem 1806 may analyze the digital representation generated by the sensors to extract eye rotation information from changes in reflections.
• eye-tracking subsystem 1806 may use corneal reflections or glints (also known as Purkinje images) and/or the center of the eye's pupil 1822 as features to track over time.
• eye-tracking subsystem 1806 may use the center of the eye's pupil 1822 and infrared or near-infrared, non-collimated light to create corneal reflections. In these embodiments, eye-tracking subsystem 1806 may use the vector between the center of the eye's pupil 1822 and the corneal reflections to compute the gaze direction of eye 1801.
• the disclosed systems may perform a calibration procedure for an individual (using, e.g., supervised or unsupervised techniques) before tracking the user's eyes.
• the calibration procedure may include directing users to look at one or more points displayed on a display while the eye-tracking system records the values that correspond to each gaze position associated with each point.
• eye-tracking subsystem 1806 may use two types of infrared and/or near-infrared (also known as active light) eye-tracking techniques: bright- pupil and dark-pupil eye tracking, which may be differentiated based on the location of an illumination source with respect to the optical elements used. If the illumination is coaxial with the optical path, then eye 1801 may act as a retroreflector as the light reflects off the retina, thereby creating a bright pupil effect similar to a red-eye effect in photography. If the illumination source is offset from the optical path, then the eye's pupil 1822 may appear dark because the retroreflection from the retina is directed away from the sensor.
• infrared and/or near-infrared also known as active light
• bright-pupil tracking may create greater iris/pupil contrast, allowing more robust eye tracking with iris pigmentation, and may feature reduced interference (e.g., interference caused by eyelashes and other obscuring features).
• Bright-pupil tracking may also allow tracking in lighting conditions ranging from total darkness to a very bright environment.
• control subsystem 1808 may control light source 1802 and/or optical subsystem 1804 to reduce optical aberrations (e.g., chromatic aberrations and/or monochromatic aberrations) of the image that may be caused by or influenced by eye 1801.
• control subsystem 1808 may use the tracking information from eye-tracking subsystem 1806 to perform such control.
• control subsystem 1808 may alter the light generated by light source 1802 (e.g., by way of image rendering) to modify (e.g., pre-distort) the image so that the aberration of the image caused by eye 1801 is reduced.
• the disclosed systems may track both the position and relative size of the pupil (since, e.g., the pupil dilates and/or contracts).
• the eye-tracking devices and components e.g., sensors and/or sources
• the frequency range of the sensors may be different (or separately calibrated) for eyes of different colors and/or different pupil types, sizes, and/or the like.
• the various eye-tracking components e.g., infrared sources and/or sensors
• described herein may need to be calibrated for each individual user and/or eye.
• the disclosed systems may track both eyes with and without ophthalmic correction, such as that provided by contact lenses worn by the user.
• ophthalmic correction elements e.g., adjustable lenses
• the color of the user's eye may necessitate modification of a corresponding eye-tracking algorithm.
• eyetracking algorithms may need to be modified based at least in part on the differing color contrast between a brown eye and, for example, a blue eye.
• FIG. 19 is a more detailed illustration of various aspects of the eye-tracking subsystem illustrated in FIG. 18.
• an eye-tracking subsystem 1900 may include at least one source 1904 and at least one sensor 1906.
• Source 1904 generally represents any type or form of element capable of emitting radiation.
• source 1904 may generate visible, infrared, and/or near-infrared radiation.
• source 1904 may radiate non-collimated infrared and/or near-infrared portions of the electromagnetic spectrum towards an eye 1902 of a user.
• Source 1904 may utilize a variety of sampling rates and speeds.
• the disclosed systems may use sources with higher sampling rates in order to capture fixational eye movements of a user's eye 1902 and/or to correctly measure saccade dynamics of the user's eye 1902.
• any type orform of eye-tracking technique may be used to track the user's eye 1902, including optical-based eye-tracking techniques, ultrasound-based eye-tracking techniques, etc.
• Sensor 1906 generally represents any type or form of element capable of detecting radiation, such as radiation reflected off the user's eye 1902.
• sensor 1906 include, without limitation, a charge coupled device (CCD), a photodiode array, a complementary metal-oxide-semiconductor (CMOS) based sensor device, and/or the like.
• CCD charge coupled device
• CMOS complementary metal-oxide-semiconductor
• sensor 1906 may represent a sensor having predetermined parameters, including, but not limited to, a dynamic resolution range, linearity, and/orother characteristic selected and/or designed specifically for eye tracking.
• eye-tracking subsystem 1900 may generate one or more glints.
• a glint 1903 may represent reflections of radiation (e.g., infrared radiation from an infrared source, such as source 1904) from the structure of the user's eye.
• glint 1903 and/or the user's pupil may be tracked using an eyetracking algorithm executed by a processor (either within or external to an artificial reality device).
• an artificial reality device may include a processor and/or a memory device in order to perform eye tracking locally and/or a transceiver to send and receive the data necessary to perform eye tracking on an external device (e.g., a mobile phone, cloud server, or other computing device).
• an external device e.g., a mobile phone, cloud server, or other computing device.
• FIG. 19 shows an example image 1905 captured by an eye-tracking subsystem, such as eye-tracking subsystem 1900.
• image 1905 may include both the user's pupil 1908 and a glint 1910 near the same.
• pupil 1908 and/or glint 1910 may be identified using an artificial-intelligence-based algorithm, such as a computer-vision-based algorithm.
• image 1905 may represent a single frame in a series of frames that may be analyzed continuously in order to track the eye 1902 of the user. Further, pupil 1908 and/or glint 1910 may be tracked over a period of time to determine a user's gaze.
• eye-tracking subsystem 1900 may be configured to identify and measure the inter-pupillary distance (IPD) of a user.
• IPD inter-pupillary distance
• eye-tracking subsystem 1900 may measure and/or calculate the IPD of the user while the user is wearing the artificial reality system.
• eye-tracking subsystem 1900 may detect the positions of a user's eyes and may use this information to calculate the user's IPD.
• the eye-tracking systems or subsystems disclosed herein may track a user's eye position and/or eye movement in a variety of ways.
• one or more light sources and/or optical sensors may capture an image of the user's eyes.
• the eyetracking subsystem may then use the captured information to determine the user's inter- pupillary distance, interocular distance, and/or a 3D position of each eye (e.g., for distortion adjustment purposes), including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and/or gaze directions for each eye.
• infrared light may be emitted by the eyetracking subsystem and reflected from each eye. The reflected light may be received or detected by an optical sensor and analyzed to extract eye rotation data from changes in the infrared light reflected by each eye.
• the eye-tracking subsystem may use any of a variety of different methods to track the eyes of a user.
• a light source e.g., infrared light-emitting diodes
• the eye-tracking subsystem may then detect (e.g., via an optical sensor coupled to the artificial reality system) and analyze a reflection of the dot pattern from each eye of the user to identify a location of each pupil of the user.
• the eye-tracking subsystem may track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of a user to estimate a gaze point (i.e., a 3D location or position in a virtual scene where the user is looking) and/or an IPD.
• a gaze point i.e., a 3D location or position in a virtual scene where the user is looking
• IPD IPD
• the distance between a user's pupil and a display may change as the user's eye moves to look in different directions.
• the varying distance between a pupil and a display as viewing direction changes may be referred to as "pupil swim" and may contribute to distortion perceived by the user as a result of light focusing in different locations as the distance between the pupil and the display changes.
• measuring distortion at different eye positions and pupil distances relative to displays and generating distortion corrections for different positions and distances may allow mitigation of distortion caused by pupil swim by tracking the 3D position of a user's eyes and applying a distortion correction corresponding to the 3D position of each of the user's eyes at a given point in time.
• knowing the 3D position of each of a user's eyes may allow for the mitigation of distortion caused by changes in the distance between the pupil of the eye and the display by applying a distortion correction for each 3D eye position. Furthermore, as noted above, knowing the position of each of the user's eyes may also enable the eye-tracking subsystem to make automated adjustments for a user's IPD.
• a display subsystem may include a variety of additional subsystems that may work in conjunction with the eye-tracking subsystems described herein.
• a display subsystem may include a varifocal subsystem, a scene-rendering module, and/or a vergence-processing module.
• the varifocal subsystem may cause left and right display elements to vary the focal distance of the display device.
• the varifocal subsystem may physically change the distance between a display and the optics through which it is viewed by moving the display, the optics, or both. Additionally, moving or translating two lenses relative to each other may also be used to change the focal distance of the display.
• the varifocal subsystem may include actuators or motors that move displays and/or optics to change the distance between them.
• This varifocal subsystem may be separate from or integrated into the display subsystem.
• the varifocal subsystem may also be integrated into or separate from its actuation subsystem and/or the eye-tracking subsystems described herein.
• the display subsystem may include a vergence-processing module configured to determine a vergence depth of a user's gaze based on a gaze point and/or an estimated intersection of the gaze lines determined by the eye-tracking subsystem.
• Vergence may refer to the simultaneous movement or rotation of both eyes in opposite directions to maintain single binocular vision, which may be naturally and automatically performed by the human eye.
• a location where a user's eyes are verged is where the user is looking and is also typically the location where the user's eyes are focused.
• the vergence-processing module may triangulate gaze lines to estimate a distance or depth from the user associated with intersection of the gaze lines.
• the depth associated with intersection of the gaze lines may then be used as an approximation for the accommodation distance, which may identify a distance from the user where the user's eyes are directed.
• the vergence distance may allow for the determination of a location where the user's eyes should be focused and a depth from the user's eyes at which the eyes are focused, thereby providing information (such as an object or plane of focus) for rendering adjustments to the virtual scene.
• the vergence-processing module may coordinate with the eye-tracking subsystems described herein to make adjustments to the display subsystem to account for a user's vergence depth.
• the user's pupils may be slightly farther apart than when the user is focused on something close.
• the eyetracking subsystem may obtain information about the user's vergence or focus depth and may adjust the display subsystem to be closer together when the user's eyes focus or verge on something close and to be farther apart when the user's eyes focus or verge on something at a distance.
• the eye-tracking information generated by the above-described eyetracking subsystems may also be used, for example, to modify various aspect of how different computer-generated images are presented.
• a display subsystem may be configured to modify, based on information generated by an eye-tracking subsystem, at least one aspect of how the computer-generated images are presented.
• the computer-generated images may be modified based on the user's eye movement, such that if a user is looking up, the computer-generated images may be moved upward on the screen. Similarly, if the user is looking to the side or down, the computer-generated images may be moved to the side or downward on the screen. If the user's eyes are closed, the computergenerated images may be paused or removed from the display and resumed once the user's eyes are back open.
• eye-tracking subsystems can be incorporated into one or more of the various artificial reality systems described herein in a variety of ways.
• one or more of the various components of system 1800 and/or eye-tracking subsystem 1900 may be incorporated into augmented-reality system 1300 in FIG. 13 and/or virtual-reality system 1400 in FIG. 14 to enable these systems to perform various eye-tracking tasks (including one or more of the eye-tracking operations described herein).
• FIG. 20A illustrates an exemplary human-machine interface (also referred to herein as an EMG control interface) configured to be worn around a user's lower arm or wrist as a wearable system 2000.
• wearable system 2000 may include sixteen neuromuscular sensors 2010 (e.g., EMG sensors) arranged circumferentially around an elastic band 2020 with an interior surface 2030 configured to contact a user's skin.
• EMG sensors 2010 e.g., EMG sensors
• any suitable number of neuromuscular sensors may be used. The number and arrangement of neuromuscular sensors may depend on the particular application for which the wearable device is used.
• a wearable armband or wristband can be used to generate control information for controlling an augmented reality system, a robot, controlling a vehicle, scrolling through text, controlling a virtual avatar, or any other suitable control task.
• the sensors may be coupled together using flexible electronics incorporated into the wireless device.
• FIG. 20B illustrates a cross-sectional view through one of the sensors of the wearable device shown in FIG. 20A.
• the output of one or more of the sensing components can be optionally processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the output of the sensing components can be performed in software.
• signal processing of signals sampled by the sensors can be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect.
• a non-limiting example of a signal processing chain used to process recorded data from sensors 2010 is discussed in more detail below with reference to FIGS. 21A and 21B.
• FIGS. 21A and 21B illustrate an exemplary schematic diagram with internal components of a wearable system with EMG sensors.
• the wearable system may include a wearable portion 2110 (FIG. 21A) and a dongle portion 2120 (FIG. 21B) in communication with the wearable portion 2110 (e.g., via BLUETOOTH or another suitable wireless communication technology).
• the wearable portion 2110 may include skin contact electrodes 2111, examples of which are described in connection with FIGS. 20A and 20B.
• the output of the skin contact electrodes 2111 may be provided to analog front end 2130, which may be configured to perform analog processing (e.g., amplification, noise reduction, filtering, etc.) on the recorded signals.
• analog processing e.g., amplification, noise reduction, filtering, etc.
• the processed analog signals may then be provided to analog-to-digital converter 2132, which may convert the analog signals to digital signals that can be processed by one or more computer processors.
• An example of a computer processor that may be used in accordance with some embodiments is microcontroller (MCU) 2134, illustrated in FIG. 21A. As shown, MCU 2134 may also include inputs from other sensors (e.g., IMU sensor 2140), and power and battery module 2142. The output of the processing performed by MCU 2134 may be provided to antenna 2150 for transmission to dongle portion 2120 shown in FIG. 21B.
• MCU microcontroller
• Dongle portion 2120 may include antenna 2152, which may be configured to communicate with antenna 2150 included as part of wearable portion 2110. Communication between antennas 2150 and 2152 may occur using any suitable wireless technology and protocol, non-limiting examples of which include radiofrequency signaling and BLUETOOTH. As shown, the signals received by antenna 2152 of dongle portion 2120 may be provided to a host computer for further processing, display, and/or for effecting control of a particular physical or virtual object or objects. [0177] Although the examples provided with reference to FIGS. 20A-20B and FIGS.
• the techniques described herein for reducing electromagnetic interference can also be implemented in wearable interfaces with other types of sensors including, but not limited to, mechanomyography (MMG) sensors, sonomyography (SMG) sensors, and electrical impedance tomography (EIT) sensors.
• MMG mechanomyography
• SMG sonomyography
• EIT electrical impedance tomography
• the techniques described herein for reducing electromagnetic interference can also be implemented in wearable interfaces that communicate with computer hosts through wires and cables (e.g., USB cables, optical fiber cables, etc.).
• computing devices and systems described and/or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein.
• these computing device(s) may each include at least one memory device and at least one physical processor.
• the term "memory device” generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and/or computer-readable instructions.
• a memory device may store, load, and/or maintain one or more of the modules described herein.
• Examples of memory devices include, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory.
• the term "physical processor” generally refers to any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions.
• a physical processor may access and/or modify one or more modules stored in the above-described memory device.
• Examples of physical processors include, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.
• modules described and/or illustrated herein may represent portions of a single module or application.
• one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks.
• one or more of the modules described and/or illustrated herein may represent modules stored and configured to run on one or more of the computing devices or systems described and/or illustrated herein.
• One or more of these modules may also represent all or portions of one or more special-purpose computers configured to perform one or more tasks.
• one or more of the modules described herein may transform data, physical devices, and/or representations of physical devices from one form to another.
• one or more of the modules recited herein may receive a real-time output of a recognition model to be transformed, transform the real-time output of the recognition model into a level of uncertainty, output a result of the transformation to a feedback system for subsequent communication to a user, and/or use the result of the transformation to modulate at least one attribute of an appropriate and/or suitable feedback for presentation to the user.
• one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and/or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and/or otherwise interacting with the computing device.
• the term "computer-readable medium” generally refers to any form of device, carrier, or medium capable of storing or carrying computer- readable instructions.
• Examples of computer-readable media include, without limitation, transmission-type media, such as carrier waves, and non-transitory-type media, such as magnetic-storage media (e.g., hard disk drives, tape drives, and floppy disks), optical-storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic- storage media (e.g., solid-state drives and flash media), and other distribution systems.
• transmission-type media such as carrier waves
• non-transitory-type media such as magnetic-storage media (e.g., hard disk drives, tape drives, and floppy disks), optical-storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic- storage media (e.g., solid-state drives and

Landscapes

• Engineering & Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Dermatology (AREA)
• General Health & Medical Sciences (AREA)
• Neurology (AREA)
• Neurosurgery (AREA)
• User Interface Of Digital Computer (AREA)
• Data Exchanges In Wide-Area Networks (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=83355015

Family Applications (1)

Country Status (3)

Family Cites Families (1)

• 2022
  • 2022-08-02 TW TW111129002A patent/TW202318180A/zh unknown
  • 2022-08-19 WO PCT/US2022/040840 patent/WO2023023299A1/en active Application Filing
  • 2022-08-19 EP EP22772649.4A patent/EP4388395A1/de active Pending

Also Published As

Similar Documents

Legal Events