WO2025212940A1

WO2025212940A1 - Management of application actions from a wearable device

Info

Publication number: WO2025212940A1
Application number: PCT/US2025/023027
Authority: WO
Inventors: Dongeek Shin; Aveek PUROHIT
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2024-04-04
Filing date: 2025-04-03
Publication date: 2025-10-09
Anticipated expiration: 2026-10-04

Abstract

According to at least one implementation, a method includes receiving a voice request and obtaining sensor data from a sensor in response to the voice request. The method further includes applying a model to the voice request and the sensor data to determine an application associated with the voice request and an action associated with the application. The method also provides for causing the application to initiate the action.

Description

MANAGEMENT OF APPLICATION ACTIONS FROM

A WEARABLE DEVICE

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63/574,718, filed on April 4, 2024, entitled “MANAGEMENT OF APPLICATION ACTIONS FROM A WEARABLE DEVICE USING AN ACTIONABLE LANGUAGE MODEL,” the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND

[0002] An extended reality (XR) device incorporates a spectrum of technologies that blend physical and virtual worlds, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). These devices immerse users in digital environments, either by blocking out the real world (VR), overlaying digital content onto the real world (AR), or blending digital and physical elements seamlessly (MR). XR devices include headsets, glasses, or screens equipped with sensors, cameras, and displays that track the movement of users and their surroundings to deliver immersive experiences across various applications such as gaming, education, healthcare, mobile computing, and industrial training.

SUMMARY

[0003] This disclosure relates to systems and methods for managing application actions on a wearable device. In some implementations, a device can be configured with a model that identifies voice input and determines sensor data to provide context associated with the voice input. In some examples, the device can be configured to process the voice request using a model (i.e., language model) to identify the sensor (and corresponding sensor data) related to the voice request. In some implementations, the sensor data comprises a text descriptor for an object identified in an image captured by the device. In some implementations, the sensor data comprises text captured in an image captured by the device. In at least one example, the device is further configured to apply the model to the voice request and the sensor data to determine an application associated with the voice request and an action associated with the application. Once the model is applied, the device can further be configured to cause the application to initiate the action. [0004] In some aspects, the techniques described herein relate to a (computer- implemented) method including: receiving (e.g., by a computing system such as a wearable computing device) a voice request; in response to the voice request, accessing (e.g., by the computing system) sensor data from a sensor; applying (e.g., by the computing system) a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing (e.g., by the computing system) the application to initiate the action.

[0005] In some aspects, the techniques described herein relate to a computing system including: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer- readable storage medium that, when executed by the at least one processor, direct the computing system to perform a method, the method including: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

[0006] In some aspects, the techniques described herein relate to a computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method including: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action. In some aspects, the techniques described herein relate to a computer program having said program instructions.

[0007] The accompanying drawings and the description below outline the details of one or more implementations. Other features will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 illustrates a computing environment to supplement user prompts with sensor data according to an implementation.

[0009] FIG. 2 illustrates a method of implementing actions based on user prompts and sensor data according to an implementation. [0010] FIG. 3 illustrates an operational scenario of implementing actions based on user prompts and sensor data according to an implementation.

[0011] FIG. 4 illustrates an operational scenario of implementing actions based on user prompts and sensor data according to an implementation.

[0012] FIG. 5 illustrates a computing system for implementing actions based on user prompts and sensor data according to an implementation.

DETAILED DESCRIPTION

[0013] Computing devices, such as wearable devices and extended reality (XR) devices, provide users an effective tool for gaming, training, education, healthcare, mobile computing, and more. An XR device merges the physical and virtual worlds, encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR) experiences. These devices can include headsets or glasses equipped with sensors, cameras, and displays that track users’ movements and surroundings, allowing them to interact with digital content in real-time. XR devices offer immersive expenences by either completely replacing the real world with a virtual one (VR), overlaying digital information onto the real world (AR), or seamlessly integrating digital and physical elements (MR). Input to XR devices may be provided through gestures, voice commands, controllers, and eye movements. Users interact with the virtual environment by manipulating objects, navigating menus, and triggering actions using these input methods. The device’s sensors and algorithms translate these input methods into various actions on the device. However, at least one technical problem exists in processing voice requests from a user that includes ambiguous or vague terms.

[0014] In at least one technical solution, a language model is introduced that processes a user’s voice input and initiates actions in applications, e.g., the applications installed on a computing device, based on the voice input. The language model is an example of a machine learning model designed to understand and work with the human language. The model can use automatic speech recognition (ASR) to transcribe spoken input, a natural language understanding (NLU) component to interpret intent, and then execute tasks (like setting alarms, sending texts, or controlling smart devices) through predefined action modules or APIs. For example, based on the word choice in a first voice request, the language model may determine that the request is associated with a calendar application and will identify an action associated with the calendar application (e.g., generate an event for the calendar). Some applications for the device may include messaging applications, calendar applications, voice or video applications, web browsing applications, or some other type of application. [0015] In some implementations, the device can be configured to supplement voice input or voice prompts with sensor data from one or more other sensors co-located on the wearable device. The additional sensor information may include images, video, additional sound information, or other information to support the request. In some examples, the sensor data is obtained or accessed from the sensors during the user’s voice input. In some implementations, the sensor data is obtained from the sensors after the intent of the voice input is mapped to one or more corresponding sensors. In some examples, the language model uses text or object recognition software to identify the objects captured from the sensors. In some implementations, text recognition is used to supplement the information provided verbally from the user. The text recognition may provide date information, location information, name information (e.g., business name, event name, and the like), or some other information to support the request. Further, image recognition may provide context associated with a voice request, such as object recognition (i.e., text description of the object, identification of an entity in an entity repository, and the like), that is not identifiable in the initial voice request. The technical effects of which are providing context associated with the user input.

[0016] As an example of using the language model, a wearable device user generates a voice request to find the nearest retail location of an object captured in a camera field of view for the device. The wearable device identifies the request and determines that additional sensor data is required to support the desired action based on the language processing of the voice request. The device obtains or accesses image data from one or more cameras and performs object recognition on the image data to identify a text descriptor of the object. Once the object is identified, the model identifies at least one action for an application based on the identified object and the voice request from the user. In at least one example, the device initiates a map application that directs the user to the retail location associated with the object. In some examples, the model identifies and initiates actions in association with applications on the wearable device, initiates actions in association with a user device (e.g., smartphone, tablet, computer, and the like) communicatively coupled to the wearable device, or initiates actions in association with services or applications provided by servers and other network devices.

[0017] In some implementations, the device can incorporate other sensors in addition to or in place of the cameras (e.g., an outward-facing camera identifying objects in the user’s field of view). The sensors can provide context and enrich voice requests. Microphones capture speech, while accelerometers and gyroscopes detect motion and gestures, allowing the system to understand if the user is moving, exercising, or gesturing. Heart rate monitors and electrodermal activity sensors can provide physiological data indicating stress or excitement. GPS and compass sensors offer location and orientation data useful for navigation or situational awareness. Proximity and ambient light sensors help determine the device’s environment, and temperature or skin contact sensors can assess whether the device is worn correctly. The device may comprise any (or all) of these sensors and use this/these sensor(s) to obtain the sensor data.

[0018] In some implementations, the device can be configured to capture a voice request using a microphone. In some examples, the device can identify a keyword or phrase indicating the user is initiating a request. In some examples, the device can passively identify a request from the word choice or language from the user. In some examples, the device can recognize the voice prompt following the user selecting a physical or virtual button. For example, the device can listen for a wake word to activate voice recognition and receives a voice prompt through a built-in microphone, which captures the user’s speech. In some implementations, when a user provides the prompt, such as “How fast was I running?” or “What’s my heart rate now?” the language model first uses natural language understanding to identify key components like intent (“ask for speed” or “request heart rate”) and relevant entities (“running,” “now”). Based on this analysis, the device can be configured to determine that certain types of data, such as from the accelerometer, GPS, or heart rate monitor, are required to generate an appropriate response.

[0019] In some examples, if a user says, “How far did I walk today?” the model recognizes the need for distance tracking and infers that data from the GPS or accelerometer is required. This mapping can be guided by predefined associations between intents and sensors or learned through training on labeled examples, enabling the model to select appropriate sensor inputs based on natural language commands.

[0020] In some examples, a user can issue a prompt such as “Can you tell me if this fruit looks ripe?” to a wearable device equipped with a camera. A language model processes the speech to extract the intent (“evaluate ripeness”) and the modality implied by the phrase “looks,” thereby determining that visual data is required. The system activates the camera to capture an image of the object in view. The system can further leverage auxiliary sensors, such as ambient light, for exposure adjustment to improve image quality. The captured image is then analyzed by a computer vision model trained to assess ripeness based on visual features like color and texture. In this way, the language model acts as an interface between natural language understanding and sensor activation, enabling context-aware, sensor-driven responses based on user speech. In some examples, the system can perform image-to-text where a system analyzes an image and generates a textual description of its contents. This can involve identifying objects, scenes, text within the image, or interpreting more complex visual contexts using computer vision and language models. The text can be used to provide context associated with vague terminology and provide a more contextually relevant action. In some implementations, the system can use Optical Character Recognition (OCR), which detects, and extracts written or printed text from visual input. The system can use computer vision techniques to locate text regions and then apply pattern recognition or deep learning models to convert the visual characters into machine-readable text. For example, the system can identify information on a poster (e.g., date, name, time, and the like)

[0021] Various embodiments of the present technology provide for a wide range of technical effects, advantages, and/or improvements to computing systems and components. For example, various examples may include one or more of the following technical effects, technical solutions, advantages, and/or improvements: 1) non-routine and unconventional use of additional sensor data to supplement voice input in performing desired application actions; 2) non-routine selection of sensor data to support a voice request; and 3) improved application action implementation using supplemental information from sensors. Thereby, the present technology allows improved results of the processing of voice requests from a user that includes ambiguous or vague terms.

[0022] FIG. 1 illustrates a computing environment 100 to supplement user prompts with sensor data according to an implementation. Computing environment 100 includes wearable device 110, poster 132, voice input 130, user device 160, network 180, and network devices 170. Wearable device 110 further includes microphone(s) 120, camera sensor(s) 121, other sensor(s) 122, display 125, local applications 150, and language model 140 with language processing 142 and sensor processing 144. User device 160 further includes applications 162, and network devices 170 further includes network devices 170 with applications and services 172.

[0023] In some implementations, wearable device 110 is representative of a virtual reality (VR) device, augmented reality (AR), extended reality (XR) device, or some other wearable device capable of providing the operations described herein. XR devices, including goggles or glasses, can create immersive experiences by overlaying digital information with the real world. To do this effectively, the devices rely on various sensors that track the user's movements, the environment, and the interaction between the digital and physical worlds. Some of the sensors employed by wearable devices include cameras, accelerometers, microphones, gyroscopes, proximity sensors, temperature sensors, and ambient light sensors. The sensors allow the wearable device to understand and interact with the user of the device, creating immersive experiences. In some implementations, wearable device 110 includes at least microphone(s) 120, camera sensor(s) 121, and other sensor(s) 122. Language model 140 receives information from microphone(s) 120, camera sensor(s) 121, and other sensor(s) 122 and determines an action based on the received information.

[0024] In some implementations, wearable device 110 can represent a device that can overlay digital information or images onto the real world, enhancing the user’s perception of their surroundings. This technology allows users to interact with virtual content as if it were part of their physical environment, blending the digital and physical worlds. In some implementations, display 125 works by projecting digital content onto the user’s field of view using screens embedded in a headset or smart glasses. In some examples, the information provided can include feedback associated with the action from voice input 130. The feedback can indicate the action taken, the status of the action, or some other status information related to the action.

[0025] Wearable device 110 can include a processing system (i.e., at least one processor) and display 125. In addition, wearable device 110 can include other components that support the operations of the user. Wearable device 110 may communicate with user device 160 via a local connection (e.g., Bluetooth, Wi-Fi, etc.) or with user device 160 and network devices 170 via network 180. Network 180 represents the internet in some examples. Although demonstrated as having a local connection with user device 160, wearable device 110 can be a standalone device in some examples. In some implementations, user device 160 represents a companion device capable of performing at least a portion of the operations associated with language model 140.

[0026] In computing environment 100, microphone(s) 120 receive voice input 130. Voice input 130 may be generated after the user provides a touch, verbal, or other command to indicate imminent voice input. Alternatively, voice input 130 may be received passively by wearable device 110 and processed to determine whether the language corresponds to a request. Although demonstrated as voice input 130, the user of wearable device 110 can use a keyboard or some other touch interface to provide the input. For example, in providing the input “add this event to my calendar,” the user may provide the input using a keyboard communicatively coupled to wearable device 110.

[0027] After voice input 130 is received, wearable device 110 implements language model 140. Language model 140 includes language processing 142 and sensor processing 144. Language processing 142 is an algorithm designed to understand and take actions based on language. Language processing 142 can be used to identify voice input 130 of the user and translate the input into actions across applications locally on wearable device 110 or user device 160. Language processing 142, as part of language model 140, can represent a machine learning model that enables systems to leam from data, identify patterns, and make decisions with minimal human intervention. Specifically, language processing 142 can identify content and patterns within voice input 130 to determine a corresponding application action associated with the voice input. The actions can add or remove events from calendars, generate messages in a messaging application, initiate a voice or video call, or provide some other action in association with an application. In some examples, the language model is trained or configured to identify actions from user speech by learning from labeled examples that link spoken commands to specific actions. It uses techniques like intent classification and slot filling to understand the user’s intent and extract relevant details.

[0028] In some implementations, language model 140 requires additional sensor input or context to resolve ambiguities associated with the language provided by the user. These ambiguities can be related to referenced entities or parameters included in voice input 130 (e.g., terms such as “this” or “that”). Using the example in computing environment 100, voice input 130 indicates a voice request to “add this event to my calendar.” However, language processing 142 may be unclear about the entity or parameter details for the event (name, location, time, etc.). Language model 140 can request and receive additional sensor data from camera sensor(s) 121 and other sensor(s) 122 to provide context. Other sensor(s) 122 may represent one or more retina tracking sensors, accelerometers, temperature sensors, or some other type of sensor located on wearable device 110. In response to receiving the sensor data, sensor processing 144 may attempt to identify context associated with voice input 130. In some examples, sensor processing 144 provides text recognition, also known as optical character recognition (OCR), which detects and converts written or printed text within images into machine-encoded text. In some examples, sensor processing 144 provides object recognition in images that identifies and classifies objects within an image into predefined categories using computer vision techniques. Sensor processing 144 can further identify the user’s gaze to determine the object the user is focused on, identify the user's orientation, or provide some other context from the sensor data about voice input 130. Here, sensor processing 144 uses the image data from camera sensor(s) 121 to identify text in poster 132, which can supplement or provide context to voice input 130. Although demonstrated within language model 140, sensor processing 144 may be a separate process outside the language model 140 that includes text information from raw sensor data (e.g., text descriptors of objects, OCR text, and the like).

[0029] In computing environment 100, after the context is identified using sensor processing 144, language model 140 and language processing 142 determine an action based on the language of voice input 130 and the text recognition from poster 132. Language model 140 uses word choice, organization, and structure to select an action most relevant to the user's intent. The action can be implemented in local applications 150, user device 160 (if connected), or network devices 170. Using the example in computing environment 100, language model 140 may initiate an action to add event information (date, time, location, name, etc.) to a calendar of user device 160. Language model 140 learns to perform actions through a configuration or training process, which correlates patterns and structures of language to different application actions. Thus, while a first voice input may correspond to a first action in language model 140, a second voice input may correspond to a second action in language model 140.

[0030] In some implementations, the action derived by language model 140 can be associated with a confidence score. A confidence score in a machine learning model represents the model’s estimated probability or certainty that a given prediction is correct (e.g., a score between zero and one). In some examples, language model 140 may not satisfy a threshold confidence score associated with a particular action. For example, the word choice, organization, and structure may suggest a first action, but the confidence score will not satisfy the criteria to initiate the action. Here, language model 140 provides feedback to the user via display 125. The feedback permits the user to confirm the action, change the action, or cancel the action. In at least one implementation, when an action is identified from voice input 130, a notification is provided via display 125 to the user (or an audio queue can be provided to the user). Once provided, the user can confirm, dismiss, or change the action. The user input can be provided via voice instruction, touch feedback via a screen or buttons, or other means. Display 125 may also confirm a completed action (e.g., messaging action, calendar update, and the like).

[0031] Although display 125 is demonstrated to provide feedback, some wearable devices may not include a display for the user. Instead, feedback can be provided via audio, lighting, or other means. For example, wearable device 110 can represent smart glasses with at least one camera sensor to provide context in association with voice prompts and implement the desired actions associated with the user. [0032] FIG. 2 illustrates method 200 of implementing actions based on user prompts and sensor data according to an implementation. Method 200 can be implemented by wearable device 110 of FIG. 1 in some examples (or by another wearable computing device or by another computing system). The steps of method 200 are described below with reference to systems and elements of computing environment 100 of FIG. 1.

[0033] Method 200 includes receiving a voice request at step 201 and obtaining or accessing sensor data from a sensor at step 202 in response to the voice request. In some implementations, a wearable device can be configured with at least one microphone to receive a voice request or voice prompt from the device user. The voice input can be received after the user touches an initiation button, provides an initiation keyword or phrase, or can automatically be detected using the context of the language from the user. In some examples, the user can give an initiating phrase and provide the voice request or input.

[0034] In response to receiving the input, in some implementations, the device can retrieve sensor data associated with one or more sensors to support the request. In retrieving the sensor data, the device can analyze the intent and context of the request and map it to relevant sensors. In some implementations, the sensor data can be selected based on terms and phrases in the voice request. For example, the user can request information associated with temperature or location, which can be derived from a temperature sensor or GPS sensor. The terms and phrases can be used to select the sensors and sensor data to support the request. In some implementations, specific terms and or phrases can be mapped to sensors on the device. This can be provided through a table, array, or other data structure that associates words or phrases with individual sensors. Accordingly, in some examples, rather than using the full language model to determine the sensor data required to support the voice input, the device can map words and phrases to potential sensor data that supports the user input. In some implementations, the language model can be applied to identify intent and potential sensor data for context associated with the request.

[0035] In some implementations, the sensor data can be chosen based on ambiguous terms or phrases associated with entities or objects in the prompt. These words or phrases lack clear context or specificity, making it difficult for the system to interpret user intent accurately. Examples include vague commands like “show me that,” “make it bigger,” or “move it over there,” which require additional spatial or contextual information to be correctly understood.

[0036] In some examples, with these words and phrases, the system can use imaging data from at least one camera sensor to provide the context associated with the user request. In some examples, the device can use a language model to process the request to determine the user’s intent and the additional contextual information to support the intent. In some implementations, the device can be configured to provide image-to-text conversion, which is extracting textual information from an image using technologies like Optical Character Recognition (OCR). It allows computers to recognize and convert printed or handwritten text in images into machine-readable text for editing, searching, or further processing. This can be used to derive information from posters, signs, and other elements within the physical space. Referring to the example from computing environment 100, in response to voice input 130, language model 140 can be configured to process the request and determine that image-to- text information is required to support the request based on the identified user intent. Camera sensor(s) 121 can capture an image of poster 132 and recognize text in the image associated with a date, time, location, or other relevant information related to the request. The relevant text can be determined using the user’s gaze in some examples to limit the text provided for the model. For example, only a portion of poster 132 may be identified as relevant based on the user’s gaze.

[0037] In some implementations, when the user initiates a voice command, using a button or a verbal trigger, the device can collect image data during voice input from the user. If required in association with the voice input, the collected image data (i.e., one or more images) can be processed to provide context in association with the verbal input. Thus, while the image data is gathered during the voice input from the user, the image data can be selected for context based on the intent and terminology in the input. For example, when the user requests to add an event to the calendar, the device can process images captured during the voice request to provide context in association with the event.

[0038] In some implementations, the device can be configured to use different sensor data processing to provide the requisite information associated with the request. In some examples, the sensor data can include object recognition data for objects captured in one or more images by the device. Image recognition for objects is a computer vision process where algorithms identify and classify objects within an image. It enables systems to detect and label items like people, animals, or other objects using machine-learning models trained on large datasets. In some examples, the sensor data can include audio data, or audio-to-text data from the device. For example, the user can provide a request to “add this song to my playlist.” In response to the request, the device can receive audio data associated with the environment to identify the song referenced by the user. [0039] Method 200 further includes applying a model to the voice request and the sensor data at step 203 to determine an application associated with the voice request and an action associated with the application (e.g., an action associated with the application and the voice request). In some implementations, a device can apply a machine learning model that takes both the voice request (in some examples converted to text) and relevant sensor data as inputs to understand the full context of the user’s intent. The model processes this combined input to classify which application (e.g., among a given list of applications) is best suited to handle the request (e.g., maps, camera, health tracker) based on patterns learned during training. It then uses intent recognition to determine the specific action the application should perform (e.g., “start navigation,” “take a photo,” “record heart rate”), enabling the system to respond accurately and contextually. Method 200 further causes the application to initiate the action at step 204.

[0040] As an illustrative example, referring to computing environment 100 of FIG. 1, device 110 uses language model 140 to process voice input 130 and related sensor data originating from camera sensor(s) 121 and other sensor(s) 122 to generate an action. In at least one implementation, the text information identified in poster 132 can provide context required to add a new event to a calendar in local applications 150. Actions can also include providing information to user device 160 or network devices 170. The voice input and context can be used to navigate menus, launch applications, update applications, control virtual objects, and perform hands-free interactions like selecting items, dictating text, or adjusting settings. It also enables more immersive experiences, such as giving voice commands to virtual characters or triggering environment changes.

[0041] In some examples, the action derived by language model 140 can be associated with a confidence score. A confidence score in a machine learning model represents the model’s estimated probability or certainty that a given prediction is correct (e.g., a score between zero and one). In some examples, language model 140 may not satisfy a threshold confidence score associated with a particular action. For example, the word choice, organization, and structure may suggest a first action, but the confidence score will not satisfy the criteria to initiate the action. Here, language model 140 provides feedback to the user via display 125. The feedback permits the user to confirm the action, change the action, or cancel the action. In at least one implementation, when an action is identified from voice input 130, a notification is provided via display 125 to the user (or an audio queue can be provided to the user). Once provided, the user can confirm, dismiss, or change the action. The user input can be provided via voice instruction, touch feedback via a screen or buttons, or other means. Display 125 may also confirm a completed action (e.g., messaging action, calendar update, and the like).

[0042] FIG. 3 illustrates an operational scenario 300 of implementing actions based on user prompts and sensor data according to an implementation. Operational scenario 300 includes user 310, input 320, model operations 330, and actions 340 and 341. Model operations 330 represent operations that can be provided by a wearable device, such as a head-worn device or XR device. Model operations 330 can be performed by wearable device 110 of FIG. 1 in some implementations.

[0043] In operational scenario 300, user 310 provides input 320 that corresponds to a voice prompt from the user. In response to input 320, the wearable device provides model operations 330 to provide one or more actions associated with input 320. Model operations 330 may identify sensor data 332 associated with input 320. In some implementations, identify sensor data 332 can process a voice prompt on an XR device by interpreting the user's intent and extracting contextual cues to determine which sensors are needed to fulfill the request. For example, when user 310 says, "Where can I get this?" while looking at or gesturing toward a coffee cup (or other object), the language model uses the prompt in combination with contextual data like eye tracking, hand tracking, and object identification 334 (e.g., at least one of eye tracking and hand tracking, and, additionally, object identification 334) from image data 333 to identify what “this” refers to. Once the object is identified, in this example as a cup of coffee, the model infers the user’s intent (e.g., finding purchase information or nearby locations) and determines that location services (GPS), internet connectivity, and potentially product databases are required to provide an accurate answer. This sensor-driven context enables the system to identify actions 335 relevant to the user.

[0044] In some implementations, the relevant actions can include executing an application, such as a mapping application, and selecting a location to purchase the corresponding object. In some examples, the wearable device can use application programming interfaces (APIs) to execute and interact with the one or more applications. In some implementations, the APIs can provide at least some of the information gathered from the sensors, such as text or objects identified from the sensors. Model operations 330 can also be configured to use APIs to interact with the one or more sensors and process the sensor data from each of the sensors. The processing can include identifying text in images, object identification, or some other processing. The processing can also include identifying the user’s gaze, identifying gestures, or some other processing associated with the raw data from the sensors to identify sensor data relevant to the user input 320.

[0045] FIG. 4 illustrates an operational scenario of implementing actions based on user prompts and sensor data according to an implementation. Operational scenario 400 includes user 410, input 420, model operations 430, and actions 440 and 441. Model operations 430 represent operations that can be provided by a wearable device, such as a head- worn device or XR device. Model operations 430 can be performed by wearable device 110 of FIG. 1 in some implementations.

[0046] In operational scenario 400, user 410 provides input 420 that corresponds to a voice prompt from the user. In response to input 420, the wearable device provides model operations 430 to provide one or more actions associated with input 420. Model operations 430 may identify sensor data 432 associated with input 420. In some implementations, identify sensor data 432 can process a voice prompt on an XR device by interpreting the user's intent and extracting contextual cues to determine which sensors are needed to fulfill the request. For example, in operational scenario 400, when the user provides input 420 for “add a reminder for this,” the device can use a language model as part of identify sensor data 432. The language model is used detect that “this” refers to something in the user’s cunent view. Based on that, the device determines it needs contextual input and activates relevant sensors, such as a sensor capable of capturing image data 433. In some implementations, other sensors can be used, such as sensors that identify user gestures (e.g., pointing at an object), user gaze, or some other relevant contextual information. From image data 433, the device can perform OCR (text identification 434) to identify text 450, text 451, and text 452 relevant to the intent of input 420.

[0047] In some implementations, the device uses the language model to map the user's intent, like setting a reminder, to a specific category of actions, such as task or time management. The device is then configured to match this category to the appropriate application using identify actions 435, like a built-in reminders or calendar app. Actions 440 and 441 can be used to execute the application and update the corresponding application. In some examples, the determined intent can be mapped to an application in the set of applications on the device. In some examples, the model can interact with the various applications using an API. The API can execute the application, provide relevant information to the application, and implement various actions within the application (such as updating a reminder application). In some examples, the actions can also be used to interact with applications on external devices (e.g., smartphones and tablets). For example, a voice command can cause an action in association with an application on a smartphone (e.g., to generate a call or add a contact to a contacts application on the connected companion device).

[0048] In some examples, the wearable device can determine a confidence score associated with an action derived from applying the language model to input 420 and the identified text. A confidence score for a language model can represent how confident the model is about its interpretation or prediction. In some implementations, the value is expressed numerically, e.g., between 0 and 1. It can help the device decide whether to act on the model’s output directly or request more input or clarification from the user. In some implementations, the device can implement a threshold that, when satisfied, the device will implement the action determined from the language model. In some implementations, when the threshold is not satisfied, the device can provide options to the user, such as implementing the determined action, changing to a different action, providing additional context, or some other information. In some examples, the options are provided via a display on the device. In some examples, the user provides additional input to select one of the options or provides additional context associated with the task.

[0049] Although demonstrated in the examples of FIG. 3 and FIG. 4 as using image data for the sensor data, a device can identify and use other sensor data associated with a request. The sensor data can be gathered from microphones (e.g., for a song or information provided by another user), from GPS sensors, or from various other sensors. In some implementations, the device can use a collection of sensors to gather the requisite information to complete the intended action or task.

[0050] FIG. 5 illustrates a computing system 500 for implementing actions based on user prompts and sensor data according to an implementation. Computing system 500 represents any apparatus, computing system, or systems with which the various operational architectures, processes, scenanos, and sequences are disclosed herein for providing actions based on user input can be implemented. Computing system 500 can be an example of an XR device, wearable device, or other computing device capable of the operations described herein. Computing system 500 can be a system of devices, such as a wearable device and a companion device (e.g., smartphone, tablet, etc.) in some examples. Computing system 500 can be representative of wearable device 110 of FIG. 1 in some examples. Computing system 500 includes storage system 545, processing system 550, communication interface 560, and input/output (I/O) device(s) 570. Processing system 50 is operatively linked to communication interface 560, I/O device(s) 570, and storage system 545. In some implementations, communication interface 560 and/or I/O device(s) 570 may be communicatively linked to storage system 545. Computing system 500 may further include other components such as a battery and enclosure that are not shown for clarity.

[0051] Communication interface 560 comprises components that communicate over communication links, such as network cards, ports, radio frequency, processing circuitry (and corresponding software), or some other communication devices. Communication interface 560 may be configured to communicate over metallic, wireless, or optical links. Communication interface 560 may be configured to use Time Division Multiplex (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format - including combinations thereof. Communication interface 560 may be configured to communicate with external devices, such as servers, user devices, or other computing devices. In some examples, communication interface 560 can communicate with at least one companion device (e.g., smartphone or tablet) associated with the wearable device.

[0052] I/O device(s) 570 may include peripherals of a computer that facilitate the interaction between the user and computing system 500. Examples of I/O device(s) 570 may include keyboards, mice, trackpads, monitors, displays, printers, cameras, microphones, external storage devices, sensors, and the like. In some implementations, I/O device(s) 570 include at least one outward-facing camera configured to capture images associated with the physical environment. In some implementations, I/O device(s) 570 consists of a see-through or video pass-through display providing a view of the physical environment. In some implementations, the computing system 500 can include a display capable of displaying an indicator or other content. In some examples, I/O device(s) 570 can include one or more sensors and cameras capable of identifying objects and information in the physical environment.

[0053] Processing system 550 comprises microprocessor circuitry (e.g., at least one processor) and other circuitry that retrieves and executes operating software (i.e., program instructions) from storage system 545. Storage system 545 may include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Storage system 545 may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems. Storage system 545 may comprise additional elements, such as a controller to read operating software from the storage systems. Examples of storage media (also referred to as computer-readable storage media or a computer-readable storage medium) include random access memory, read- only memory, magnetic disks, optical disks, and flash memory, as well as any combination or variation thereof, or any other type of storage media. In some implementations, the storage media may be non-transitory. In some instances, at least a portion of the storage media may be transitory'. In no case is the storage media a propagated signal.

[0054] Processing system 550 is typically mounted on a circuit board that may hold the storage system. The operating software of storage system 545 comprises computer programs, firmware, or some other form of machine-readable program instructions. The operating software of storage system 545 comprises language model application 524. The operating software on storage system 545 may further include an operating system, utilities, drivers, network interfaces, applications, or some other type of software. When read and executed by processing system 550 the operating software on storage system 545 directs computing system 500 to operate as described herein. In at least one implementation, the operating software can provide method 200 described in FIG. 2. The operating software can provide or cause the at least one processor to manage actions from user prompts and contextual information derived from sensors as described herein.

[0055] In at least one implementation, language model application 524 directs processing system 550 to receive a voice request and obtain sensor data from a sensor in response to the voice request. In some examples, language model application 524 can interpret the user’s voice request and identify key elements like objects, actions, or references (e.g., “this” or “here”). Based on this interpretation, the model determines which contextual data is needed and maps that to the required sensors, such as eye tracking for gaze, cameras for object recognition, or GPS for location. The device then activates or queries those sensors to gather the necessary information to complete the request. In some examples, language model application 524 can process the raw sensor data to refine or determine sensor data relevant to the user prompt. In some examples, this can include object detection in images. In some examples, this can consist of image-to-text operations. In some examples, the sensor data can correspond to audio, temperature, location, or other information associated with the device. In some implementations, the sensor data can combine any set of sensors and data described herein.

[0056] Language model application 524 directs processing system 550 to determine an application associated with the voice request and determine an action associated with the application. Once determined, language model application 524 directs processing system 550 to cause the application to initiate the action. In some implementations, language model application 524 can change the voice to text and process the text to identify the user’s intent and relevant entities (e.g., objects and text identified via sensor data), generating a structured command that can map to an action for an application.

[0057] Example claim clauses are provided below. Although these are examples, these clauses should not be considered exhaustive.

[0058] Clause 1. A method comprising: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

[0059] Clause 2. The method of clause 1, further comprising: processing the voice request by the model to identify the sensor as related to the voice request.

[0060] Clause 3. The method of clause 1 or 2, wherein accessing the sensor data from the sensor comprises: performing object recognition on at least one image from the sensor to identify text descriptors for at least one object in the at least one image; and using the text descriptors for the at least one object as the sensor data.

[0061] Clause 4. The method of any one of the preceding clauses, wherein accessing the sensor data from the sensor comprises: performing text recognition on at least one image from the sensor to identify text in the at least one image; and using the text as the sensor data.

[0062] Clause 5. The method of any one of the preceding clauses, wherein applying the model to the voice request and the sensor data comprises determining a confidence score associated with the action, and the method further comprises: determining that the confidence score satisfies a threshold; and causing the application to initiate the action in response to determining that the confidence score satisfies the threshold.

[0063] Clause 6. The method of any one of the preceding clauses, wherein the sensor data comprises text captured from at least one image, and wherein causing the application to initiate the action comprises: communicating at least a portion of the text to the application via an application programming interface.

[0064] Clause 7. The method of any one of the preceding clauses, further comprising: displaying feedback indicating a status of the action.

[0065] Clause 8. The method of any one of the preceding clauses, further comprising: accessing additional sensor data from at least one additional sensor; wherein applying the model to the voice request and the sensor data comprises applying the model to the voice request, the sensor data, and the additional sensor data.

[0066] Clause 9. A computing system comprising: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the computing system to perform a method, the method comprising: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

[0067] Clause 10. The computing system of clause 9, wherein the method further comprises: processing the voice request by the model to identify the sensor as related to the voice request.

[0068] Clause 11. The computing system of clause 9 or 10, wherein accessing the sensor data from the sensor comprises: performing object recognition on at least one image from the sensor to identify text descriptors for at least one object in the at least one image; and using the text descriptors for the at least one object as the sensor data.

[0069] Clause 12. The computing system of any one of clauses 9 to 11, wherein accessing the sensor data from the sensor comprises: performing text recognition on at least one image from the sensor to identify text in the at least one image; and using the text as the sensor data.

[0070] Clause 13. The computing system of any one of clauses 9 to 12, wherein applying the model to the voice request and the sensor data comprises determining a confidence score associated with the action, and the method further comprises: determining that the confidence score satisfies a threshold; and causing the application to initiate the application in response to determining that the confidence score satisfies the threshold.

[0071] Clause 14. The computing system of any one of clauses 9 to 13, wherein the sensor data comprises text captured from at least one image, and wherein causing the application to initiate the action comprises: communicating at least a portion of the text to the application via an application programming interface.

[0072] Clause 15. The computing system of any one of clauses 9 to 14, wherein the method further comprises: displaying feedback indicating a status of the action.

[0073] Clause 16. The computing system of any one of clauses 9 to 15, wherein the method further comprises: accessing additional sensor data from at least one additional sensor; wherein applying the model to the voice request and the sensor data comprises applying the model to the voice request, the sensor data, and the additional sensor data.

[0074] Clause 17. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

[0075] Clause 18. The computer-readable storage medium of clause 17, wherein the method further comprises: processing the voice request by the model to identify the sensor as related to the voice request.

[0076] Clause 19. The computer-readable storage medium of clause 17 or 18, wherein accessing the sensor data from the sensor comprises: performing object recognition on at least one image from the sensor to identify text descriptors for at least one object in the at least one image; and using the text descriptors for the at least one object as the sensor data.

[0077] Clause 20. The computer-readable storage medium of any one of clauses 17 to 19, wherein accessing the sensor data from the sensor comprises: performing text recognition on at least one image from the sensor to identify text in the at least one image; and using the text as the sensor data.

[0078] In this specification and the appended claims, the singular forms “a,” “an,” and “the” do not exclude the plural reference unless the context dictates otherwise. Further, conjunctions such as “and,” “or,” and “and/or” are inclusive unless the context dictates otherwise. For example, “A and/or B” includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and/or physical or logical couplings between the various elements. Many alternative or additional functional relationships, physical connections, or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the implementations disclosed herein unless the element is specifically described as “essential” or “critical.”

[0079] Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that a precise value or range thereof is not required and need not be specified. As used herein, the terms discussed above will have ready and instant meaning to one of ordinary skill in the art.

[0080] Moreover, terms such as up, down, top, bottom, side, end, front, back, etc. are used herein concerning a currently considered or illustrated orientation. If they are considered concerning another orientation, such terms must be correspondingly modified.

[0081] Further, in this specification and the appended claims, the singular forms “a,” “an” and “the” do not exclude the plural reference unless the context dictates otherwise. Moreover, conjunctions such as “and,” “or,” and “and/or” are inclusive unless the context dictates otherwise. For example, “A and/or B” includes A alone, B alone, and A with B.

[0082] Although certain example methods, apparatuses, and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It is to be understood that the terminology employed herein is to describe aspects and is not intended to be limiting. On the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

2. The method of claim 1, further comprising: processing the voice request by the model to identify the sensor as related to the voice request.

3. The method of claim 1 or 2, wherein accessing the sensor data from the sensor comprises: performing object recognition on at least one image from the sensor to identify text descriptors for at least one object in the at least one image; and using the text descriptors for the at least one object as the sensor data.

4. The method of any one of the preceding claims, wherein accessing the sensor data from the sensor comprises: performing text recognition on at least one image from the sensor to identify text in the at least one image; and using the text as the sensor data.

5. The method of any one of the preceding claims, wherein applying the model to the voice request and the sensor data comprises determining a confidence score associated with the action, and the method further comprises: determining that the confidence score satisfies a threshold; and causing the application to initiate the action in response to determining that the confidence score satisfies the threshold.

6. The method of any one of the preceding claims, wherein the sensor data comprises text captured from at least one image, and wherein causing the application to initiate the action comprises: communicating at least a portion of the text to the application via an application programming interface.

7. The method of any one of the preceding claims, further comprising: displaying feedback indicating a status of the action.

8. The method of any one of the preceding claims, further comprising: accessing additional sensor data from at least one additional sensor; wherein applying the model to the voice request and the sensor data comprises applying the model to the voice request, the sensor data, and the additional sensor data.

9. A computing system comprising: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the computing system to perform a method, the method comprising: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

10. The computing system of claim 9, wherein the method further comprises: processing the voice request by the model to identify the sensor as related to the voice request.

11. The computing system of claim 9 or 10, wherein accessing the sensor data from the sensor comprises: performing object recognition on at least one image from the sensor to identify text descriptors for at least one object in the at least one image; and using the text descriptors for the at least one object as the sensor data.

12. The computing system of any one of claims 9 to 11, wherein accessing the sensor data from the sensor comprises: performing text recognition on at least one image from the sensor to identify text in the at least one image; and using the text as the sensor data.

13. The computing system of any one of claims 9 to 12, wherein applying the model to the voice request and the sensor data comprises determining a confidence score associated with the action, and the method further comprises: determining that the confidence score satisfies a threshold; and causing the application to initiate the action in response to determining that the confidence score satisfies the threshold.

14. The computing system of any one of claims 9 to 13, wherein the sensor data comprises text captured from at least one image, and wherein causing the application to initiate the action comprises: communicating at least a portion of the text to the application via an application programming interface.

15. The computing system of any one of claims 9 to 14, wherein the method further comprises: displaying feedback indicating a status of the action.

16. The computing system of any one of claims 9 to 15, wherein the method further comprises: accessing additional sensor data from at least one additional sensor; wherein applying the model to the voice request and the sensor data comprises applying the model to the voice request, the sensor data, and the additional sensor data.

17. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising: receiving a voice request; in response to the voice request, accessing sensor data from a sensor; applying a model to the voice request and the sensor data to: determine an application associated with the voice request, and determine an action associated with the application; and causing the application to initiate the action.

18. The computer-readable storage medium of claim 17, wherein the method further comprises: processing the voice request by the model to identify the sensor as related to the voice request.

19. The computer-readable storage medium of claim 17 or 18, wherein accessing the sensor data from the sensor comprises: performing object recognition on at least one image from the sensor to identify text descriptors for at least one object in the at least one image; and using the text descriptors for the at least one object as the sensor data.

20. The computer-readable storage medium of any one of claims 17 to 19, wherein accessing the sensor data from the sensor comprises: performing text recognition on at least one image from the sensor to identify text in the at least one image; and using the text as the sensor data.