WO2025174372A1

WO2025174372A1 - Visual gesture detection using text descriptions

Info

Publication number: WO2025174372A1
Application number: PCT/US2024/015978
Authority: WO
Inventors: Ruofei DU; Zhongyi Zhou
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2024-02-15
Filing date: 2024-02-15
Publication date: 2025-08-21
Anticipated expiration: 2026-08-15

Abstract

In described techniques, an image of a gesture may be received, and a numerical representation of the image may be generated. A comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture may be determined. Then, the gesture may be identified as corresponding to the stored gesture, based on the comparison.

Description

VISUAL GESTURE DETECTION USING TEXT DESCRIPTIONS

BACKGROUND

[0001] Gesture detection generally refers to an inference of meaning or intent that is derived from a person’s movement(s). For example, a hand gesture may refer to a particular configuration of a person’s hand that indicates a desired action to be taken. Gesture detection may be performed using machine learning (ML) models, such as a model configured to identify a particular hand gesture from among a defined set of known hand gestures.

SUMMARY

[0002] Described techniques enable fast and efficient generation of a gesture classifier for virtually any desired hand gesture, or set of hand gestures, with little or no technical knowledge and model training being required. For example, an application developer or other user may generate a gesture classifier for the gestures of “rock, paper, scissors,” simply by request. The resulting, generated classification model may then be used in conjunction with a camera or other image sensor to distinguish between the defined hand gestures, as such hand gestures occur in real time or near real time.

[0003] In a general aspect, a method includes receiving an image of a gesture, generating a numerical representation of the image, determining a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture, and identifying the gesture as corresponding to the stored gesture, based on the comparison.

[0004] In another general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and comprises instructions. When executed by at least one computing device (e.g., by at least one processor of the computing device), the instructions are configured to cause the at least one computing device to receive an image of a gesture, generate a numerical representation of the image, determine a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture, and identify the gesture as corresponding to the stored gesture, based on the comparison.

[0005] In another general aspect, a head-mounted device (HMD) includes at least one frame for positioning the wearable device on a body of a user, at least one display, at least one processor, and at least one memory storing instructions. When executed, the instructions cause the at least one processor to receive an image of a gesture, generate a numerical representation of the image, determine a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture, and identify the gesture as corresponding to the stored gesture, based on the comparison.

[0006] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 A illustrates an example gesture classification model generation process.

[0008] FIG. IB illustrates an example use of a gesture classifier constructed using the techniques of FIG. 1A.

[0009] FIG. 2 is a block diagram of a system of visual detection of gestures using text descriptions.

[0010] FIG. 3 is a flowchart illustrating example operations of the system of FIG. 2.

[0011] FIG. 4 is a block diagram of a more specific example implementation of the system of FIG. 2.

[0012] FIG. 5 illustrates example hand landmarks that may be used in the system of FIG. 4.

[0013] FIG. 6 is a flowchart illustrating a second example of more detailed operations of the system of FIGS. 2 and 4.

[0014] FIG. 7 is a third person view of a user in an ambient computing environment.

[0015] FIGS. 8A and 8B illustrate front and rear views of an example implementation of a pair of smartglasses.

DETAILED DESCRIPTION

[0016] Described systems and techniques enable rapid generation of a customized, real-time gesture classifier, with little or no training or fine-tuning. Application developers, including game developers, may be enabled to generate virtually any desired gesture classifier for inclusion in applications being developed. Moreover, users may utilize developed classifiers in a wide variety of contexts, in a fast and intuitive manner.

[0017] Various types of gesture classifiers are known. For example, gesture classifiers are used in computing environments to recognize a hand gesture that indicates a user interface command. In other examples, a gesture classifier may be trained to recognize alphabet letters in American Sign Language (ASL).

[0018] Known gesture classifiers typically have at least the technical problem(s) of either requiring extensive training and/or being unable to operate sufficiently close to real time. Moreover, development of gesture classifiers using known techniques may involve a level of technical expertise that is not widely available, so that some developers may avoid or limit use of gesture classifiers in developed applications.

[0019] Therefore, at least one technical problem facing developers is the difficulty in obtaining a gesture classifier, and particularly in obtaining a customized gesture classifier that might be needed for a particular application or other use. For example, a model may be trained to recognize a set of hand gestures, using training data that includes labelled examples of each gesture. However, such models may be difficult to update or customize, so developers may be prone to include a larger set of hand gestures than may actually be needed by a given user. In addition to increasing training data and training time, such approaches may limit a speed at which gestures are recognized during runtime. Moreover, such approaches may be prone to overfitting or other difficulties that may limit an accuracy of a trained gesture classifier.

[0020] Consequently, the technical solutions described herein solve at least one technical problem including complex training and lengthy /laborious training times to obtain gesture classifiers. At least one technical problem solved by the described techniques includes undesirably slow gesture classifiers. At least one technical problem solved by the described techniques includes gesture classifiers that do not include desired gestures. At least one technical problem solved by the described techniques includes the inability to control a user interface or enact other desired aspects of application control using gestures.

[0021] The technical solutions described herein use a visual transformer, encoder, or other type of model (e.g., an artificial intelligence (Al) model) that is configured (e.g., trained) to recognize (e.g., identify) a defined set of landmarks, such as hand landmarks (e.g., finger/joint position, palm). For example, such a visual transformer may characterize one or more fingers as being raised or curled, but without classifying each combination of such finger positions as a specific gesture.

[0022] Instead, the visual transformer may output a numerical representation of a current gesture. Such a numerical representation may include any representation of a gesture image, or portion of a gesture image, using one or more numbers. For example, as described in more detail, below, a numerical representation of a gesture image may be determined as a visual embedding that relates to or characterizes a hand position of a gesture image.

[0023] Meanwhile, a text transformer, encoder or other model may input a textual description of a new, specific set of hand gestures to be classified, such as when a developer wishes to develop a “rock, paper, scissors” game. A model (e.g., a large language model (LLM)) may be leveraged to generate a lengthy textual description of each desired gesture. For example, a LLM may be prompted to generate a verbal description of a hand imitating “rock,” and may output “curling all fingers into the palm of the hand, with the thumb overlapping the index and middle fingers.”

[0024] Then, a text transformer may encode the textual description of each desired gesture into a corresponding numerical representation of the textual description. As with the numerical representation of an output of the visual transformer referenced above, the numerical representation of the textual description may include any number(s) used to represent the textual description. For example, as described in more detail, below, a numerical representation of the textual description may be determined as a text embedding that relates to or characterizes the textual description.

[0025] Then, for example, a resulting model may be deployed that includes the pregenerated text embeddings and the visual transformer, along with a comparator configured to compare current, real-time visual embeddings against the pre-stored textual embeddings. At runtime, the visual transformer may generate a real-time visual embedding of a current hand gesture, which may then be compared for similarity to each of the pre-generated textual embeddings to classify the current hand gesture (e.g., may be compared to determine a higher or highest similarity or similarity score). Any suitable measure of numerical similarity may be used. For example, the visual embeddings and the textual embeddings may be generated in a common vector space, and similarity may be based on a Euclidean distance between the two embeddings within the vector space (e.g., cosine similarity).

[0026] Advantageously, the visual transformer need not have been configured (e.g., trained) with the exact or specific gestures matching the set of gestures included in the generated gesture classifier model (in other examples, the visual transformer may be finetuned if desired). Therefore, it may not be necessary to provide any additional configuring (e.g., training) or fine-tuning of either the text transformer or the visual transformer when generating a desired gesture classifier.

[0027] Therefore, at least one technical solution provided by the described techniques includes using generating a gesture classifier for desired, specified gestures on-demand, with little or no training required, and in response to simple verbal prompts. At least one technical solution provided by described techniques includes recognizing the requested gestures in real time or near real time, with high levels of accuracy. At least one technical solution provided by described techniques includes using a text transformer to generate and store text embeddings for a desired set of gestures, and a visual transformer that compares a visual embedding of a current gesture to each of the text embeddings to determine a corresponding gesture being used.

[0028] In the example of FIG. 1 A, a user 102 may represent an application developer or other user who desires to develop a game, user interface control, application, or other use for a set of recognized gestures. In the example, the user 102 provides a prompt 104, which may be written/typed or verbal.

[0029] In the example of FIG. 1 A, examples are provided with respect to a prompt that specifies hand gestures for a game of ‘rock, paper, scissors.’ Many of the following example implementations also describe this specific use case. Of course, it will be appreciated that such a use case, and included hand gestures (and other types of gestures), are not limiting, and are included merely for the sake of example and explanation. Various other examples are provided as well, which should also be understood to be non-limiting.

[0030] A gesture classifier generator 106 may input the prompt 104 and output a corresponding gesture classifier 108. Consequently, the user 102 may obtain the gesture classifier 108 with little or no technical skill, and for virtually any desired set of gestures.

[0031] For example, other prompts may include “a gesture classifier for thumbs up or thumbs down” to indicate approval or disapproval may be generated. In other examples, “a gesture classifier for pointing or encircling” may be used to obtain a gesture classifier for UI control/sel ection. In other examples, essentially random gestures may be specified for recognition for a desired purpose of the user 102. For example, “a gesture classifier for the number 1, the number 2, the number 3” may be used to select between three options that the user 102 develops as part of an application being built.

[0032] As further illustrated in FIG. 1 A, interactions between the user 102 and the gesture classifier generator 106 may be iterative, as the user 102 may be provided with an ability to revise intermediate outputs of the gesture classifier 108 if needed. For example, as referenced above and described in more detail, below, the gesture classifier 108 may output intermediate results including verbal descriptions of each of the specified gestures, and the user 102 may thus be provided with opportunities to provide updates or revisions to the prompt 104 that ensure accuracy and desired outcome with respect to the gesture classifier 108. [0033] In FIG. IB, the gesture classifier 108 constructed in FIG. lAmay be deployed for use by a user 103, which may represent the user 102 of FIG. 1 A or a different user. For example, the user 103 may make a gesture 112, shown in FIG. IB as a hand open with fingers and thumb extended and palm facing outward.

[0034] A camera 114 may capture an image of the gesture 112, which may then be provided to the gesture classifier 108. The gesture classifier 108 may recognize the gesture 112 from among the defined set of ‘rock, paper, scissors’ as corresponding to ‘paper,’ and a display 116 may thus designate the gesture 112 as ‘paper.’

[0035] In the simplified example of FIG. IB, the display 116 displays ‘paper,’ but in various example implementations and use cases, the gesture classifier 108 may recognize the gesture 112 as ‘paper’ and then perform some associated, pre-designated task defined by the user 102 when constructing the gesture classifier 108. For example, a specified UI control action may be taken. Or, in the context of a game of ‘rock, paper, scissors,’ a second user may simultaneously make a gesture of ‘rock,’ and the game may provide an indication of the user 103 winning the game, or an output of ‘paper covers rock.’

[0036] FIG. 2 illustrates a gesture classifier generator 206 as an example implementation of the gesture classifier 108 of FIGS. 1 A and IB, and a gesture classifier 208 as an example implementation of the gesture classifier 108 of FIGS. 1 A and IB. For example, the gesture classifier generator 206 may be implemented on a suitable computing device or platform, examples of which are provided below, e.g., with respect to FIG. 7.

[0037] In the example of FIG. 2, the gesture classifier generator 206 includes, or utilizes, a large language model (LLM) 202. The LLM 202 represents a type of artificial intelligence algorithm using deep learning techniques to provide language generation. The LLM 202 is an example of neural network models, and other neural network models may be used, as well.

[0038] LLMs, such as the LLM 202, are generally used to identify, distinguish, translate, summarize, predict, and/or generate text, in response to a prompt or other request. LLMs may be trained using extremely large datasets to determine statistical associations between words in the datasets, and then may be fine-tuned for specific purposes, if needed.

[0039] A prompt handler 204 may be configured to receive a prompt, such as the prompt 104 of FIG. 1A, for desired gestures to be included in the gesture classifier 208 to be generated. The prompt handler 204 may receive the prompt 104, or other suitable prompt, and provide the prompt to the LLM 202. The LLM 202 may then generate and provide textual descriptions of each of the desired hand gestures. [0040] As the LLM 202 provides automated text generation, the LLM 202 is capable of providing relatively lengthy or verbose descriptions, as needed, of desired hand gestures. A text embedder 210 may be configured to input the textual descriptions and output numerical representations of the textual descriptions, shown in FIG. 2 as text embeddings 212. That is, a text embedding may be generated for each textual description of each requested gesture.

[0041] In example implementations, as described in more detail, below, the text embedder 210 may be implemented using a text transformer. A transformer in this context refers generally to a component or set of algorithms originally developed for use in large language models, such as the LLM 202, and the LLM 202 may include a separate transformer used in that context.

[0042] In general, a transformer refers to a deep learning model or algorithm that uses a self-attention mechanism in which different parts of input data are weighted differently to provide or indicate context. For example, for a single word that has different meanings in different contexts (such as ‘bank’ when referring to a river bank or a financial institution), weights may be added to words in a sentence to indicate context between words.

[0043] For these and related reasons, words and other text may be represented as embeddings. Embeddings refer to numerical representations of words within a common or single graph space, and may also be referred to as vectors within a common or single vector space. That is, the numerical representation may represent an embedding or vector that is encoded in a manner that results in similar words being close to one another within a highdimensional vector space, while dissimilar words are farther from one another within the high-dimensional vector space (which may also be referred to as an embedding space or a latent space).

[0044] Many types of text embedding algorithms exist or may be developed, and are often used in various Al contexts and use cases, such as performing searches or providing content moderation. Examples in which text embedding or text encoding is included or provided include, in addition to transformers in LLMs as already mentioned, the word2vec algorithm, the CLIP model (Contrastive Language-Image Pre-training), or variations of the BERT (Bidirectional Encoder Representations from Transformers) may be used.

[0045] Thus, the prompt handler 204 may provide the prompt 104 to the LLM 202, and then provide the resulting textual descriptions to the text embedder 210 to obtain the text embeddings 212. In example implementations, as referenced with respect to FIG. 1 A, the preceding operations may be performed iteratively to ensure accuracy.

[0046] For example, upon receiving the prompt 104 and using the LLM 202 to generate the textual descriptions, the prompt handler 204 may interact with the user 102 to verify accuracy and sufficiency of the textual descriptions. For example, when the prompt 104 includes the requested gesture ‘rock,’ the prompt handler 204 may initially return a textual description of a gesture used to refer to ‘rock and roll,’ such as “index finger and pinky finger extended with thumb wrapped around middle and ring fingers curled into palm.” The user 102 may then specify that the desired rock gesture refers to a stone, and may receive an updated gesture description of, “curling all fingers into the palm of the hand, with the thumb overlapping the index and middle fingers.”

[0047] In other examples, the generated textual description may be accurate, but may be insufficient or too short for purposes of obtaining a useful text embedding. For example, as referenced above and described in detail, below, lengthy or verbose descriptions may result in some cases in more useful text embeddings that are more likely to result in accurate gesture recognition by the gesture classifier 208.

[0048] Therefore, the prompt handler 204 may be configured to ensure a minimum length, number of words, or other measure of quantity of description of each textual description. For example, description thresholds may be set for these types of measures, and the prompt handler 204 may request longer or additional descriptions from the LLM 202 if the thresholds are not met. In other examples, the thresholds may be added to the prompt 104 by the prompt handler 204. In other examples, the user 102 may be provided with an ability to request longer textual descriptions, in addition to disambiguating gestures and associated descriptions.

[0049] The gesture classifier generator 206 may further include, or have access to, training data 215 for training a visual embedder 216. Just as the text embedder 210 inputs textual descriptions and outputs the text embeddings 212, the visual embedder 216 may be trained and otherwise configured to receive a gesture image and output a visual embedding (shown as visual embeddings 218 at the gesture classifier 208, as described below).

[0050] As described in more detail, below, the training data 215 may be a relatively generic training data set, in that the training data 215 need not include labelled images of specific gesture images that might later be encountered by the gesture classifier 208 at runtime. For example, the training data 215 may include training images of gestures in which hands/fingers are described with respect to hand landmarks, as described in more detail below with respect to FIG. 5. As a result, the training data 215 used for the visual embedder 216 may be small compared to conventional training data for existing gesture recognition models, and may be widely applicable to virtually any type of gesture that might be stored in the context of the text embeddings 212.

[0051] The training data 215, or other training data, may be used in training the text embedder 210. For example, the training data 215 may include text descriptions of gestures that state a placement of fingers and poses of the fingers. For example, the LLM 202, or another LLM, may be used to generate a gesture dataset from provided, approved labels, in order to improve operations of the text embedder 210. Similar to the visual embedder 216, the training data 215 for the text embedder need not include all gestures that might later be received by the prompt handler 204. Rather, both the text embedder 210 and the visual embedder 216 may be trained on a wide range of available, labelled gestures, sufficient to enable techniques described herein.

[0052] Once generated, the text embeddings 212 may be stored in, or associated with, the gesture classifier 208 by the gesture classifier generator 206. The gesture classifier 208 may be generated as including an image handler 214 that may be configured to interact with the camera 114 of FIG. IB to receive a gesture image, such as the image of the gesture 112 in FIG. IB. The trained visual embedder 216 may be deployed to the gesture classifier 208 as well, and, as just described, may be configured to receive the gesture image and output corresponding visual embeddings 218 based thereon.

[0053] Therefore, a comparator 220 may be used to compare the visual embeddings 218 with the text embeddings 212 to identify the gesture 112. For example, the comparator 220 may compare a visual embedding of the ‘paper’ gesture 112 with each of three text embeddings for rock, paper, and scissors, to determine a best match with ‘paper.’

[0054] Advantageously, as described in more detail, below, with respect to FIG. 4, the visual embeddings 218 may be constructed to be quickly and easily compared to the text embeddings 212. For example, both embeddings 218, 212 may be constructed within the same vector space. Then, a comparison may be made based on a Euclidean distance between the visual embedding and each potential text embedding, e.g., using a cosine similarity calculation.

[0055] In example implementations, and similar to the text embedder 210, the visual embedder 216 may be implemented using a transformer-based approach. For example, whereas a text transformer may break down sentences into words or other tokens, a visual transformer may split an image into individual segments or patches (e.g., an ‘x by x’ grid of patches of each image). Then, each patch may be formulated as a vector by, e.g., concatenating the channels of pixels therein for linear projection to a desired input dimension. Additional example details related to operations of the visual embedder 216 are provided below, e.g., with respect to FIG. 4.

[0056] FIG. 3 is a flowchart illustrating example operations of the system of FIG. 2. In the example of FIG. 3, operations are illustrated as separate, sequential operations. However, in various example implementations, the illustrated operations may be implemented in a different order than illustrated, in an overlapping or parallel manner, and/or in a nested, iterative, looped, or branched fashion. Further, various operations or suboperations may be included, omitted, or substituted.

[0057] In FIG. 3, an image of a gesture is received (302). For example, a camera or other image sensor may capture an image of a gesture, such as the image of the gesture 112 captured by the camera 114 in FIG. IB. The image handler 214 of the gesture classifier 208 may interface with the camera 114 to receive the captured gesture image.

[0058] A numerical representation of the image may be generated (304). For example, the visual embedder 216 may generate visual embeddings 218. For example, the visual embeddings 218 may include a vector of a defined size within a vector space, generated by a visual transformer of the visual embedder 216.

[0059] A comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture may be determined (306). For example, the comparator of FIG. 2 may compare the visual embeddings 218 with one or more of the text embeddings 212. As described with respect to FIG. 2, and in more detailed examples, below, with respect to FIG. 4, both the visual embeddings 218 and the text embeddings 212 may be constructed as being the same size within a single vector space, to facilitate ease of comparison.

[0060] The gesture may thus be identified as corresponding to the stored gesture, based on the comparison (308). For example, with reference to the description of FIG. IB, the comparator 220 may identify the image of the gesture 112 as corresponding to the defined gesture ‘paper’ within the stored gestures corresponding to the text embeddings 212, from among defined gestures of ‘rock, paper, scissors.’

[0061] In the more detailed example of FIG. 4, a prompt 402 of ‘rock, paper, scissors’, corresponding to the prompt 104 of FIG. 1A, is processed by a LLM 404 to obtain corresponding textual description 406 for ‘rock,’ textual description 408 for ‘paper,’ and textual description 410 for ‘scissors.’ As shown, textual description 406 includes “curling all fingers into the palm of the hand, with the thumb overlapping the index and middle fingers.” The textual description 408 includes “extending all fingers and the thumb, with the palm facing outward.” The textual description 410 includes “extending the index and middle fingers and crossing them over each other, with the ring and pinky fingers curled into the palm.”

[0062] A text encoder 412, as an example of the text embedder 210, may be configured to encode a text embedding 414 for the textual description 406, a text embedding 416 for the textual description 408, and a text embedding 418 for the textual description 410. In the example of FIG. 4, each text embedding 414, 416, 418 may be generated as a series or vector of 768 numbers or dimensions. Other vector sizes, such as a series of 1536 numbers, or any suitable size, may also be used.

[0063] After deployment, a gesture image 420 may be received. In the example of FIG. 4, similar to the image of the gesture 112 of FIG. IB, a gesture matching the ‘paper’ gesture is illustrated. A visual encoder 422 provides an example implementation of the visual embedder 216 of FIG. 2, and operates to generate a visual embedding 424. As illustrated, the visual embedding 424 may be implemented as a vector of 768 numbers, that is, may be the same size as each of the text embeddings 414, 416, 418.

[0064] Therefore, the comparator 220 of FIG. 2 may compute a similarity matrix 426 to compare the visual embedding 424 with each of the text embeddings 414, 416, 418. As shown in the example, the similarity matrix 426 results in a match of -0.3 for the text embedding 414 corresponding to ‘rock,’ a match of 0.9 for the text embedding 416 corresponding to ‘paper,’ and a match of 0.2 for the text embedding 418 corresponding to ‘scissors.’ As 0.9 is the highest degree of similarity, an identification of the gesture image 420 as corresponding to ‘paper.’

[0065] In FIG. 4, for a given gesture classifier being generated/used, suitable text embeddings 414, 416, 418 may be generated or encoded once, at a time of generation of the gesture classifier. Consequently, speed of generation may be less relevant as compared to a level of detail/accuracy of the text embeddings 414, 416, 418.

[0066] Advantageously, and as noted above with respect to FIG. 2, the visual encoder 422, or any visual embedding approach, need not be trained with the specific gestures of the prompt 402 for the system of FIG. 4 to function correctly. For example, the visual encoder 422 may be trained with a sample, generic training set of labelled gesture images that include a wide variety of potential gestures, which may or may not include the gestures of the prompt 402.

[0067] Moreover, even if the training data 215 does include a given gesture, the gesture may be associated with a different or additional label than desired with respect to the prompt 402. For example, the gesture for ‘scissors’ may be labelled in the training data 215 as ‘the number two.” Nonetheless, as the visual embedding 424 may be constructed based on hand landmarks as described with respect to FIG. 5, the similarity matrix 426 may still be used to correctly match the visual embedding 424 to the text embedding 416, as described above. In this sense, the gesture classifiers 108, 208 described herein may be referred to as zero-shot classifiers, as they are capable of classifying a gesture even if that specific gesture has not been processed before during either training time and/or during runtime.

[0068] Further, as noted above, the visual encoder 422 may be designed for speed of generation of the visual embedding 424, in order to provide a real-time or near real-time experience for users. For example, as also referenced above, the visual encoder 422 may be implemented using a visual transformer. In general, transformers may be trained using very large datasets, and some resulting full-sized models may use more time than desired/allowed to produce the visual embedding 424.

[0069] In FIGS. 2 and 4, the training dataset 215 for the visual embedder 216 or the visual encoder 422 may be maintained at a relatively small size, because a comprehensive set of training data for all potential gestures is not required. Such a reduced training set and volume of potential gestures to be processed may contribute to a smaller number of parameters, and increased speed, of the visual encoder 422.

[0070] In more specific examples, the visual encoder 422 may be provided as a distilled model, in which a first model is trained undergoes conventional pre-training and is otherwise trained and fine-tuned using a traditionally large dataset, and then the visual encoder 422 is generated as a reduced or distilled model that provides much of the benefit of the large-scale model, with many fewer parameters and less processing time/memory being required.

[0071] In the examples of FIGS. 1 A-4, gestures are largely described with respect to single or static hand gestures. In various example implementations, however, gestures may include a series or combinations of gestures, which may be captured as a video of a set of image frames. In other examples, other types of gestures besides hand gestures may be used. For example, gestures involving arm, leg, or head movements may be captured.

[0072] In addition, the approach of the gesture classifiers 108, 208 may be used in contexts other than gestures. For example, virtually any classification task that involves classifying an image may be performed when a textual description of images to be classified is available for textual embedding and when images to be classified may be characterized using generic landmarks or other characterizations that enable a visual description (as compared to a simple label). In some cases, even a pre-generated label may be sufficient if the resulting visual embedding is adequate to make the types of similarity comparisons described herein.

[0073] FIG. 5 illustrates example hand landmarks that may be used in the system of FIG. 4. As illustrated, each landmark may be provided a unique number, with a landmark 502 at a base of the palm being assigned 0, and a landmark 20 at the tip of the pinky being assigned 20, and intervening landmarks numbered sequentially along each finger (e.g., at each finger joint).

[0074] By constructing the training data 215 with landmark labels provided for each gesture, it is therefore possible to describe each labelled hand gesture in terms of relative finger positions, rather than (or in addition to) label names. For example, the training data 215 may include an instance of a hand making a fist, similar to the ‘rock’ gesture. For this example, the corresponding visual embedding would be based on a characterization of, e.g., the landmark 20 being close to the landmark 0, as the pinky is fully curled.

[0075] Similar comments would apply to a remainder of the fingers in FIG. 5, so that a resulting characterization of the gesture expressed in a visual embedding should be sufficiently similar to the ‘rock’ gesture, even if the example as stored in the training data is not labelled, or labeled ‘fist.’ Therefore, gesture classifications may meaningfully be made based on comparisons of characterizations using hand landmarks in the training data (expressed as visual embeddings) and using the text descriptions obtained from a LLM (expressed as text embeddings).

[0076] FIG. 6 is a flowchart illustrating a second example of more detailed operations of the system of FIGS. 2 and 4. In the example of FIG. 6, a text encoder may be trained using a generated dataset (600). For example, any text descriptions of associated gestures may be used, and any number of desired labelled gestures may be generated using a LLM. An extent of training may as large or as much as needed, resulting in as many model parameters as needed/practical, as such training may be performed a single time (or as-needed) when constructing gesture classifier generator(s) 106, 206, and because generation of text embeddings for a given gesture classifier may only occur once or as-needed.

[0077] A visual encoder may be trained using landmark-based training data (602). As described with respect to FIGS. 2, 4 and 5, training data 215 may be constructed in which a plurality of hand gestures are labelled using the hand landmarks of FIG. 5 to characterize relative finger positions, perhaps in conjunction with descriptions generated by a suitable LLM. Similar to the training a text encoder as just described, a vision encoder may be trained once prior to deployment of a corresponding gesture classifier. In contrast to the text encoder, however, it may be advantageous to maintain a reduced set of model parameters for the visual encoder (e.g., to use a distilled model), to enhance a speed of gesture recognition during runtime. Additionally, or alternatively, the visual encoder may be trained using a set of labelled gesture images, including some or all of a set of labelled gesture images used to train the text encoder.

[0078] Described approaches may enable use of a relatively smaller dataset than approaches that seek to train models using all or substantially all anticipated hand gestures that may be recognized. Moreover, the training data 215 may be further customized when information regarding an intended use of a gesture classifier to be generated is known. For example, the training data 215 may be customized for single hand gestures or gestures that use two hands together, or for static/dynamic hand gestures, or for other types of gestures.

[0079] A prompt may be received naming a desired one or more hand gestures, e.g., a set of hand gestures, which may be submitted to a LLM (604). The prompt may name each desired gesture by a single word or by a more detailed description/request.

[0080] Resulting text description(s) may be received from the LLM, and modified/iterated as needed (606). For example, as described above, an initial text description of a gesture may be inaccurate based on ambiguities existing between a single name and multiple hand gestures, such as ‘rock’ referring either to a stone or to ‘rock and roll’. Iterations with the LLM may enable a requesting user to provide additional details or otherwise disambiguate a request. Additionally, a user may request a longer or additional description if needed, as longer descriptions may enable greater odds of successful matches later during comparisons of corresponding text embeddings and visual embeddings.

[0081] Text embeddings of the text descriptions may be generated with a text encoder (608). For example, a transformer, such as used in a LLM context, may be trained to generate text embeddings. As already described, such a transformer may be a relatively large model in terms of training data and number of parameters used, as there are no significant time constraints on generation of the set of text embeddings.

[0082] A resulting gesture classifier may thus be deployed that includes the trained visual encoder and the generated text embeddings (610). That is, as described, the text embeddings may be deployed as static quantities or characterizations, while the visual encoder is deployed as a relatively small (e.g., distilled) model that is operational within the gesture classifier.

[0083] A gesture image may then be received (612). For example, an image of gesture may be obtained from a camera. [0084] Visual embeddings may be generated for the gesture, using the visual encoder (614). For example, as described with respect to FIG. 4, a 768 parameter vector may be generated.

[0085] The generated visual embeddings may then be compared to each of the stored text embeddings to determine/identify the received gesture (616). For example, as in FIG. 4, a similarity matrix may be used based on a comparison of the visual embedding with the stored text embedding(s) within a common vector space in which all of the embeddings are constructed. When a set of gestures is used, a highest similarity among the set may indicate an identified gesture. When only a single gesture is used, gesture identification may be based on a pre-defined similarity threshold.

[0086] Using described techniques, real-time or near real-time generation of the visual embeddings and comparison thereof to the stored text embeddings may be provided. For example, gesture recognition using described techniques may be performed at 15 frames per second (fps) or better.

[0087] Finally in FIG. 6, the recognized gesture may be mapped for use in a corresponding context (618). Examples are provided above in which gestures are use in the context of a game, but as also noted above, it will be appreciated that gesture recognition as described herein may be used in any suitable gesture context.

[0088] For example, gesture recognition may be useful in the context of remote meetings, in which users may interact with conferencing software and other users using defined gestures. Described techniques may also be used in the context of sign language interpretation and translation. Described techniques may also be used in controlling a VR or other extend reality setting, or in interacting with a smart home to control, e.g., lights and temperature.

[0089] Described techniques provide for creation of real-time hand gesture recognition models, while removing a need for training examples and fine-tuning. As a result, designers and prototypers may create tailored models for specific applications, without collecting training data or fine-tuning the model(s).

[0090] Example applications, as referenced above, include application in VR, AR, and other extended reality (XR) applications, which may be implemented using a variety of devices, including head-mounted devices (HMDs). For example, gesture recognition in the context of smartglasses may provide a convenient interaction methodology, without consuming space for user controls on the smartglasses, or using separate, connected devices (such as handheld controllers). [0091] However, current gesture detection techniques ask users to train a model on the users’ own gestures, which can be a time-consuming process. For example, Interactive Machine Learning (IML) toolkits instruct users to direct a recognition model via demonstration, which may be arduous and potentially induce overfitting issues.

[0092] Described techniques simplify the gesture recognition process for smartglasses and other devices. As described, designers and prototypers may use described techniques to create a customized, real-time model that recognizes preferred gestures, without using additional training data. Consequently, described techniques enable a wider use of smartglasses, while making smartglasses more useful for a wider range of applications.

[0093] FIGS. 7, 8A, and 8B provide additional example context in which described techniques may be implemented, including example smartglasses and other devices/applications.

[0094] FIG. 7 is a third person view of a user 702 (analogous to the user 102 of FIG. 1) in an ambient environment 7000, with one or more external computing systems shown as additional resources 752 that are accessible to the user 702 via a network 7200. FIG. 7 illustrates numerous different wearable devices that are operable by the user 702 on one or more body parts of the user 702, including a first wearable device 750 in the form of glasses worn on the head of the user, a second wearable device 754 in the form of ear buds worn in one or both ears of the user 702, a third wearable device 756 in the form of a watch worn on the wrist of the user, and a computing device 706 held by the user 702. In FIG. 7, the computing device 706 is illustrated as a handheld computing device but may also be understood to represent any personal computing device, such as a table or personal computer.

[0095] In some examples, the first wearable device 750 is in the form of a pair of smart glasses including, for example, a display, one or more images sensors that can capture images of the ambient environment, audio input/output devices, user input capability, computing/processing capability and the like. Additional examples of the first wearable device 750 are provided below, with respect to FIGS. 8A and 8B.

[0096] In some examples, the second wearable device 754 is in the form of an ear worn computing device such as headphones, or earbuds, that can include audio input/output capability, an image sensor that can capture images of the ambient environment 7000, computing/processing capability, user input capability and the like. In some examples, the third wearable device 756 is in the form of a smart watch or smart band that includes, for example, a display, an image sensor that can capture images of the ambient environment, audio input/output capability, computing/processing capability, user input capability and the like. In some examples, the handheld computing device 706 can include a display, one or more image sensors that can capture images of the ambient environment, audio input/output capability, computing/processing capability, user input capability, and the like, such as in a smartphone. In some examples, the example wearable devices 750, 754, 756 and the example handheld computing device 706 can communicate with each other and/or with external computing system(s) 752 to exchange information, to receive and transmit input and/or output, and the like. The principles to be described herein may be applied to other types of wearable devices not specifically shown in FIG. 7 or described herein.

[0097] The user 702 may choose to use any one or more of the devices 706, 750, 754, or 756, perhaps in conjunction with the external resources 752, to implement any of the implementations described above with respect to FIGS. 1-6. For example, the user 702 may use an application executing on the device 706 and/or the smartglasses 750 to execute the gesture classifier generator 106 or the gesture classifier 108 of FIG. 1A.

[0098] As referenced above, the device 706 may access the additional resources 752 to facilitate the various techniques described herein, or related techniques. In some examples, the additional resources 752 may be partially or completely available locally on the device 706. In some examples, some of the additional resources 752 may be available locally on the device 706, and some of the additional resources 752 may be available to the device 706 via the network 7200. As shown, the additional resources 752 may include, for example, server computer systems, processors, databases, memory storage, and the like. In some examples, the processor(s) may include training engine(s), transcription engine(s), translation engine(s), rendering engine(s), and other such processors. In some examples, the additional resources may include ML model(s), such as an Al model used by the text embedder 210 of FIG. 2.

[0099] The device 706 may operate under the control of a control system 760. The device 706 can communicate with one or more external devices, either directly (via wired and/or wireless communication), or via the network 7200. In some examples, the one or more external devices may include various ones of the illustrated wearable computing devices 750, 754, 756, another mobile computing device similar to the device 706, and the like. In some implementations, the device 706 includes a communication module 762 to facilitate external communication. In some implementations, the device 706 includes a sensing system 764 including various sensing system components. The sensing system components may include, for example, one or more image sensors 765, one or more position/orientation sensor(s) 764 (including for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer and other such sensors), one or more audio sensors 766 that can detect audio input, one or more image sensors 767 that can detect visual input, one or more touch input sensors 768 that can detect touch inputs, and other such sensors. The device 706 can include more, or fewer, sensing devices and/or combinations of sensing devices. Various ones of the communications modules may be used to control brightness settings among devices described herein, and various sensors may be used individually or together to perform the types of gaze, depth, and/or brightness detection described herein.

[00100] Captured still and/or moving images may be displayed by a display device of an output system 772, and/or transmitted externally via a communication module 762 and the network 7200, and/or stored in a memory 770 of the device 706. The device 706 may include one or more processor(s) 774. The processors 774 may include various modules or engines configured to perform various functions. In some examples, the processor(s) 774 may include, e.g, training engine(s), transcription engine(s), translation engine(s), rendering engine(s), and other such processors. The processor(s) 774 may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s) 774 can be semiconductor-based including semiconductor material that can perform digital logic. The memory 770 may include any type of storage device or non-transitory computer-readable storage medium that stores information in a format that can be read and/or executed by the processor(s) 774. The memory 770 may store applications and modules that, when executed by the processor(s) 774, perform certain operations. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory 770.

[00101] Although not shown separately in FIG. 7, it will be appreciated that the various resources of the computing device 706 may be implemented in whole or in part within one or more of various wearable devices, including the illustrated smartglasses 750, earbuds 754, and smartwatch 756, which may be in communication with one another to provide the various features and functions described herein.

[00102] An example head mounted wearable device 800 in the form of a pair of smart glasses is shown in FIGS. 8A and 8B, for purposes of discussion and illustration. The example head mounted wearable device 800 includes a frame 802 having rim portions 803 surrounding glass portion, or lenses 807, and arm portions 830 coupled to a respective rim portion 803. In some examples, the lenses 807 may be corrective/prescription lenses. In some examples, the lenses 807 may be glass portions that do not necessarily incorporate corrective/prescription parameters. A bridge portion 809 may connect the rim portions 803 of the frame 802. In the example shown in FIGS. 8A and 8B, the wearable device 800 is in the form of a pair of smart glasses, or augmented reality glasses, simply for purposes of discussion and illustration.

[00103] In some examples, the wearable device 800 includes a display device 804 that can output visual content, for example, at an output coupler providing a visual display area 805, so that the visual content is visible to the user. In the example shown in FIGS. 8A and 8B, the display device 804 is provided in one of the two arm portions 830, simply for purposes of discussion and illustration. Display devices 804 may be provided in each of the two arm portions 830 to provide for binocular output of content. In some examples, the display device 804 may be a see through near eye display. In some examples, the display device 804 may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world, for example, through the lenses 807, next to content (for example, digital images, user interface elements, virtual content, and the like) output by the display device 804. In some implementations, waveguide optics may be used to depict content on the display device 804.

[00104] The example wearable device 800, in the form of smart glasses as shown in FIGS. 8A and 8B, includes one or more of an audio output device 806 (such as, for example, one or more speakers), an illumination device 808, a sensing system 810, a control system 812, at least one processor 814, and an outward facing image sensor 816 (for example, a camera). In some examples, the sensing system 810 may include various sensing devices and the control system 812 may include various control system devices including, for example, the at least one processor 814 operably coupled to the components of the control system 812. In some examples, the control system 812 may include a communication module providing for communication and exchange of information between the wearable device 800 and other external devices. In some examples, the head mounted wearable device 800 includes a gaze tracking device 815 to detect and track eye gaze direction and movement. Data captured by the gaze tracking device 815 may be processed to detect and track gaze direction and movement as a user input. In the example shown in FIGS. 8A and 8B, the gaze tracking device 815 is provided in one of two arm portions 830, simply for purposes of discussion and illustration. In the example arrangement shown in FIGS. 8A and 8B, the gaze tracking device 815 is provided in the same arm portion 830 as the display device 804, so that user eye gaze can be tracked not only with respect to objects in the physical environment, but also with respect to the content output for display by the display device 804. In some examples, gaze tracking devices 815 may be provided in each of the two arm portions 830 to provide for gaze tracking of each of the two eyes of the user. In some examples, display devices 804 may be provided in each of the two arm portions 830 to provide for binocular display of visual content.

[00105] The wearable device 800 is illustrated as glasses, such as smartglasses, augmented reality (AR) glasses, or virtual reality (VR) glasses. More generally, the wearable device 800 may represent any head-mounted device (HMD), including, e.g., goggles, helmet, or headband. Even more generally, the wearable device 800 and the computing device 706 may represent any wearable device(s), handheld computing device(s), or combinations thereof.

[00106] Use of the wearable device 800, and similar wearable or handheld devices such as those shown in FIG. 7, enables useful and convenient use case scenarios of implementations of FIGS. 1-6. For example, the image sensor 816 may be used to capture a gesture for classification.

[00107] A first example implementation, referred to as Example 1, includes: receiving an image of a gesture; generating a numerical representation of the image; determining a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture; and identifying the gesture as corresponding to the stored gesture, based on the comparison.

[00108] Example 2 includes the method of Example 1, further comprising: determining the comparison includes computing a first similarity of the numerical representation of the image with respect to the numerical representation of the textual description and computing a second similarity of the numerical representation of the image with respect to a second numerical representation of a second textual description of a second stored gesture; and identifying the gesture as corresponding to the stored gesture, is based on determining that the first similarity is higher than the second similarity.

[00109] Example 3 includes the method of Examples 1 or 2, further comprising: generating, at a large language model, the textual description in response to a prompt.

[00110] Example 4 includes the method of any one of the preceding examples, wherein generating the numerical representation of the image includes generating a visual embedding using a visual encoder.

[00111] Example 5 includes the method of any one of the preceding examples, wherein the numerical representation of the textual description includes a textual embedding obtained from a text encoder.

[00112] Example 6 includes the method of any one of the preceding examples, wherein receiving the image of the gesture includes receiving the image of a hand gesture.

[00113] Example 7 includes the method of any one of the preceding examples, wherein the textual description includes a description of finger placement of a hand gesture.

[00114] Example 8 includes the method of any one of the preceding examples, comprising: determining the comparison including computing a similarity of the numerical representation of the image and the numerical representation of the textual description.

[00115] Example 9 includes the method of any one of the preceding examples, comprising generating a gesture classifier configured to identify the gesture as corresponding to the stored gesture, including: generating the numerical representation of the textual description for inclusion in the gesture classifier; including a visual encoder for generating the numerical representation of the image in the gesture classifier; and including a comparator in the gesture classifier for determining the comparison.

[00116] Example 10 includes the method of any one of the preceding examples, wherein: determining the comparison between the numerical representation of the image and a second numerical representation of a second textual description of a second stored gesture; and identifying the gesture as corresponding to the stored gesture or the second stored gesture, based on the comparison.

[00117] An eleventh example implementation, referred to as Example 11, includes a computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receive an image of a gesture; generate a numerical representation of the image; determine a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture; and identify the gesture as corresponding to the stored gesture, based on the comparison. [00118] Example 12 includes the computer program product of claim 11, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine the comparison between the numerical representation of the image and a second numerical representation of a second textual description of a second stored gesture and identifying the gesture as corresponding to the stored gesture or the second stored gesture, based on the comparison.

[00119] Example 13 includes the computer program product of example 11 or 12, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate, at a large language model, the textual description in response to a prompt. [00120] Example 14 includes the computer program product of any one of examples 11-13, wherein the numerical representation of the textual description includes a textual embedding obtained from a text encoder, and further wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate the numerical representation of the image including generating a visual embedding using a visual encoder.

[00121] Example 15 includes the computer program product of any one of examples 11-14, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate a gesture classifier configured to identify the gesture as corresponding to the stored gesture, including: generating the numerical representation of the textual description for inclusion in the gesture classifier; including a visual encoder for generating the numerical representation of the image in the gesture classifier; and including a comparator in the gesture classifier for determining the comparison. [00122] Example 16 includes the computer program product of any one of examples 11-15, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine the comparison including computing a similarity of the numerical representation of the image and the numerical representation of the textual description.

[00123] Example 17 includes the computer program product of any one of examples 11-16, wherein the numerical representation of the textual description of the stored gesture further includes a second numerical representation of a second textual description of a second stored gesture, and further wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determining the comparison includes computing a first similarity of the numerical representation of the image with respect to the numerical representation of the textual description and computing a second similarity of the numerical representation of the image with respect to a second numerical representation of a second textual description of a second stored gesture; and identifying the gesture as corresponding to the stored gesture, is based on determining that the first similarity is higher than the second similarity.

[00124] An eighteenth example implementation, referred to as Example 18, includes a head-mounted device (HMD) comprising: at least one frame for positioning the HMD on a face of a user; at least one image sensor; at least one processor; and at least one memory, the at least one memory storing a set of instructions, which, when executed, cause the at least one processor to: receive an image of a gesture from the image sensor; generate a numerical representation of the image; determine a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture; and identify the gesture as corresponding to the stored gesture, based on the comparison.

[00125] Example 19 includes the HMD of example 18, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the HMD to: determine the comparison includes computing a first similarity of the numerical representation of the image with respect to the numerical representation of the textual description and computing a second similarity of the numerical representation of the image with respect to a second numerical representation of a second textual description of a second stored gesture; and identify the gesture as corresponding to the stored gesture, is based on determining that the first similarity is higher than the second similarity.

[00126] Example 20 includes the HMD of example 18 or 19, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the HMD to: generate a gesture classifier configured to identify the gesture as corresponding to the stored gesture, including: generating the numerical representation of the textual description for inclusion in the gesture classifier; including a visual encoder for generating the numerical representation of the image in the gesture classifier; and including a comparator in the gesture classifier for determining the comparison.

[00127] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[00128] These computer programs (also known as modules, programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine- readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. [00129] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or LED (light emitting diode)) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[00130] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[00131] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[00132] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the description and claims.

[00133] In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[00134] Further to the descriptions above, a user is provided with controls allowing the user to make an election as to both if and when systems, programs, devices, networks, or features described herein may enable collection of user information (e.g., information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that user information is removed. For example, a user’s identity may be treated so that no user information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[00135] The computer system (e.g., computing device) may be configured to wirelessly communicate with a network server over a network via a communication link established with the network server using any known wireless communications technologies and protocols including radio frequency (RF), microwave frequency (MWF), and/or infrared frequency (IRF) wireless communications technologies and protocols adapted for communication over the network.

[00136] In accordance with aspects of the disclosure, implementations of various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[00137] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, may be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein. [00138] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and/or "including," when used in this specification, specify the presence of the stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof.

[00139] It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to, or "on," another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to, or "directly on," another element, there are no intervening elements present. As used herein the term "and/or" includes any and all combinations of one or more of the associated listed items.

[00140] Spatially relative terms, such as "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[00141] Example implementations of the concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized implementations (and intermediate structures) of example implementations. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and/or tolerances, are to be expected. Thus, example implementations of the described concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Accordingly, the regions illustrated in the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example implementations.

[00142] It will be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element could be termed a "second" element without departing from the teachings of the present implementations.

[00143] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and/or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[00144] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and/or sub-combinations of the functions, components, and/or features of the different implementations described.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving an image of a gesture; generating a numerical representation of the image; determining a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture; and identifying the gesture as corresponding to the stored gesture, based on the comparison.

2. The method of claim 1, comprising: determining the comparison includes computing a first similarity of the numerical representation of the image with respect to the numerical representation of the textual description and computing a second similarity of the numerical representation of the image with respect to a second numerical representation of a second textual description of a second stored gesture; and identifying the gesture as corresponding to the stored gesture, is based on determining that the first similarity is higher than the second similarity.

3. The method of claim 1 or 2, comprising: generating, at a large language model, the textual description in response to a prompt.

4. The method of any one of the preceding claims, wherein generating the numerical representation of the image includes generating a visual embedding using a visual encoder.

5. The method of any one of the preceding claims, wherein the numerical representation of the textual description includes a textual embedding obtained from a text encoder.

6. The method of any one of the preceding claims, wherein receiving the image of the gesture includes receiving the image of a hand gesture.

7. The method of any one of the preceding claims, wherein the textual description includes a description of finger placement of a hand gesture.

8. The method of any one of the preceding claims, comprising: determining the comparison including computing a similarity of the numerical representation of the image and the numerical representation of the textual description.

9. The method of any one of the preceding claims, comprising generating a gesture classifier configured to identify the gesture as corresponding to the stored gesture, including: generating the numerical representation of the textual description for inclusion in the gesture classifier; including a visual encoder for generating the numerical representation of the image in the gesture classifier; and including a comparator in the gesture classifier for determining the comparison.

10. The method of any one of the preceding claims, wherein: determining the comparison between the numerical representation of the image and a second numerical representation of a second textual description of a second stored gesture; and identifying the gesture as corresponding to the stored gesture or the second stored gesture, based on the comparison.

11. A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receive an image of a gesture; generate a numerical representation of the image; determine a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture; and identify the gesture as corresponding to the stored gesture, based on the comparison.

12. The computer program product of claim 11, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine the comparison between the numerical representation of the image and a second numerical representation of a second textual description of a second stored gesture and identifying the gesture as corresponding to the stored gesture or the second stored gesture, based on the comparison.

13. The computer program product of claim 11 or 12, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate, at a large language model, the textual description in response to a prompt.

14. The computer program product of any one of claims claim 11-13, wherein the numerical representation of the textual description includes a textual embedding obtained from a text encoder, and further wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate the numerical representation of the image including generating a visual embedding using a visual encoder.

15. The computer program product of any one of claims 11-14, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate a gesture classifier configured to identify the gesture as corresponding to the stored gesture, including: generating the numerical representation of the textual description for inclusion in the gesture classifier; including a visual encoder for generating the numerical representation of the image in the gesture classifier; and including a comparator in the gesture classifier for determining the comparison.

16. The computer program product of any one of claims 11-15, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine the comparison including computing a similarity of the numerical representation of the image and the numerical representation of the textual description.

17. The computer program product of any one of claims 11-16, wherein the numerical representation of the textual description of the stored gesture further includes a second numerical representation of a second textual description of a second stored gesture, and further wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determining the comparison includes computing a first similarity of the numerical representation of the image with respect to the numerical representation of the textual description and computing a second similarity of the numerical representation of the image with respect to a second numerical representation of a second textual description of a second stored gesture; and identifying the gesture as corresponding to the stored gesture, is based on determining that the first similarity is higher than the second similarity.

18. A head-mounted device (HMD) comprising: at least one frame for positioning the HMD on a face of a user; at least one image sensor; at least one processor; and at least one memory, the at least one memory storing a set of instructions, which, when executed, cause the at least one processor to: receive an image of a gesture from the image sensor; generate a numerical representation of the image; determine a comparison of the numerical representation of the image and a numerical representation of a textual description of a stored gesture; and identify the gesture as corresponding to the stored gesture, based on the comparison.

19. The HMD of claim 18, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the HMD to: determine the comparison includes computing a first similarity of the numerical representation of the image with respect to the numerical representation of the textual description and computing a second similarity of the numerical representation of the image with respect to a second numerical representation of a second textual description of a second stored gesture; and identify the gesture as corresponding to the stored gesture, is based on determining that the first similarity is higher than the second similarity.

20. The HMD of claim 18 or 19, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the HMD to: generate a gesture classifier configured to identify the gesture as corresponding to the stored gesture, including: generating the numerical representation of the textual description for inclusion in the gesture classifier; including a visual encoder for generating the numerical representation of the image in the gesture classifier; and including a comparator in the gesture classifier for determining the comparison.