EP4728487A1 - Contrastive feature masking vision model for open-vocabulary detection - Google Patents

Abstract

Machine Seaming models exhibiting improved, regional -level prediction of open- vocabulary image-text embeddings for input images are provided. These vision-language models may include vision transformers, These models can then be fine-tuned, m combination with object detection models that receive outputs therefrom, to detect and identify objects, including novel objects, in input images. Such an object detection and identification model can be further improved by combination with a '"frozen" version of such a vision-language model that has not been subject to fine-tuning. Such models can be trained by employing both (i) a contrastive loss that compares the image-level embedding vector determined, for an input image to an embedding vector determined from text that describes the image and (ii) a reconstructive loss that compares, in the embedding space, a representation of the full image with an embedding-space recon struction of the image form a masked subset of the image.

Description

CONTRASTIVE FEATURE MASKING VISION MODEL FOR OPEN- VOCABULARY DETECTION CROSS-REFERENCE TO RELATED APPLICATION [0001] This application claims priority to U.S. Provisional Application No.63/508,742, filed June 16, 2023, which is hereby incorporated by reference in its entirety. BACKGROUND [0002] Artificial neural networks, convolutional neural networks, transformers, deep learning models, and/or other machine learning models can be used to classify inputs, to filter or otherwise modify inputs, to project inputs into a semantically relevant or otherwise useful embedding space, to classify images, text, or other inputs, to segment images, to generate textual responses to input text, to assess sentiment in input text, or to provide other beneficial outputs from applied inputs. However, limitations in the amount and type of training data available to train such models can present difficulties with respect to the accuracy and types of outputs achievable by models trained using such limited training data, as well as difficulties regarding the computational cost of such training. SUMMARY [0003] In a first aspect, a method is provided that includes: (i) obtaining training data, wherein the training data comprises a plurality of images and a plurality of text descriptions, wherein each text description of the plurality of text descriptions describes a respective image of the plurality of images; (ii) using the training data, iteratively training (a) a first model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an image-text embedding space, and (b) a second model to receive input text and generate therefor an embedding vector that represents the input text in the image-text embedding space, wherein iteratively training the first and second models comprises: (iii) applying a first text description of the plurality of text descriptions to the second model to generate a first embedding vector in the image-text embedding space; (iv) segmenting, into a first set of image tiles, a first image of the plurality of images that is described by the first text description; (v) applying the first set of image tiles to the first model to generate a first set of embedding vectors in the image-text embedding space; (vi) based on the first set of embedding vectors, determining a first summary embedding vector for the first image; (vii) comparing the first summary embedding vector to the first embedding vector to generate a first contrastive loss; (viii) applying a first masked subset of the first set of image tiles to the first model to generate a second set of embedding vectors in the image-text embedding space; (ix) applying the second set of embedding vectors to a third model to generate a third set of embedding vectors in the image-text embedding space, wherein each image tile of the first set of image tiles is represented by a respective vector of the third set of embedding vectors; (x) comparing the first set of embedding vectors to the third set of embedding vectors to generate a first reconstructive loss; and (xi) updating (a) the second model based on the first contrastive loss, (b) the third model based on the first reconstructive loss, and (c) the first model based on the first contrastive loss and the first reconstructive loss. [0004] In a second aspect, a method is provided that includes: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space. [0005] In a third aspect, a method is provided that includes: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space; (iv) applying, to a detector model, the target set of embedding vectors to generate a predicted target location of an object in the target image; (v) based on the predicted target location, selecting a subset of the target set of embedding vectors that represent image tiles of the target set of image tiles that depict the object in the target image; and (vi) based on the selected subset of the target set of embedding vectors, determining a target summary embedding vector in the image-text embedding space for the object in the objective image. [0006] In a fourth aspect, a method is provided that includes: (i) obtaining training data, wherein the training data comprises a plurality of images; and (ii) using the training data, iteratively training a model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an embedding space, wherein iteratively training the model comprises: (a) segmenting, into a first set of image tiles, a first image of the plurality of images; (b) applying, to the model to generate a first set of embedding vectors in the embedding space, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles within the first image; (c) based on the first set of embedding vectors, determining a first loss; (d) segmenting, into a second set of image tiles, a second image of the plurality of images; (e) applying, to the model to generate a second set of embedding vectors in the embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles within the second image; (f) based on the second set of embedding vectors, determining a second loss; and (g) updating the model based on the first loss and the second loss. [0007] In a fifth aspect, a method is provided that includes: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to a trained model, representations of the target set of image tiles to generate a target set of embedding vectors in an embedding space, wherein the trained model is a model as trained in the fourth aspect. [0008] In another aspect, a non-transitory computer readable medium is provided having stored thereon program instructions executable by at least one processor to cause the at least one processor to perform the above methods. [0009] In another aspect a system is provided that includes: (i) at least one processor; and (ii) a non-transitory computer-readable medium, having stored therein instructions executable by the at least one processor to cause the system to perform the above methods. [0010] These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation. BRIEF DESCRIPTION OF THE FIGURES [0011] Figure 1 illustrates aspects of training a visual machine learning model, according to an example embodiment. [0012] Figure 2 depicts examples of outputs generated from the training of a visual machine learning model trained as depicted in Figure 1, according to an example embodiment. [0013] Figure 3 illustrates aspects of training a visual machine learning model, according to an example embodiment. [0014] Figure 4 depicts examples of inference outputs generated from a visual machine learning model trained as depicted in Figure 3, according to an example embodiment [0015] Figure 5 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments. [0016] Figure 6 is a simplified block diagram showing some of the components of an example computing system. [0017] Figure 7 is a flowchart of an example method. [0018] Figure 8 is a flowchart of an example method. [0019] Figure 9 is a flowchart of an example method. [0020] Figure 10 is a flowchart of an example method. [0021] Figure 11 is a flowchart of an example method. DETAILED DESCRIPTION [0022] Examples of methods and systems are described herein. It should be understood that the words "exemplary," “example,” and “illustrative,” are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "exemplary," “example,” or “illustrative,” is not necessarily to be construed as preferred or advantageous over other embodiments or features. Further, the exemplary embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations. I. Overview [0023] It is desirable for a variety of applications to obtain models that can identify the contents of images. This can be done, e.g., to perform semantic lookup of images related to a textual or image query, identify objects or people in images, cluster images, label images, perform similar image lookup, or other information related to the contents of images. It is desirable for the content-identifying output of such a model to be generic, rather than limited to an enumerated set of objects or classes set at training time for the model. For example, the output of the model could be embedding vector(s) that represent the content(s) of an input image in an image-text embedding space, e.g., in an open-vocabulary manner. This allows entirely novel text and/or images to be applied to the model, even text or image contents that were not represented at all in the training data used to train the model. This can be useful in, e.g., robotics applications wherein a robot interacting with the real world is likely to encounter previously unknown or unseen objects (e.g., a new device or tool, a prize statuette, an unconventional mug). It is also desirable that the output of such a model include region or location information about the contents of the image, e.g., to permit the identity of multiple different objects, at respective locations within an image, to be determined. [0024] However, the generation of such a model may be difficult. This difficulty may be related to the relative lack of training data that includes images labeled, in an open- vocabulary or otherwise extensible manner, at a variety of different locations within the images (e.g., to label multiple different objects within each training image). Instead, images with image-level labels (e.g., that describe the overall contents or theme of the image, rather than of specific objects/regions therein) are more readily available. Such training data has been leveraged in the past to generate models that can predict, at the whole-image level, the contents of input images, potentially in an open-vocabulary or other semantically open-ended manner. [0025] The models and methods of training thereof described herein are improved in that they result in models that are capable, in a semantically open-ended manner (e.g., in an open vocabulary manner), of predicting the contents of images at the sub-image level (e.g., the identity of multiple objects or other contents within an input image). The outputs of these models are embedding vectors in a semantically-encoded image-text embedding space, allowing the model to represent novel image contents and/or textual descriptions thereof that were not represented in the training data used to train the model. These benefits are provided in part by using model architectures (e.g., Vision Transformers) that represent input images as tiles (e.g., as tokens representing the projection of non-overlapping tiles of an input image into an input embedding space) and that generate respective output embedding vectors therefor in an image-text embedding space (i.e., a separate output embedding vector for each tile of the input image). These benefits are also provided in part by generating multiple output embedding vectors for the input image in the image-text embedding space, allowing the model to be trained using a contrastive loss (against the predicted output embedding vectors generated by an auxiliary text model from an input textual description of the image input to the vision model) and a reconstructive loss (against a reconstruction of the input image, in the image-text embedding space, from a degraded version of the input image that discards a fraction of the input image tiles). Accordingly, training data that only includes image-level textual descriptions can be leveraged to train a model that is able to represent the contents of input images at higher resolutions (e.g., at the level of non-overlapping tiles of an input image) in an open-vocabulary manner (e.g., by generating representative embedding vectors for each tile of an input image in an image-text embedding space). [0026] A model trained in this manner can then be fine-tuned, in combination with the training of a number of “detector head” output models, to predict the locations and identities (in the image-text embedding space) of discrete objects or other contents at different locations within an input image. The pre-training of the vision model on training data that includes only image-level open-vocabulary text descriptions allows the fine-tuned model and detector heads to be trained using relatively fewer instances of training data that includes sub-image-level textual descriptions or labels. Such a fine-tuned model and detector heads can be improved, with respect to embedding vectors generated to identify objects in an input image, by combining the embedding vector outputs of the fine-tuned vision model (or replacing those outputs entirely) with outputs from the pre-trained model prior to the fine-tuning (which may be referred to as a “frozen” model, as it has not been updated beyond the original training with the image-level textual description training data). Such a “frozen” model may retain more open-vocabulary knowledge relative to a fine-tuned model, as the fine-tuned model may be trained on sub-image-level training data that is more limited with respect to the types of objects represented and/or the textual descriptions thereof. [0027] The training data used to train a vision model as described herein (or pre-train such a model, when it is later subjected to fine-tuning) can broadly include a variety of types of images paired with textual descriptions thereof. The textual descriptions can be long-form (e.g., full sentences) or short (single words, few-word phrases, or unconnected sets of words each of which describes a different object in the image or aspect of the image). In some examples, the textual descriptions could be class labels from an enumerated set of classes; in such examples, the class labels can be converted into text by mapping the class labels to textual descriptions of each of the classes (e.g., to class titles in a metadata file describing the contents of a training dataset that includes the images and associated class labels). [0028] Such a dataset, including images and associated textual descriptions thereof, can then be used to train two models. The first model is a visual model (e.g., a Vision Transformer) as described herein, which receives tiles (e.g., non-overlapping tiles, e.g., as tokens projected into an input embedding space) of an input image and generates respective representative embedding vectors that represent the respective tiles of the input image in an image-text embedding space. The second model (e.g., a Transformer) receives textual descriptions of the training images (e.g., as sequences of tokens that represent words or other portions of the textual descriptions) and outputs an embedding vector that represents the textual description in the image-text embedding space. [0029] The first model does not operate on each tile separately; rather, the model operates on all of the tiles together, allowing contextual information from other tiles to influence the embedding vector generated for a particular tile (e.g., via an attention mechanism of a Transformer-type vision model). Accordingly, the first model could be a Vision Transformer (ViT), one or more convolutional neural networks (CNNs), or some other model architecture or combination of architectures (e.g., a CNN that receives input image tiles and outputs representative projection vectors, which are then input into a ViT). The tile- representing model inputs (e.g., projection vector tokens) could include positional information (e.g., position embeddings added to and/or concatenated onto a tile-content-representing embedding) to represent the location of each tile within the input image and/or relative to other tiles. [0030] To train the first and second models, individual training examples (which include an image and a paired text description describing the image) are applied to the first and second models to generate outputs that are then used to determine losses. The losses (potentially combined across many training inputs, e.g., via batch processing) are then used to update the models (e.g., via gradient descent). The text description is applied to the second model to generate an embedding vector therefor and the image is applied to the first model twice; once to generate a set of embedding vectors for all (or a majority) of the tiles of the input image based on all (or a majority) of the tiles as input and a second time only a fraction of the tiles of the input are applied to generate respective embedding vectors. The embedding vectors for the fraction of the tiles are then applied to a decoder to reconstruct the embedding vectors for the entire image. These reconstruction embedding vectors are compared to the embedding vectors generated based on all (or a majority) of the image tiles to generate a ‘reconstructive’ loss. The embedding vectors generated based on all (or a majority) of the image tiles are also summarized and compared to the embedding vector for the textual description to generate a ‘contrastive’ loss. The first model is trained based on both losses, while the decoder model is trained only using the reconstructive loss and the second model is trained only using the contrastive loss. [0031] Information represented by the contrastive loss allows the vision model to generate output embeddings in a combined image-text embedding space. Information represented by the reconstructive loss allows the vision model to learn how to recognize image contents at the region-level (e.g., at the level of individual tiles of the input images). This region-level learning can then be used for downstream region-level predictions (e.g., to identify the identity of objects or other contents of specific portions of a novel image, rather than the image as a whole), while the contrastive learning allows those predictions to represent the image contents in an open-ended semantic space that is also accessible to textual representations (e.g., using the trained second model). [0032] Figure 1 depicts aspects of such a process 100 for training a first model 110 to predict, for tiles of an input image, respective embedding vectors in an image-text embedding space and a second model 120 to predict, for tokens of input text, an embedding vector in the same image-text embedding space. An input image 101a is divided into a plurality of image tiles 105a (e.g., non-overlapping tiles of the input image). Positional embedding information 160 may be added to representations of the tiles (e.g., to projection vectors generated from the tiles) prior to applying them to the first model 110. A subset 105c of the tiles are applied to the first model 110 a first time to generate therefor respective embedding vectors 115b in the image-text embedding space. This set of embedding vectors 115b for the subset of the image tiles 105c is then applied to a decoder model 130 to generate a set of embedding vectors 135 in the image-text embedding space for all of the tiles of the input image 101a (or for all of the tiles of the input image that were not in the subset of the image tiles 105c). All of the image tiles 105b (or a majority of the image tiles, which are not part of the subset 105a) are also applied to the first model 110 a second time to generate therefor respective embedding vectors 115a in the image-text embedding space. The text description 101b of the input image 101a is applied to the second model 120 to generate therefor an embedding vector 117 in the image- text embedding space. [0033] The set of reconstructed embedding vectors 135 are compared to the set of non- reconstructed embedding vectors 115a to generate a reconstructive loss 150a. The set of non- reconstructed embedding vectors 115a are summarized 140 into a summary vector 145 and compared to the text description embedding vector 117 to generate a contrastive loss 150b. The losses 150a, 150b can then be used to update the various models; both losses 150a, 150b to update the first model 110, the contrastive loss 150b to update the second model 120, and the reconstructive loss 150a to update the decoder 130. [0034] The summary vector 145 that represents, in the image-text embedding space, the overall contents of the input image 101a can be determined 140 based on the set of non- reconstructed embedding vectors 115a in a variety of ways, e.g., an arithmetic, geometric, or other variety of average across the embedding vectors in the set 115a, an average angle and magnitude across the embedding vectors in the set 115a, an average of a most-populated cluster of the embedding vectors in the set 115a following a clustering operation performed on the embedding vectors in the set 115a, or some other method. For example, global average pooling could be applied across the set of non-reconstructed embedding vectors 115a to generate the summary vector 145. [0035] Once the summary vector 145 and the text description embedding vector 117 have been generated, they can be compared to generate the contrastive loss 150a. This can be done in a variety of ways, e.g., across a batch of applied training images and associated text descriptions. In some examples, the contrastive loss for a batch of inputs could be determined as an average of an image-to-text loss across the batch of training inputs and a text-to-image loss across the batch of training inputs. For example, an image-to-text loss could be determined for a batch of training examples by determining a sum, across each pair of text description embedding vectors and summary vectors in the batch of training examples, of a logarithm of the cosine similarity between a given summary vector and the corresponding text description embedding vector divided by the sum, across each text description embedding vector in the batch, of the cosine similarities between the given summary vector and each text description embedding vector in the batch (e.g., similarly or identically to the InfoNCE loss formulation). An example of such an image-to-text loss is represented in Equation (1) as: [0037] where L_I2T is the image-to-text loss, B is the number of training examples in the batch, vi is the summary embedding vector determined for the image of the i^th training example, li is the text description embedding vector for the text description of the i^th training example, and IJ is a scaling factor. The text-to-image loss for the batch could be determined symmetrically to the image-to-text loss (swapping the image embedding vectors v for the text embedding vectors l, and vice versa, in Equation (1)), and then averaged with the image-to-text loss to determine the overall reconstructive loss. [0038] Generating the reconstructive loss 150a could include, e.g., determining cosine similarities, L2 or L1 distances, or some other comparisons between corresponding pairs of embedding vectors of each set (i.e., that correspond to the same tile of the input image). The comparison could be performed only on a subset of the embedding vectors, e.g., based only on those embedding vectors that were not output from the first model 110 following application of the masked subset of the image tiles 105c (i.e., only those embedding vectors that were masked from the first model 110 and thus created entirely by the decoder 130 based on the set of embedding vectors 115b generated for the masked subset of the image tiles 105c). For example, the distances (e.g., cosine distances) between the reconstructed embedding vectors 135 for only the masked image tiles and the corresponding non-reconstructed embedding vectors 115a as in Equation (2): [0040] where L_rec is the reconstructive loss, B is the number of training examples in the batch, f_k is the non-reconstructed embedding vector 115a predicted for the k^th image tile, f_k is the reconstructed embedding vector 135 predicted by the decoder 130 for the k^th image tile, sg() denotes stop gradient, and M is the set of masked image tiles. [0041] Figure 2 depicts aspects of the image reconstruction for two example input images (top and bottom). The leftmost column is the original input images. The center column depicts a non-masked subset of the image tiles of the input images that are applied to the first model to generate output embeddings that are then applied, with positional information as to the locations of the non-masked image tiles, to a decoder to generate predicted embedding vectors for the masked image tiles. The right column depicts the ability of the decoder to predict, from the embedding vectors for only the non-masked tiles, embedding vectors for the masked tiles (with the color indicating the similarity between the reconstructed embeddings and the embedding output from the model when applying all of the image tiles of an input image). [0042] Note that some additional computational cost is added by applying the non- masked image tiles (or projection vectors thereof) to the first model 110 twice (once on their own, to generate a subset of the embedding vectors to be applied to the decoder 130 for reconstruction, and another time along with the rest of the image tiles to generate the ‘baseline’ image embedding outputs against which the reconstructed embeddings and text description embedding are compared to generate the loss functions). This computational cost can be reduced to by applying only the masked image tiles to the first model 110 to generate the set of non-reconstructed embedding vectors 115a (i.e., by omitting the masked image tiles 105c from the inputs applied to the first model 110 to generate the non-reconstructed embedding vectors 115a). As shown via experiment (below), this results in the same or similar accuracy while significantly reducing the computational cost. [0043] In examples wherein the first model 110 is a Transformer (e.g., a ViT) or some other tile-based vision model, the inputs applied to the model for each of the image tiles (e.g., projections of the image tiles into an input embedding space) could have positional information 160 added thereto. This could include summing the projection vectors for the image tiles with positional embedding vectors (e.g., fixed 2D sinusoidal positional embeddings), concatenating such vectors together, or combining them in some other manner. However, when a tile-based image model as described herein is trained using datasets that comprises low-resolution images paired with whole-image-level textual descriptions, the model can come to ‘over-rely’ on the positional information. This can result in difficulties when using the trained model for downstream tasks on higher resolution images for tasks that are sub-image-level (e.g., to identify and/or locate individual object or other contents as discrete location within an image). [0044] To compensate for this, positional embedding information could be added during training only to a fraction of the training examples, training batches, or in some other manner whereby the positional embedding information is ‘dropped out’ of some subset of the training runs. For example, positional embedding information could be added only to 50% (or some other fraction) of the training examples/training batches (with the remaining 50% having no positional embedding information added thereto). As shown below, this positional embedding dropout training method provides benefits with respect to the accuracy, number of training iterations, and amount of training data while also resulting in trained models are more capable of downstream fine-tuning on region-level tasks or other downstream tasks. [0045] Note that this positional embedding dropout training method is not limited only to those models and model training methods described herein. The positional embedding dropout method can be generically useful in any circumstance where a vision model is being trained to predict an output based on inputs that represent respective different locations within an input image (e.g., respective non-overlapping image tiles) and that may or more not have added thereto positional information related to the absolute or relative locations of the different inputs within an overall input image (e.g., any variety of Vision Transformer). [0046] The models and training methods described herein provide improved accuracy, region-level prediction, and open-vocabulary or similarly semantically extended predictions while using fewer training examples that are less complex and less difficult to acquire (e.g., images that are labeled with image-level, open-vocabulary descriptions, rather than images labeled with the locations of specific objects/contents within the images, whose labels may be limited to an enumerated set of classes rather than open-vocabulary). These models and training methods also allow image-text training data from multiple different sources, e.g., different sets of training images labeled with respective different enumerated sets of classes, since the ‘incompatible’ class labels can be easily converted into open-vocabulary text descriptions (e.g., by replacing a label with the text name or title for the corresponding class). [0047] Models as described herein (e.g., trained as described herein) can be used for a variety of downstream tasks, since they represent a large amount of open-vocabulary knowledge about the content of images and can predict such content, in an image-text embedding space, in a regionally-specific manner (e.g., an embedding vector for each non- overlapping tile of an input image). Such a model could be used to classify a whole image, but to do so in a manner that permits the generation of multiple different classifications for the image (e.g., by clustering or otherwise determining multiple modes in the distribution of the set of embedding vectors output by the model for an input image) and/or that is extensible to novel sets of labels/classes, as it provides such classification outputs in an image-text embedding space (e.g., novel classes/labels, or descriptions thereof, could be projected into the image-text embedding space using a textual model as described herein, e.g., the second model 120). Such models can be used for improved image-text retrieval (e.g., in a ‘zero-shot’ context, without further fine-tuning), improved image transfer detection (e.g., in a ‘zero-shot’ context, without further fine-tuning), image segmentation in an open-vocabulary image-text space (e.g., by performing clustering on the set of output embedding vectors to determine a set of segments within the image), image segmentation using novel object classes (e.g., by projecting textual descriptions into the image-text space and then determining distances from the embeddings of the textual descriptions and the embeddings output by the model for the various tiles or other regions of the image), or some other applications. [0048] In some examples, a model as trained herein could be used to detect and identify objects or other discrete contents within an image. This could include training one or more detector models to predict, based on the set of embedding vectors output from such a model, the location(s) of objects in an input image. The identity of the identified objects could then be determined based on the embedding vector(s) output from the model that correspond to the determined location(s) of the object(s). Such a training process could also include fine-tuning the vision model to adapt to the object-locating application. The extensive, open-vocabulary image knowledge represented by the per-trained vision model allows relatively fewer training examples to be used to train the detector model(s) and/or to fine-tune the underlying vision model. This leads to a reduced computational cost to train these models, as well as befits with respect to reducing the number of training examples, as such examples (being annotated multiple times, and with the specific locations of objects) are more difficult to obtain than images that are labeled as whole images. Additionally, the models trained in this manner retain some of the open-vocabulary knowledge from the pre-training, allowing such models to be used to predict the location and identity of objects that were not part of the set of objects represented in the relatively limited training data used to train such models. [0049] Figure 3 depicts aspects of such a process 200 for training a vision model 210a and detector model(s) 220 to predict, in combination, the location and identity of object in an input image 201. This training is based on a set of training data that includes a set of images and, for each image, annotation data indicating the location(s) and identity(s) of one or more objects in the image. In this second set of training data, the identities of the objects are limited to an enumerated list of object identities, and text descriptions for each of object identity in the enumerated list are available (e.g., “joystick,” “TV,” “couch,” “recliner,” “cabinet or shelf,” hardback book,” “electric/gas stove”). To train the detector model(s) 220 and to further train (or “fine-tune”) the vision model 210a trained as described above, an example input image 201 is separated into image tiles 205 which are then applied as inputs to the vision model 210a (e.g., via projection into an input image space, and potentially with positional embedding information added thereto). The vision model 210a then generates a set of embedding vectors 215a representing the image tiles 205 in the image-text embedding space. The detector model(s) 220 could have a Mask R-CNN architecture or some other architecture (e.g., an alternative two- stage detector head architecture). [0050] This set of embedding vectors 215a is then applied to one or more detector models 220 to generate respective predicted location(s) 225a of objects in the image 201. Such predicted locations 225a could represent bounding boxes or other bounding shapes, singular points, points and size information (e.g., the location and diameter of a circle) or some other information about the location and/or extent of the object(s) in the image. The predicted locations 225a are then used 230 to generate, for each object, respective summary embedding vectors 235a in the image-text embedding space based on respective subsets of one or more of the set of embedding vectors 215a that correspond to the respective locations 225a. [0051] To determine the identity of each of the objects, a set of text descriptions for the enumerated set of object classes is projected into the image-text embedding space (e.g., using the trained second model 120) to generate respective object identity embedding vectors 241. Each of the summary embedding vectors 235a is compared 240a to each of the object identity embedding vectors 241 to determine an identity for each of the objects. This could include determining cosine similarities, L2 distances, or some other similarity metric and then, for each object, selecting the object identity whose corresponding object identity embedding vector 241 has the highest similarity metric. [0052] To train the detector model(s) 220 and to fine-tune the fine-tuned vision model 210a, loss values can be determined for each of the objects. This could include determining a location loss based on a difference between the predicted location of an object as output by a detector model 220 and the ‘true’ location of a corresponding object in the input image 201 as indicated by annotation information associated therewith. The detector model 220 (and optionally the fine-tuned vision model 210a) could then be updated based on the location loss. An identification loss could be determined based on a difference between the predicted identities of each identified object and the ‘true’ identities of corresponding objects in the input image 201 as indicated by annotation information associated therewith. The fine-tuned vision model 210a could then be updated based on the identification loss. [0053] The predicted locations 225a can be used 230 to generate, for each object, respective summary embedding vectors 235a in a variety of ways. This can include selecting, based on a predicted location 225a a subset of one or more of the set of embedding vectors 215a that overlap with, contain, or otherwise correspond to the predicted location 225a (e.g., using the RoI-align function). Where the location 225a is a single point, then the subset could include only the single embedding vector 215a whose image tile overlaps with the single point, or a set of embedding vectors 215a whose image tiles are within a specified distance of the single point. Where the location 225a defines an area (e.g., a bounding box), the subset could include only embedding vectors 215a whose image tiles are entirely contained within the bounding box, only embedding vectors 215a whose image tiles at least partially overlap with the bounding box, or only embedding vectors 215a whose image tiles are within a specified distance of the bounding box. The selected subset of embedding vectors 215a could be used to generate a single summary embedding vector 235a in a variety of ways, e.g., via averaging, global average pooling, or some other method. For example, the summary embedding vector 235a could be a weighted combination of the subset of embedding vectors 235a in the selected subset weighted based on, e.g., distance of their respective image tiles from the location 225a, degree of overlap of their respective image tiles with an area defined by the location 225a, or some other weighting based on the location 225a. Where the subset of embedding vectors 235a includes only a single embedding vector, the summary embedding vector 235a could be set to the single embedding vector. [0054] The above steps can be used at training time, with training images input, to train the models 210a, 220 by comparing their outputs to known ‘true’ annotations of the training images. Inference proceeds similarly, with the outputs accepted as predictions rather than being compared to any known ‘true’ outputs. This can include identifying the locations and summary embedding vectors (in the image-text embedding space) of objects contained within an input image. Additionally, the identity of such objects could be determined, from an enumerated set of possible objects, by projecting text descriptions of those objects into the image-text space (e.g., by applying the text descriptions to the second model 120) to generate a set of object identity embedding vectors 241 and then comparing the predicted object embedding vectors to the set of object identity embedding vectors 241 to determine which of the enumerated set of objects is likely represented at the predicted location in the input image. Such an enumerated set of objects could include the enumerated set of objects represented in the training dataset as well as a set of novel objects. As such, the set of object identity embedding vectors 241 used at inference time would be a union of the set of object identity embedding vectors 241 used during training time (which represent the ‘baseline’ set of objects) and a set of additional object identity embedding vectors 241 representing the ‘novel’ set of objects not seen during training. [0055] The learning rate used to update the detector models 220 and the fine-tuned vision model 210a could be the same. However, in some examples it could be beneficial to set a faster learning rate for the detector models than for the fine-tuned vision model 210a. For example, it can be beneficial to set the learning rate for the fine-tuned vision model 210a to approximately half (e.g., between 0.4 and 0.6, or between 0.45 and 0.55) the learning rate for the detector models 220. This can allow the detector heads, which may not have had the benefit of any pre-training, to learn more quickly while preserving the knowledge already present in the pre-trained fine-tuned vision model 210a, avoiding ‘forgetting’ during the fine-tuning phase. These benefits are experimentally assessed below. [0056] A ‘frozen’ version of the vision model, without additional object detection- based fine-tuning, could additionally or alternatively be used to generate the summary embedding vectors for objects whose locations have been detected using the detector model(s) 220 and the fine-tuned vision model 210a. This could be done to leverage the unmodified open- vocabulary vision-language knowledge present in the pre-trained vision model, which has been trained on the full scope of text examples present in the initial training dataset, before some of that knowledge is potentially lost due to fine-tuning of the model using the relatively less broad scope of objects represented in the objects-annotated-with-locations training dataset used to fine-tune the vision model and to train the detector models. [0057] The use of such a frozen vision model 210b for inference is also depicted in Figure 3. [0058] The image tiles 205 are then applied as inputs to the frozen vision model 210b (e.g., via projection into an input image space, and potentially with positional embedding information added thereto). The frozen vision model 210b then generates a set of embedding vectors 215b representing the image tiles 205 in the image-text embedding space. The predicted locations 225a (generated by the fine-tuned vision model 210a and detector model(s) 220) are then used 230 to generate, for each object, respective summary embedding vectors 235b in the image-text embedding space based on respective subsets of one or more of the set of embedding vectors 215b that correspond to the respective locations 225b. To determine the identity of each of the objects, a set of text descriptions for a enumerated set of novel object classes is projected into the image-text embedding space (e.g., using the trained second model 120) to generate respective object identity embedding vectors 241 that are added to the set of object identity embedding vectors 241 determined for the objects that were represented in the training dataset. Each of the summary embedding vectors 235b may compared 240c to each of the object identity embedding vectors 241 to determine an identity for each of the objects. Additionally or alternatively, the summary embedding vectors 235b output from the frozen vision model 210b could be combined (e.g., by a linear combination or some other weighted or unweighted combination) with the summary embedding vectors 235a output from the fine-tuned vision model 210a and compared 240b to each of the object identity embedding vectors 241 to determine an identity for each of the objects. Such comparisons could include determining cosine similarities, L2 distances, or some other similarity metric and then, for each object, selecting the object identity whose corresponding object identity embedding vector 241 has the highest similarity metric. [0059] Whichever method is used to determine the overall summary embedding vector for an object (just the embeddings output from the frozen vision model, just the embeddings output from the fine-tuned vision model, or a combination), a detection score could be determined for the identified object to determine a quality or confidence of the identification. This could be used to assess the quality of the identification (e.g., to compare against alternative object identification methods) and/or to reject the identification of a particular object in the enumerated set of objects for a “background,” “null,” or “reject” class (e.g., to reject the identified object as “not an object” or just as background content of the input image). An ‘ensemble’ detection score could be determined, for cases where the output is compared to a set of possible object classes that includes both the base object classes of the fine-tuning and a set of novel object classes, as in Equation (3): [0061] is the ensemble detection score for the i^th detected putative object in an input image, pi is a raw similarity output (e.g., a cosine similarity followed by a softmax) for the comparison between the summary embedding output for the object from the fine-tuned model and the identity embedding for the selected object class, zi is a raw similarity output (e.g., a cosine similarity followed by a softmax) for the comparison between the summary embedding output for the object from the frozen model and the identity embedding for the selected object class, C_B is the base set of possible objects from the training dataset, C_N is the set of novel objects, and Į and ȕ א [0,1] control the weights for determining the detection score for when the predicted object is in the base or novel object sets, respectively. [0062] Figure 4 depicts a number of example images applied to models trained as described herein in order to predict the locations and identities of objects depicted in the images. The detection scores for each object are also depicted as percentile scores in Figure 4. II. Example Machine Learning Models and Training Thereof [0063] A machine learning model as described herein may include, but is not limited to: an artificial neural network (e.g., Transformers, layered models wherein each layer includes two or more sub-layers one or more of which could include artificial neural networks, convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and/or a heuristic machine learning system), a support vector machine, a regression tree, an ensemble of regression trees (also referred to as a regression forest), a decision tree, an ensemble of decision trees (also referred to as a decision forest), or some other machine learning model architecture or combination of architectures. [0064] An artificial neural network (ANN) could be configured in a variety of ways. For example, the ANN could include two or more layers, could include units having linear, logarithmic, or otherwise-specified output functions, could include fully or otherwise- connected neurons, could include recurrent and/or feed-forward connections between neurons in different layers, could include filters or other elements to process input information and/or information passing between layers, or could be configured in some other way to facilitate the processing of input sequences, sets of embedding vectors representing input sequences, downstream vectors and/or set of vector determined by the operation of one or more layers or sublayers of a multi-layer model, and/or individual vectors (e.g., embedding vectors representing tokens of an input sequence and/or embeddings of tiles of an input image which may or may not include positional information encoding the location of such tokens relative to each other within a length of text and/or image, downstream vectors representing the processing of such embedding vectors by one or more layers or sublayers of a multi-layer model). [0065] An ANN could include one or more filters that could be applied to the input and the outputs of such filters could then be applied to the inputs of one or more neurons of the ANN. For example, such an ANN could be or could include a convolutional neural network (CNN). Convolutional neural networks are a variety of ANNs that are configured to facilitate ANN-based classification or other processing based on images or other large-dimensional inputs whose elements are organized within two or more dimensions. The organization of the ANN along these dimensions may be related to some structure in the input structure (e.g., as relative location within the one-dimensional space of sequence of tokens can be related to similarity or relevance between tokens of the sequence). [0066] In example embodiments, a CNN includes at least one two-dimensional (or higher-dimensional) filter that is applied to an input; the filtered input is then applied to neurons of the CNN (e.g., of a convolutional layer of the CNN). The convolution of such a filter and an input could represent the color values of a pixel or a group of pixels from the input, in embodiments where the input is an image. A set of neurons of a CNN could receive respective inputs that are determined by applying the same filter to an input. Additionally or alternatively, a set of neurons of a CNN could be associated with respective different filters and could receive respective inputs that are determined by applying the respective filter to the input. Such filters could be trained during training of the CNN or could be pre-specified. For example, such filters could represent wavelet filters, center-surround filters, biologically-inspired filter kernels (e.g., from studies of animal visual processing receptive fields), or some other pre-specified filter patterns. [0067] A CNN or other variety of ANN could include multiple convolutional layers (e.g., corresponding to respective different filters and/or features), pooling layers, rectification layers, fully connected layers, or other types of layers. Convolutional layers of a CNN represent convolution of an input image, or of some other input (e.g., of a filtered, downsampled, or otherwise-processed version of an input image), with a filter. Pooling layers of a CNN apply non-linear downsampling to higher layers of the CNN, e.g., by applying a maximum, average, L2-norm, or other pooling function to a subset of neurons, outputs, or other features of the higher layer(s) of the CNN. Rectification layers of a CNN apply a rectifying nonlinear function (e.g., a non-saturating activation function, a sigmoid function) to outputs of a higher layer. Fully connected layers of a CNN receive inputs from many or all of the neurons in one or more higher layers of the CNN. The outputs of neurons of one or more fully connected layers (e.g., a final layer of an ANN or CNN) could be used to determine information about areas of an input image (e.g., for each of the pixels of an input image) or for the image as a whole. [0068] Neurons in a CNN can be organized according to corresponding dimensions of the input. For example, where the input is a sequence of token (a one-dimensional input, with each token representing one or more words, or fractions of words, in an input text string), neurons of the CNN (e.g., of an input layer of the CNN, of a pooling layer of the CNN) could correspond to locations in the one-dimensional input string/sequence. Connections between neurons and/or filters in different layers of the CNN could be related to such locations. [0069] FIG.5 shows diagram 500 illustrating a training phase 502 and an inference phase 504 of trained machine learning model(s) 532, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms, on an input set of training data to recognize patterns in the training data and provide output inferences and/or predictions about (patterns in the) training data. Such output could take the form of filtered or otherwise modified versions of the input, e.g., an input sequence that represents text in a source language could be modified by the machine learning model into (i) an output sequence that represents text in a target language that has similar meaning or semantic content as the input sequence and/or (ii) an output set of embedding vectors that represent, in a semantic embedding space, the meaning or semantic content of the input sequence. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG.5 shows training phase 502 where one or more machine learning algorithms 520 are being trained on training data 510 to become trained machine learning model 532. Then, during inference phase 504, trained machine learning model 532 can receive input data 530 and one or more inference/prediction requests 540 (perhaps as part of input data 530) and responsively provide as an output one or more inferences and/or predictions 550. [0070] As such, trained machine learning model(s) 532 can include one or more models of one or more machine learning algorithms 520. Machine learning algorithm(s) 520 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and/or a heuristic machine learning system), a Transformer, a support vector machine, a regression tree, an ensemble of regression trees (also referred to as a regression forest), a decision tree, an ensemble of decision trees (also referred to as a decision forest), or some other machine learning model architecture or combination of architectures. For example, the trained machine learning model(s) 532 could include a plurality of artificial neural networks and other elements related to such networks (e.g., mixing or weighting matrices, attention heads or other attentional mechanisms, sums, products, feedforward connections) arranged according to the multi-layer and sublayer architecture of a Transformer or similar model architecture designed to process input sequences. Machine learning algorithm(s) 520 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning. [0071] In some examples, machine learning algorithm(s) 520 and/or trained machine learning model(s) 532 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and/or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 520 and/or trained machine learning model(s) 532. In some examples, trained machine learning model(s) 532 can be trained, reside and execute to provide inferences on a particular computing device, and/or otherwise can make inferences for the particular computing device. [0072] During training phase 502, machine learning algorithm(s) 520 can be trained by providing at least training data 510 as training input using unsupervised, supervised, semi- supervised, and/or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of training data 510 to machine learning algorithm(s) 520 and machine learning algorithm(s) 520 determining one or more output inferences based on the provided portion (or all) of training data 510. Supervised learning involves providing a portion of training data 510 to machine learning algorithm(s) 520, with machine learning algorithm(s) 520 determining one or more output inferences based on the provided portion of training data 510, and the output inference(s) are either accepted or corrected based on correct results associated with training data 510. In some examples, supervised learning of machine learning algorithm(s) 520 can be governed by a set of rules and/or a set of labels for the training input, and the set of rules and/or set of labels may be used to correct inferences of machine learning algorithm(s) 520. [0073] Semi-supervised learning involves having correct results for part, but not all, of training data 510. During semi-supervised learning, supervised learning is used for a portion of training data 510 having correct results, and unsupervised learning is used for a portion of training data 510 not having correct results. Reinforcement learning involves machine learning algorithm(s) 520 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 520 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 520 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 520 and/or trained machine learning model(s) 532 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning. [0074] In some examples, machine learning algorithm(s) 520 and/or trained machine learning model(s) 532 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 532 being pre-trained on one set of data and additionally trained using training data 510. More particularly, machine learning algorithm(s) 520 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 504. Then, during training phase 502, the pre-trained machine learning model can be additionally trained using training data 510, where training data 510 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 520 and/or the pre- trained machine learning model using training data 510 of CD1’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 520 and/or the pre-trained machine learning model has been trained on at least training data 510, training phase 502 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 532. [0075] In particular, once training phase 502 has been completed, trained machine learning model(s) 532 can be provided to a computing device, if not already on the computing device. Inference phase 504 can begin after trained machine learning model(s) 532 are provided to computing device CD1. [0076] During inference phase 504, trained machine learning model(s) 532 can receive input data 530 and generate and output one or more corresponding inferences and/or predictions 550 about input data 530. As such, input data 530 can be used as an input to trained machine learning model(s) 532 for providing corresponding inference(s) and/or prediction(s) 550 to kernel components and non-kernel components. For example, trained machine learning model(s) 532 can generate inference(s) and/or prediction(s) 550 in response to one or more inference/prediction requests 540. In some examples, trained machine learning model(s) 532 can be executed by a portion of other software. For example, trained machine learning model(s) 532 can be executed by an inference or prediction daemon to be readily available to provide inferences and/or predictions upon request. Input data 530 can include data from computing device CD1 executing trained machine learning model(s) 532 and/or input data from one or more computing devices other than CD1. [0077] Input data 530 can include a collection of text strings provided by one or more sources. The collection of text strings can include natural language, artificially generated language, text from books, texts from online forums or chats, texts from emails, text applied to describe associated image(s), and/or other text. Other types of input data are possible as well. [0078] Inference(s) and/or prediction(s) 550 can include output text strings, output token sequences, output sets of embedding vectors, numerical values, and/or other output data produced by trained machine learning model(s) 532 operating on input data 530 (and training data 510). In some examples, trained machine learning model(s) 532 can use output inference(s) and/or prediction(s) 550 as input feedback 560. Trained machine learning model(s) 532 can also rely on past inferences as inputs for generating new inferences. III. Example Systems [0079] Figure 6 illustrates an example computing device 600 that may be used to implement the methods described herein. By way of example and without limitation, computing device 600 may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, or handheld computer, a server), elements of a cloud computing system, a robot, a drone, an autonomous vehicle, or some other type of device. It should be understood that computing device 600 may represent a physical computing device such as a server, a particular physical hardware platform on which a machine learning application operates in software, or other combinations of hardware and software that are configured to carry out machine learning functions as described herein. [0080] As shown in Figure 6, computing device 600 may include a communication interface 602, a user interface 604, a controller 606 (which may include one or more processors), and data storage 608, all of which may be communicatively linked together by a system bus, network, or other connection mechanism 610. [0081] Communication interface 602 may function to allow computing device 600 to communicate, using analog or digital modulation of electric, magnetic, electromagnetic, optical, or other signals, with other devices, access networks, and/or transport networks. Thus, communication interface 602 may facilitate circuit-switched and/or packet-switched communication, such as plain old telephone service (POTS) communication and/or Internet protocol (IP) or other packetized communication. For instance, communication interface 602 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 602 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. Communication interface 602 may also take the form of or include a wireless interface, such as a Wifi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 602. Furthermore, communication interface 602 may comprise multiple physical communication interfaces (e.g., a Wifi interface, a BLUETOOTH® interface, and a wide-area wireless interface). [0082] In some embodiments, communication interface 602 may function to allow computing device 600 to communicate, with other devices, remote servers, access networks, and/or transport networks. For example, the communication interface 602 may function to access one or more machine learning models and/or input therefor via communication with a remote server or other remote device or system in order to allow the computing device 600 to use the machine learning model(s) to generate outputs (e.g., sequences of model outputs generated as new input data is obtained) based on input data. For example, the computing system 600 could be an inference server and the remote system could be a smartphone that generated an image (e.g., an image from a camera) to be applied to a machine learning model in order to determine information about the contents of the image, e.g., the location(s) of objects within the image and their identity (e.g., as embedding vector(s) in an image-text embedding space and/or as identities of the objects determined by comparing such embedding vectors with sets of embedding vectors in the image-text space that correspond to descriptions of known objects). [0083] User interface 604 may function to allow computing device 600 to interact with a user, for example to receive input from and/or to provide output to the user. Thus, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface 604 may also include one or more output components such as a display screen which, for example, may be combined with a presence-sensitive panel. The display screen may be based on CRT, LCD, and/or LED technologies, or other technologies now known or later developed. User interface 604 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and/or other similar devices. [0084] Controller 606 may comprise one or more general purpose processors – e.g., microprocessors – and/or one or more special purpose processors – e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, tensor processing units (TPUs), or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing (e.g., application of CNN kernels or other filters to images via, e.g., convolution), machine learning model training, execution, and/or inference, among other applications or functions. Data storage 608 may include one or more volatile and/or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with controller 606. For example, a portion of the data storage 608 may be implemented as cache or other on-chip memory of a graphics processing unit or tensor processing unit integrated circuit and/or as RAM or some other variety of storage that is collocated with a GPU or TPU, e.g., on a graphics card, tensor acceleration card, or other semi-discrete subsystem of the overall system 600. Such storage could be used to store parameters that define a machine learning model (e.g., weights or other parameters of units of a multi-layer neural network or other multi-unit machine learning model). Data storage 608 may include removable and/or non-removable components. [0085] Controller 606 may be capable of executing program instructions 618 (e.g., compiled or non-compiled program logic and/or machine code) stored in data storage 608 to carry out the various functions described herein. Therefore, data storage 608 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing device 600, cause computing device 600 to carry out any of the methods, processes, or functions disclosed in this specification and/or the accompanying drawings. The execution of program instructions 618 by controller 606 may result in controller 606 using data 612. [0086] By way of example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device driver(s), and/or other modules) and one or more application programs 620 (e.g., functions for executing trained machine learning models and/or training such models) installed on computing device 600. Data 612 may include stored training data 614 (e.g., images and text associated with the images and that describes the images as a whole and/or specific objects or other contents represented by specified portions of the images) that could be used to train (e.g., to pre-train, to fine-tune) one or more machine learning models 616. [0087] Application programs 620 may communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs 620 reading and/or writing a trained machine learning model 616, transmitting or receiving information via communication interface 602, receiving and/or displaying information on user interface 604, and so on. [0088] Application programs 620 may take the form of “apps” that could be downloadable to computing device 600 through one or more online application stores or application markets (via, e.g., the communication interface 602). However, application programs can also be installed on computing device 600 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) of the computing device 600. IV. Example Methods [0089] Figure 7 is a flowchart of a method 700 as described herein. The method 700 includes obtaining training data, wherein the training data comprises a plurality of images and a plurality of text descriptions, wherein each text description of the plurality of text descriptions describes a respective image of the plurality of images (710). The method 700 additionally includes, using the training data, iteratively training (i) a first model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an image-text embedding space, and (ii) a second model to receive input text and generate therefor an embedding vector that represents the input text in the image-text embedding space (720). Iteratively training the first and second models includes: applying a first text description of the plurality of text descriptions to the second model to generate a first embedding vector in the image-text embedding space (721); segmenting, into a first set of image tiles, a first image of the plurality of images that is described by the first text description (723); applying the first set of image tiles to the first model to generate a first set of embedding vectors in the image-text embedding space (725); based on the first set of embedding vectors, determining a first summary embedding vector for the first image (727); comparing the first summary embedding vector to the first embedding vector to generate a first contrastive loss (729); applying a first masked subset of the first set of image tiles to the first model to generate a second set of embedding vectors in the image-text embedding space (731); applying the second set of embedding vectors to a third model to generate a third set of embedding vectors in the image-text embedding space, wherein each image tile of the first set of image tiles is represented by a respective vector of the third set of embedding vectors (733); comparing the first set of embedding vectors to the third set of embedding vectors to generate a first reconstructive loss (735); and updating (i) the second model based on the first contrastive loss, (ii) the third model based on the first reconstructive loss, and (iii) the first model based on the first contrastive loss and the first reconstructive loss (737). The method 700 could include additional or alternative steps or features. [0090] Figure 8 is a flowchart of a method 800 as described herein. The method 800 includes obtaining a target image (810). The method 800 additionally includes segmenting, into a target set of image tiles, the target image (820). The method 800 additionally includes applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space (830). The method 800 could include additional or alternative steps or features. [0091] Figure 9 is a flowchart of a method 900 as described herein. The method 900 includes obtaining a target image (910). The method 900 additionally includes segmenting, into a target set of image tiles, the target image (920). The method 900 additionally includes applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space (930). The method 900 additionally includes applying, to a detector model, the target set of embedding vectors to generate a predicted target location of an object in the target image (940). The method 900 additionally includes, based on the predicted target location, selecting a subset of the target set of embedding vectors that represent image tiles of the target set of image tiles that depict the object in the target image (950). The method 900 additionally includes, based on the selected subset of the target set of embedding vectors, determining a target summary embedding vector in the image-text embedding space for the object in the objective image (960). The method 900 could include additional or alternative steps or features. [0092] Figure 10 is a flowchart of a method 1000 as described herein. The method 1000 includes obtaining training data, wherein the training data comprises a plurality of images (1010). The method 1000 also includes, using the training data, iteratively training a model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an embedding space (1020). Iteratively training the model includes: segmenting, into a first set of image tiles, a first image of the plurality of images (1021); applying, to the model to generate a first set of embedding vectors in the embedding space, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles within the first image (1023); based on the first set of embedding vectors, determining a first loss (1025); segmenting, into a second set of image tiles, a second image of the plurality of images (1027); applying, to the model to generate a second set of embedding vectors in the embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles within the second image (1029); based on the second set of embedding vectors, determining a second loss (1031); and updating the model based on the first loss and the second loss (1033). The method 1000 could include additional or alternative steps or features. [0093] Figure 11 is a flowchart of a method 1100 as described herein. The method 1100 includes obtaining a target image (1110). The method 1100 additionally includes segmenting, into a target set of image tiles, the target image (1120). The method 1100 additionally includes applying, to a trained model, representations of the target set of image tiles to generate a target set of embedding vectors in an embedding space, wherein the trained model is the model as trained in method 1000. The method 1000 could include additional or alternative steps or features. V. Experimental Results [0094] The embodiments described herein were implemented and experimentally validated. The particulars and results of this experimental validation are provided in this section. Note that the particulars described in this section are intended only as illustrative, non- limiting examples of the embodiments described herein. [0095] A vision-language model as described herein was pre-trained to generically provide region-level image-text embedding vectors for regions across an input image. This pre- trained model was then used (via fine-tuning and the training of adjunct detection model(s)) to determine the location and identity (represented by embedding vectors in the image-text embedding space) of objects in input images. [0096] Pretraining setup. For the image-text pretraining, the ViT-B/16 and ViT-L/16 models were used as the image encoder (to generate, from image tiles of an input image, corresponding representative embedding vectors in the image-text embedding space), with an input image size of 224. Fixed 2D sinusoidal positional embeddings were used to represent image tile positional information within the representations thereof input into the vision models. During training, Positional Embedding Dropout (PED) with a drop probability of 0.5 was used. PED refers to the addition of positional embedding information to the inputs presented to the model in only a fraction of training steps; in the remainder of training steps, such positional information is not added. The whole-image embedding vector was obtained by global average pooling at the last ViT layer. The text encoder, to which was applied the descriptive text for each training image, was a 12-layer Transformer, with the input sequences truncated to a fixed length of 64 tokens. The L2-normalized image and text embeddings and a learnable scaling temperature were the input to the InfoNCE contrastive loss. [0097] The feature reconstruction decoder (which received the output from the vision model when provided as input a masked subset of image tiles of a training image) was a 2- layer ViT. This is unlike the 8-layer counterpart model used for the training of masked autoencoder (MAE) models, which are designed for raw pixel reconstruction. The reconstruction loss was cosine distance, scaled by a loss coefficient 2.0, and was added to the contrastive loss. The same image-text training dataset as Jia et al. was used. Unless otherwise noted, a batch size of 4k was used for ablation and 16k for comparisons, and training proceeded for 500k iterations using the AdamW optimizer with an initial learning rate (LR) of 5e-4 and linear LR decay.10k warm-up iterations were used with a weight decay of 0.01. [0098] Detection fine-tuning setup. The fine-tuned model and set of detection models were trained on images annotated with the locations and identity of objects from a set of base categories CB, with training image sizes of 1024x1024. The positional embeddings (PE) were bilinearly interpolated to fit the higher resolution. PED was not applied during the detection training, and a lower learning rate was set for the backbone vision model (e.g., 0.5x) compared to the rest of the model (e.g., the detection heads). [0099] CLIP templates were used and the average text embeddings of each category were determined to represent each category. Training was performed with a batch size of 128, the SGD optimizer with momentum 0.9, an initial learning rate of 0.18/0.02, and 36.8k/11.3k iterations on LVIS/COCO datasets. [00100] Results [00101] LVIS benchmark. The embodiments described herein were compared with other methods on the LVIS open-vocabulary detection benchmark which contains a diverse set of 1203 object categories. The base categories CB for training were the ‘frequent’ and ‘common’ categories, and novel categories CN were the ‘rare’ categories which are held out for testing. The main metric was mask APr, and the mean over three runs was reported for reproducibility. [00102] Table 1: LVIS open-vocabulary object detection. The contrastive feature masking vision transformer models (CFM-ViT) described herein outperformed the best existing approach by +7.6 APr, and the other ViT-based approach outperformed by +8.3 APr using the same backbone model.

[00103] Table 1 reports that the best CFM-ViT model achieved 33.9 APr, a significant improvement over the best existing ViT-based method OWL-ViT by +8.3 APr. CFM-ViT using a smaller ViT-B/16 backbone outperformed OWL-ViT with ViT-L/14 by +3.2 APr. Furthermore, compared to the previous best approach ViLD-Ens with EffNet-B7 backbone, CFM-ViT achieved a +7.6 APr improvement. Notably, CFM-ViT has a simple fine-tuning recipe using only vanilla detection losses, without the use of long-tail recognition losses, knowledge distillation, weak supervision, or pseudo box/mask labels, all of which are common among previous open-vocabulary detection methods. [00104] COCO benchmark. The embodiments described herein were compared to other methods using the COCO open-vocabulary detection benchmark. This setup uses 48 base categories for training and 17 novel categories for testing. The main metric is AP50 of novel categories (‘novel AP’). Due to fewer training categories, the CFM-ViT model exhibited a tendency to overfit to these categories using only the vanilla detection losses. This is because CFM-ViT does not use any auxiliary objectives such as pseudo box/mask labels, knowledge distillation, or weak supervision to counter-balance overfitting on this benchmark. However, Table 2 shows that CFM-ViT is still very competitive among previous methods that leverage auxiliary objectives. Moreover, CFM-ViT represents the first ViT-based method on this benchmark, as the other ViT-based approaches have only provided benchmarks on LVIS. [00105] Table 2: COCO open-vocabulary object detection (box AP50). CFM-ViT represents the first ViT-based approach and demonstrates a very competitive novel AP without using pseudo labeling or weak supervision. y: RegionCLIP uses an off-the-shelf RPN during its pretraining. z: Rasheed et al. uses an external MViT detector during pretraining. *: The other ViT-based method reported their results on LVIS only. [00106] Zero-shot Image-Text Retrieval. In addition to experimental evaluations of region-level open-vocabulary detection, the image-level representation embodiments described herein were also evaluated with respect to zero-shot image-text retrieval. The same CFM-ViT model as in the last row of Table 1 (ViT-L, batch size 16k) was used and pre-training continued on higher resolution, e.g., 448, for an additional 40K iterations. [00107] Table 3 shows a comparison of the embodiments as described herein with other dual-encoder methods on Flickr30K and MS COCO benchmarks. The CFM-ViT models as described herein outperformed state-of-the-art methods of similar or larger model size, on 8 out of 12 metrics. [00108] Table 3: Zero-shot image-text retrieval results on Flickr30K and COCO benchmarks. The pretrained model described herein was compared to other methods. The models described herein outperformed the state-of-the-art CoCa-Large with the same backbone in 8 out of 12 metrics. [00109] Zero-shot Transfer Detection. To assess CFM-ViT’s ability to generalize in zero-shot transfer detection, its performance on the Objects365-v1 validation split was evaluated. The same detector models trained on the LVIS base categories (as reported in Table 1) was used, replacing the LVIS embeddings with Objects365 vocabulary embeddings for transfer detection without fine-tuning. All categories were assumed to be novel and set Į,ȕ=(0.0, 0.65) in Eq. (3) above. The best model as described herein achieved 18.7 AP, outperforming ViLD by +6.9 AP and DetPro by +5.6 AP, as shown in Table 4. Given the different backbone capacity (R50 vs ViT), this comparison mainly serves to demonstrate that the CFM-ViT embodiments described herein can achieve strong cross-dataset generalization. [00110] Table 4: Transfer detection on Objects365 (Box APs). All models were trained on the LVIS base categories and tested on the Objects365 dataset, without fine-tuning. [00111] Ablation Study [00112] The design of the CFM-ViT’s pre-training and open-vocabulary detector was subjected to a number of ablation studies. Evaluate was performed on the LVIS open vocabulary detection benchmark. The image encoder was ViT-L/16, and a contrastive batch size is 4k was used by default. [00113] Table 5: Ablation studies on LVIS open-vocabulary detection benchmark. Training was performed on base (‘frequent’ + ‘common’) categories, testing on novel (‘rare’) categories, and the APr was reported. The ViT-L/16 backbone was used with a contrastive batch size of 4k unless otherwise noted. [00114] (a) Masked reconstruction. ‘baseline’ is the contrastive image-text pretraining. The masked feature reconstruction described herein provides a +3.3 APr improvement. Reconstruction in the raw pixel space or the first-layer feature space showed no benefit. [00115] (b) Positional Embedding Dropout (PED) improved the baseline by 1.1 APr. It achieved a further gain of +2.7 when used with masked feature reconstruction. PED outperformed ‘no positional embedding’ by 3.5 / 1.6 APr with/without feature reconstruction. [00116] (c) Masking contrastive branch (i.e., only presenting those image tiles to the non-masked vision model to that were not presented, as part of the masked subset of image tiles, to the masked vision model in order to perform image embedding reconstruction) recovered the training computational efficiency with little or no performance drop, outperforming the baseline by +3.0 APr.

[00117] (d) Backbone fine-tuning lr ratio (bblr) with respect to added detector layers. [00118] (e) Frozen backbone inference. When using standard positional embeddings, the ‘frozen backbone’ alone underperformed the fine-tuned encoder. In contrast, with the encoder pre-trained with PED, the frozen backbone inference surpassed the finetuned counterpart by +2.0 and +1.3 APr. [00119] (f) Scalabiltiy: The benefit of going from the ‘baseline’ methods to the CFM- ViT embodiments described herein across different model and contrastive batch sizes. CFM- ViT improved the baselines by +2.4 to +5.1 APr.

[00120] Masked feature reconstruction. Table 5a ablates the masked image-text pre- training embodiments described herein. Masked feature reconstruction offers a clear benefit of +3.3 APr over the contrastive image-text pre-training baseline. In this case, the feature reconstruction target was the output features of the image encoder. This was compared with other reconstruction targets: normalized image pixels and the features from the first patchifying layer. Neither provided improvements over the baseline. In contrast, the masked feature reconstruction embodiments described herein clearly improve upon the strong baseline and shows advantages in open-vocabulary detection. [00121] Positional embedding dropout. In Table 5b, the results of ablating the positional embedding dropout (‘PED’) are shown. PED brought a gain of +1.1 APr over the baseline (PE without dropout). This shows that PED effectively reduces overfitting to the low-resolution whole-image PE during pre-training, thus adapting better to the high-resolution detection task through fine- tuning. In addition, PED achieved a further gain of +2.7 when used together with masked feature reconstruction. PED was compared with another baseline which used no positional embeddings at all in the input to the ViT encoder (‘no PE’). The PED method outperformed the ‘no PE’ baseline by 3.5 / 1.6 APr with/without feature reconstruction. Note that the positional embeddings in the reconstruction decoder were always retained. Finally, PED allows the use of the frozen backbone as a strong region classifier as shown in Table 5e. [00122] Faster training by masking contrastive branch. Table 5c studies image masking ratios of the contrastive (which receives all or most of the image tiles of the input image and whose output is used to determine the contrastive loss and may be used to determine the reconstructive loss) and reconstruction (which receives as input only the minority of the image tiles that are part of the masked subset and whose output is only used to determine the reconstructive loss) encoders. By default, the contrastive encoder was applied to intact images during training, i.e.100% of the tokens representing image tiles of the input image. Adding the reconstruction tower with 25% input tokens results in 1.23x more training cost. To maintain the overall training cost and efficiency, feeding only 75% tokens to the contrastive encoder that are mutually exclusive with the token input to the reconstruction branch was evaluated. This masking technique fully recovers the training efficiency with little or no accuracy loss, outperforming the baseline by +3.0 APr. [00123] Backbone learning rate ratio. CFM-ViT leverages the retention of pre-trained knowledge in the backbone model to recognize novel categories. Table 5d reports the advantage of setting the backbone model learning rate lower than the rest of the detector model(s) during the fine-tuning, with a ratio of 0.5x exhibiting improved performance. Higher ratios lead to increased forgetting, while lower ratios limit the ability to adapt to the detection task. [00124] Frozen backbone inference. The ablation studies described above did not involve “frozen” backbone inference, that is, the use of the initial pre-trained model without fine-tuning (the model being “frozen”) in addition to the fine-tuned version of the model to provide enhanced embedding vectors for identified objects/regions within an image. All ablations used the fine-tuned ViT backbone to compute the VLM scores (pi in and Eq. (3) above). In Table 5e, the efficacy of the “frozen” backbone as a region classifier was evaluated by substituting the fine-tuned ViT encoder with a “frozen” ViT encoder (which has not been subject to any ‘fine-tuning’ following the initial pre-training). These experiments show that the frozen backbone underperformed the fine-tuned encoder when using standard positional embeddings, which applied to both the baseline with and without feature reconstruction loss. However, pre-training the ViT encoder with positional embedding dropout (PED) lead to significantly improved performance when used in combination with the “frozen” backbone, surpassing the fine-tuned backbone by +2.0/+1.3 APr, without/with feature reconstruction loss. This result demonstrates the efficacy of PED in reducing the domain gap between contrastive pre-training and detection fine-tuning, thus improving zero-shot region classification. Combined with feature reconstruction, the full method described herein achieved an overall improvement of +5.1 APr over the baseline. [00125] Model size and batch size. Table 5f reports the effect of model size and batch size in CFM-ViT pretraining on the downstream open-vocabulary detection. We Increasing the batch size from 4k to 16k lead to an improvement of +2.7 / 1.4 APr for both ViT-B/L, while upgrading from ViT-B to ViT-L resulted in an improvement of +5.9 / 5.6 APr for both batch sizes. These results align with observations from the contrastive learning literature that larger batch sizes and model sizes are both highly beneficial. CFM-ViT consistently outperformed the baseline by +2.4 to +5.1 APr, across all batch and model sizes tested, further demonstrating its efficacy. [00126] Image-text retrieval. In addition to ablations on open vocabulary detection, the effects of positional embedding dropout and masked feature reconstruction on zero-shot image- level retrieval was investigated, with the results reported in terms of Recall@1 metrics on Flickr30K and MS COCO datasets. Table 6 shows that positional embedding dropout effectively preserves the quality of image-level representation, while masked feature reconstruction yields an average improvement of 1% Recall@1 across all metrics. [00127] Table 6: Pretraining evaluation on zero-shot image-text retrieval (Recall@1). The image-level representation of the pre-trained model described herein on Flickr30k and COCO retrieval tasks was evalutated. The positional embedding dropout (PED) ablated and masked feature reconstruction added. ViT-L/16 and batch size 4k were used. [00128] Visualizations [00129] Feature reconstruction. Fig. 2 depicts feature reconstruction results from the pre-trained model. For visualization, we compute the similarity map (right-most column) between the reconstructed image features and the image-text embedding of a textual description of the corresponding input image. As shown, the learned feature reconstructions are semantically plausible with respect to the queried image-text pairs. [00130] Open-vocabulary detection outputs. Fig. 4 shows the outputs of a CFM-ViT model as described herein on LVIS novel categories (top row) and zero-shot transfer detection on Objects365 (bottom row). For both visualizations, the same model as in the last row of Table 1 was used, which was trained on the LVIS base categories. On both datasets, CFM-ViT was able to detect many novel categories unavailable during training. VI. Conclusion [00131] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless the context indicates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein. [00132] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and as discussed herein, each step, block and/or communication may represent a processing of information and/or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as steps, blocks, transmissions, communications, requests, responses, and/or messages may be executed out of order from that shown or discussed, including in substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer steps, blocks and/or functions may be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts may be combined with one another, in part or in whole. [00133] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a step or block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data may be stored on any type of computer-readable medium, such as a storage device, including a disk drive, a hard drive, or other storage media. [00134] The computer-readable medium may also include non-transitory computer- readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and/or random access memory (RAM). The computer- readable media may also include non-transitory computer-readable media that stores program code and/or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, and/or compact-disc read only memory (CD-ROM), for example. The computer-readable media may also be any other volatile or non-volatile storage systems. A computer-readable medium may be considered a computer- readable storage medium, for example, or a tangible storage device. [00135] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions may be between software modules and/or hardware modules in different physical devices. [00136] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims. VII. Enumerated Example Embodiments [00137] Embodiments of the present disclosure may thus relate to one of the enumerated example embodiments (EEEs) listed below. It will be appreciated that features indicated with respect to one EEE can be combined with other EEEs. [00138] EEE 1 is a method comprising: (i) obtaining training data, wherein the training data comprises a plurality of images and a plurality of text descriptions, wherein each text description of the plurality of text descriptions describes a respective image of the plurality of images; (ii) using the training data, iteratively training (a) a first model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an image-text embedding space, and (b) a second model to receive input text and generate therefor an embedding vector that represents the input text in the image-text embedding space, wherein iteratively training the first and second models comprises: (1) applying a first text description of the plurality of text descriptions to the second model to generate a first embedding vector in the image-text embedding space; (2) segmenting, into a first set of image tiles, a first image of the plurality of images that is described by the first text description; (3) applying the first set of image tiles to the first model to generate a first set of embedding vectors in the image-text embedding space; (4) based on the first set of embedding vectors, determining a first summary embedding vector for the first image; (5) comparing the first summary embedding vector to the first embedding vector to generate a first contrastive loss; (6) applying a first masked subset of the first set of image tiles to the first model to generate a second set of embedding vectors in the image-text embedding space; (7) applying the second set of embedding vectors to a third model to generate a third set of embedding vectors in the image-text embedding space, wherein each image tile of the first set of image tiles is represented by a respective vector of the third set of embedding vectors; (8) comparing the first set of embedding vectors to the third set of embedding vectors to generate a first reconstructive loss; and (9) updating (A) the second model based on the first contrastive loss, (B) the third model based on the first reconstructive loss, and (C) the first model based on the first contrastive loss and the first reconstructive loss. [00139] EEE 2 is the method of EEE 1, wherein the first model comprises a vision transformer. [00140] EEE 3 is the method of EEE 2, wherein applying the first set of image tiles to the first model to generate the first set of embedding vectors comprises applying, to the first model, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles, wherein applying the first masked subset of image tiles to the first model to generate the second set of embedding vectors comprises applying, to the first model, representations of the first masked subset of image tiles that include positional embedding information about the location of respective image tiles of the first masked subset of image tiles, and wherein iteratively training the first and second models further comprises: (i) applying a second text description of the plurality of text descriptions to the second model to generate a second embedding vector in the image-text embedding space; (ii) segmenting, into a second set of image tiles, a second image of the plurality of images that is described by the second text description; (iii) applying, to the first model to generate a fourth set of embedding vectors in the image-text embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles; (iv) based on the fourth set of embedding vectors, determining a second summary embedding vector for the second image; (v) comparing the second summary embedding vector to the second embedding vector to generate a second contrastive loss; (vi) applying, to the first model to generate a fifth set of embedding vectors in the image-text embedding space, representations of the second masked subset of image tiles that do not include positional embedding information about the location of respective image tiles of the second masked subset of image tiles; (vii) applying the fifth set of embedding vectors to the third model to generate a sixth set of embedding vectors in the image-text embedding space, wherein each image tile of the second set of image tiles is represented by a respective vector of the sixth set of embedding vectors; (viii) comparing the fourth set of embedding vectors to the sixth set of embedding vectors to generate a second reconstructive loss; and (ix) further updating (a) the second model based on the second contrastive loss, (b) the third model based on the second reconstructive loss, and (c) the first model based on the second contrastive loss and the second reconstructive loss. [00141] EEE 4 is the method of any preceding EEE, wherein generating the first contrastive loss comprises generating the first contrastive loss across a batch of first embedding vectors and first summary vectors by: (i) determining a text-to-image loss by determining a sum, across each pair of first embedding vector and first summary vector in the batch, of a logarithm of the cosine similarity between a given first embedding vector and the corresponding first summary vector divided by the sum, across each first summary vector in the batch, of the cosine similarities between the given first embedding vector and each first summary vector in the batch; (ii) determining an image-to-text loss by determining a sum, across each pair of first embedding vector and first summary vector in the batch, of a logarithm of the cosine similarity between a given first summary vector and the corresponding first embedding vector divided by the sum, across each first embedding vector in the batch, of the cosine similarities between the given first summary vector and each first embedding vector in the batch; and (iii) determining the first contrastive loss as an average of the text-to-image loss and the image-to-text loss. [00142] EEE 5 is the method of any preceding EEE, wherein applying the first set of image tiles to the first model to generate the first set of embedding vectors comprises applying, to the first model, only those image tiles of the first set of image tiles that are not part of the first masked subset. [00143] EEE 6 is the method of any preceding EEE, wherein determining the first summary embedding vector for the first image comprises performing global average pooling across the first set of embedding vectors. [00144] EEE 7 is the method of any preceding EEE, further comprising: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to the first model following training of the first model, representations of the target set of image tiles to generate a target set of embedding vectors in the image-text embedding space. [00145] EEE 5 is the method of EEE 7, further comprising: (i) determining, based on the target set of embedding vectors, a target summary embedding vector for the target image; (ii) receiving a query text; (iii) applying, to the second model following training of the second model, the query text to generate a query embedding vector in the image-text embedding space; (iv) determining a degree of similarity between the target summary embedding vector and the query embedding vector; and (v) responsive to at least one of (a) determining that the degree of similarity is less than a threshold degree of similarity or (b) determining that the degree of similarity is greater than at least one alternative degree of similarity between the query embedding vector and at least one target summary embedding vector for at least one alternative target image. [00146] EEE 9 is the method of any preceding EEE, further comprising generating (i) a fine-tuned model by further training of the first model and (ii) a detector model, by: (a) obtaining additional training data that comprises a set of images, each image annotated with the location and identity of at least one object within the image, wherein the identities of the objects represented in the set of images of the additional training data are limited to an enumerated list of object identities, and wherein the additional training data additionally comprises, for each object identity in the enumerated list of object identities, a text description for the object identity; (b) applying, to the second model following training of the second model, the text descriptions for the object identities to generate respective object identity embedding vectors in the image-text embedding space; and (c) using the additional training data, iteratively training (1) the fine-tuned model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in the image- text embedding space, and (2) a detector head to receive sets of embedding vectors output from the fine-tuned model and generate therefrom a predicted location of an object in the input image and a predicted embedding vector, in the image-text embedding space, that represents the identity of the object in the input image, wherein iteratively training the fine-tuned model and the detector head comprises: (A) segmenting, into a third set of image tiles, a third image of the additional training data, wherein the third image is annotated with a first location and a first identity of a first object represented in the third image; (B) applying the third set of image tiles to the fine-tuned model to generate a seventh set of embedding vectors in the image-text embedding space; (C) applying the seventh set of embedding vectors to the detector model to generate a first predicted location of the first object; (D) based on the first predicted location, selecting a subset of the seventh set of embedding vectors that represent image tiles of the third set of image tiles that depict the first object; (E) based on the selected subset of the seventh set of embedding vectors, determining a predicted summary embedding vector in the image-text embedding space for the first object; (F) comparing the first predicted location to an annotated location of the first object within the third image to generate a location loss; (G) comparing the first summary embedding vector to the object identity embedding vector that corresponds to the identity of the first object to generate an identification loss; and (H) updating the fine-tuned model and the detector model based on the location loss and the identification loss. [00147] EEE 10 is the method of EEE 9, wherein the first model comprises a vision transformer, wherein applying the first set of image tiles to the first model to generate the first set of embedding vectors comprises applying, to the first model, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles, wherein applying the first masked subset of image tiles to the first model to generate the second set of embedding vectors comprises applying, to the first model, representations of the first masked subset of image tiles that include positional embedding information about the location of respective image tiles of the first masked subset of image tiles, and wherein iteratively training the first and second models further comprises: (i) applying a second text description of the plurality of text descriptions to the second model to generate a second embedding vector in the image-text embedding space; (ii) segmenting, into a second set of image tiles, a second image of the plurality of images that is described by the second text description; (iii) applying, to the first model to generate a fourth set of embedding vectors in the image-text embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles; (iv) based on the fourth set of embedding vectors, determining a second summary embedding vector for the second image; (v) comparing the second summary embedding vector to the second embedding vector to generate a second contrastive loss; (vi) applying, to the first model to generate a fifth set of embedding vectors in the image-text embedding space, representations of the second masked subset of image tiles that do not include positional embedding information about the location of respective image tiles of the second masked subset of image tiles; (vii) applying the fifth set of embedding vectors to the third model to generate a sixth set of embedding vectors in the image-text embedding space, wherein each image tile of the second set of image tiles is represented by a respective vector of the sixth set of embedding vectors; (viii) comparing the fourth set of embedding vectors to the sixth set of embedding vectors to generate a second reconstructive loss; and (ix) further updating (a) the second model based on the second contrastive loss, (b) the third model based on the second reconstructive loss, and (c) the first model based on the second contrastive loss and the second reconstructive loss. [00148] EEE 11 is the method of EEE 9 or 10, further comprising: (i) obtaining an objective image; (ii) obtaining a set of text descriptions for a set of novel objects; (iii) applying, to the second model following training of the second model, the text descriptions for the novel objects to generate respective novel object identity embedding vectors in the image-text embedding space; (iv) segmenting, into an objective set of image tiles, the objective image; (v) applying, to the fine-tuned model following training of the fine-tuned model, representations of the objective set of image tiles to generate an objective set of embedding vectors in the image-text embedding space; (vi) applying, to the detector model following training of the detector model, the objective set of embedding vectors to generate a predicted objective location of an object in the objective image; (vii) based on the predicted objective location, selecting a subset of the objective set of embedding vectors that represent image tiles of the objective set of image tiles that depict the object in the objective image; (viii) based on the selected subset of the objective set of embedding vectors, determining an objective summary embedding vector in the image-text embedding space for the object in the objective image; and (ix) determining an identity of the object in the objective image by comparing the objective summary embedding vector to the novel object identity embedding vectors. [00149] EEE 12 is the method of EEE 11, further comprising: (i) applying, to the first model following training of the first model, representations of the objective set of image tiles to generate a frozen set of embedding vectors in the image-text embedding space; (ii) based on the predicted objective location, selecting a subset of the frozen set of embedding vectors that represent image tiles of the objective set of image tiles that depict the object in the objective image; and (iii) based on the selected subset of the objective set of embedding vectors, determining a frozen summary embedding vector in the image-text embedding space for the object in the objective image, wherein determining the identity of the object in the objective image comprises comparing a combination of (a) the objective summary embedding vector and (b) the frozen summary embedding vector to the novel object identity embedding vectors. [00150] EEE 13 is a method comprising: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to a first trained vision- language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space. [00151] EEE 14 is the method of EEE 13, wherein the first trained vision-language model is the first model as trained in any of EEEs 1-12. [00152] EEE 15 is the method of any of EEEs 13-14, further comprising: (i) determining, based on the target set of embedding vectors, a target summary embedding vector for the target image; (ii) receiving a query text; (iii) applying, to a second trained language model, the query text to generate a query embedding vector in the image-text embedding space; (iv) determining a degree of similarity between the target summary embedding vector and the query embedding vector; and (v) responsive to at least one of (a) determining that the degree of similarity is less than a threshold degree of similarity or (b) determining that the degree of similarity is greater than at least one alternative degree of similarity between the query embedding vector and at least one target summary embedding vector for at least one alternative target image. [00153] EEE 16 is a method comprising: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; (iii) applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space; (iv) applying, to a detector model, the target set of embedding vectors to generate a predicted target location of an object in the target image; (v) based on the predicted target location, selecting a subset of the target set of embedding vectors that represent image tiles of the target set of image tiles that depict the object in the target image; and (vi) based on the selected subset of the target set of embedding vectors, determining a target summary embedding vector in the image-text embedding space for the object in the objective image. [00154] EEE 17 is the method of EEE 16, wherein the first trained vision-language model is the fine-tuned model as trained in any of EEEs 9-12 and the detector model is the detector model as trained in any of EEEs 9-12. [00155] EEE 18 is the method of EEE 16 or 17, further comprising: (i) receiving a query text; (ii) applying, to a second trained language model, the query text to generate a query embedding vector in the image-text embedding space; (iii) determining a degree of similarity between the target summary embedding vector and the query embedding vector; and (iv) responsive to at least one of (a) determining that the degree of similarity is less than a threshold degree of similarity or (b) determining that the degree of similarity is greater than at least one alternative degree of similarity between the query embedding vector and at least one target summary embedding vector for at least one alternative target image. [00156] EEE 19 is the method of any of EEEs 16-17, further comprising: determining an identity of the object in the target image by comparing the target summary embedding vector to a set of pre-defined object identity embedding vectors. [00157] EEE 20 is a method comprising: (i) obtaining training data, wherein the training data comprises a plurality of images; and (ii) using the training data, iteratively training a model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an embedding space, wherein iteratively training the model comprises: (a) segmenting, into a first set of image tiles, a first image of the plurality of images; (b) applying, to the model to generate a first set of embedding vectors in the embedding space, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles within the first image; (c) based on the first set of embedding vectors, determining a first loss; (d) segmenting, into a second set of image tiles, a second image of the plurality of images; (e) applying, to the model to generate a second set of embedding vectors in the embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles within the second image; (f) based on the second set of embedding vectors, determining a second loss; and (g) updating the model based on the first loss and the second loss. [00158] EEE 21 is the method of EEE 20, wherein the model comprises a vision transformer. [00159] EEE 22 is the method of any of EEEs 20-21, wherein updating the model based on the first loss and the second loss comprises: (i) updating the model based on the first loss, wherein applying the representations of the second set of image tiles to the model to generate the second set of embedding vectors comprises applying the representations of the second set of image tiles to the model as updated based on the first loss; and (ii) further updating the first model based on the second loss. [00160] EEE 23 is the method of any of EEEs 20-22, further comprising: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to the model following training of the model, representations of the target set of image tiles to generate a target set of embedding vectors in the embedding space. [00161] EEE 24 is a method comprising: (i) obtaining a target image; (ii) segmenting, into a target set of image tiles, the target image; and (iii) applying, to a trained model, representations of the target set of image tiles to generate a target set of embedding vectors in an embedding space, wherein the trained model is the model as trained in any of EEEs 20-23. [00162] EEE 25 is a non-transitory computer readable medium having stored thereon program instructions executable by at least one processor to cause the at least one processor to perform the method of any preceding EEE. [00163] EEE 26 is a system comprising: (i) at least one processor; and (ii) a non- transitory computer-readable medium, having stored therein instructions executable by the at least one processor to cause the system to perform the method of any of EEEs 1-24.

Claims

CLAIMS What is claimed is: 1. A method comprising: obtaining training data, wherein the training data comprises a plurality of images and a plurality of text descriptions, wherein each text description of the plurality of text descriptions describes a respective image of the plurality of images; using the training data, iteratively training (i) a first model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an image-text embedding space, and (ii) a second model to receive input text and generate therefor an embedding vector that represents the input text in the image-text embedding space, wherein iteratively training the first and second models comprises: applying a first text description of the plurality of text descriptions to the second model to generate a first embedding vector in the image-text embedding space; segmenting, into a first set of image tiles, a first image of the plurality of images that is described by the first text description; applying the first set of image tiles to the first model to generate a first set of embedding vectors in the image-text embedding space; based on the first set of embedding vectors, determining a first summary embedding vector for the first image; comparing the first summary embedding vector to the first embedding vector to generate a first contrastive loss; applying a first masked subset of the first set of image tiles to the first model to generate a second set of embedding vectors in the image-text embedding space; applying the second set of embedding vectors to a third model to generate a third set of embedding vectors in the image-text embedding space, wherein each image tile of the first set of image tiles is represented by a respective vector of the third set of embedding vectors; comparing the first set of embedding vectors to the third set of embedding vectors to generate a first reconstructive loss; and updating (i) the second model based on the first contrastive loss, (ii) the third model based on the first reconstructive loss, and (iii) the first model based on the first contrastive loss and the first reconstructive loss. The method of claim 1, wherein the first model comprises a vision transformer.

3. The method of claim 2, wherein applying the first set of image tiles to the first model to generate the first set of embedding vectors comprises applying, to the first model, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles, wherein applying the first masked subset of image tiles to the first model to generate the second set of embedding vectors comprises applying, to the first model, representations of the first masked subset of image tiles that include positional embedding information about the location of respective image tiles of the first masked subset of image tiles, and wherein iteratively training the first and second models further comprises: applying a second text description of the plurality of text descriptions to the second model to generate a second embedding vector in the image-text embedding space; segmenting, into a second set of image tiles, a second image of the plurality of images that is described by the second text description; applying, to the first model to generate a fourth set of embedding vectors in the image-text embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles; based on the fourth set of embedding vectors, determining a second summary embedding vector for the second image; comparing the second summary embedding vector to the second embedding vector to generate a second contrastive loss; applying, to the first model to generate a fifth set of embedding vectors in the image-text embedding space, representations of the second masked subset of image tiles that do not include positional embedding information about the location of respective image tiles of the second masked subset of image tiles; applying the fifth set of embedding vectors to the third model to generate a sixth set of embedding vectors in the image-text embedding space, wherein each image tile of the second set of image tiles is represented by a respective vector of the sixth set of embedding vectors; comparing the fourth set of embedding vectors to the sixth set of embedding vectors to generate a second reconstructive loss; and further updating (i) the second model based on the second contrastive loss, (ii) the third model based on the second reconstructive loss, and (iii) the first model based on the second contrastive loss and the second reconstructive loss.

4. The method of claim 1, wherein generating the first contrastive loss comprises generating the first contrastive loss across a batch of first embedding vectors and first summary vectors by: determining a text-to-image loss by determining a sum, across each pair of first embedding vector and first summary vector in the batch, of a logarithm of the cosine similarity between a given first embedding vector and the corresponding first summary vector divided by the sum, across each first summary vector in the batch, of the cosine similarities between the given first embedding vector and each first summary vector in the batch; determining an image-to-text loss by determining a sum, across each pair of first embedding vector and first summary vector in the batch, of a logarithm of the cosine similarity between a given first summary vector and the corresponding first embedding vector divided by the sum, across each first embedding vector in the batch, of the cosine similarities between the given first summary vector and each first embedding vector in the batch; and determining the first contrastive loss as an average of the text-to-image loss and the image-to-text loss.

5. The method of claim 1, wherein applying the first set of image tiles to the first model to generate the first set of embedding vectors comprises applying, to the first model, only those image tiles of the first set of image tiles that are not part of the first masked subset.

6. The method of claim 1, wherein determining the first summary embedding vector for the first image comprises performing global average pooling across the first set of embedding vectors.

7. The method of any preceding claim, further comprising: obtaining a target image; segmenting, into a target set of image tiles, the target image; and applying, to the first model following training of the first model, representations of the target set of image tiles to generate a target set of embedding vectors in the image-text embedding space.

8. The method of claim 7, further comprising: determining, based on the target set of embedding vectors, a target summary embedding vector for the target image; receiving a query text; applying, to the second model following training of the second model, the query text to generate a query embedding vector in the image-text embedding space; determining a degree of similarity between the target summary embedding vector and the query embedding vector; and responsive to at least one of (i) determining that the degree of similarity is less than a threshold degree of similarity or (ii) determining that the degree of similarity is greater than at least one alternative degree of similarity between the query embedding vector and at least one target summary embedding vector for at least one alternative target image.

9. The method of any of claims 1-6, further comprising generating (i) a fine-tuned model by further training of the first model and (ii) a detector model, by: obtaining additional training data that comprises a set of images, each image annotated with the location and identity of at least one object within the image, wherein the identities of the objects represented in the set of images of the additional training data are limited to an enumerated list of object identities, and wherein the additional training data additionally comprises, for each object identity in the enumerated list of object identities, a text description for the object identity; applying, to the second model following training of the second model, the text descriptions for the object identities to generate respective object identity embedding vectors in the image-text embedding space; and using the additional training data, iteratively training (i) the fine-tuned model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in the image-text embedding space, and (ii) a detector head to receive sets of embedding vectors output from the fine-tuned model and generate therefrom a predicted location of an object in the input image and a predicted embedding vector, in the image-text embedding space, that represents the identity of the object in the input image, wherein iteratively training the fine-tuned model and the detector head comprises: segmenting, into a third set of image tiles, a third image of the additional training data, wherein the third image is annotated with a first location and a first identity of a first object represented in the third image; applying the third set of image tiles to the fine-tuned model to generate a seventh set of embedding vectors in the image-text embedding space; applying the seventh set of embedding vectors to the detector model to generate a first predicted location of the first object; based on the first predicted location, selecting a subset of the seventh set of embedding vectors that represent image tiles of the third set of image tiles that depict the first object; based on the selected subset of the seventh set of embedding vectors, determining a predicted summary embedding vector in the image-text embedding space for the first object; comparing the first predicted location to an annotated location of the first object within the third image to generate a location loss; comparing the first summary embedding vector to the object identity embedding vector that corresponds to the identity of the first object to generate an identification loss; and updating the fine-tuned model and the detector model based on the location loss and the identification loss.

10. The method of claim 9, wherein the first model comprises a vision transformer, wherein applying the first set of image tiles to the first model to generate the first set of embedding vectors comprises applying, to the first model, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles, wherein applying the first masked subset of image tiles to the first model to generate the second set of embedding vectors comprises applying, to the first model, representations of the first masked subset of image tiles that include positional embedding information about the location of respective image tiles of the first masked subset of image tiles, and wherein iteratively training the first and second models further comprises: applying a second text description of the plurality of text descriptions to the second model to generate a second embedding vector in the image-text embedding space; segmenting, into a second set of image tiles, a second image of the plurality of images that is described by the second text description; applying, to the first model to generate a fourth set of embedding vectors in the image-text embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles; based on the fourth set of embedding vectors, determining a second summary embedding vector for the second image; comparing the second summary embedding vector to the second embedding vector to generate a second contrastive loss; applying, to the first model to generate a fifth set of embedding vectors in the image-text embedding space, representations of the second masked subset of image tiles that do not include positional embedding information about the location of respective image tiles of the second masked subset of image tiles; applying the fifth set of embedding vectors to the third model to generate a sixth set of embedding vectors in the image-text embedding space, wherein each image tile of the second set of image tiles is represented by a respective vector of the sixth set of embedding vectors; comparing the fourth set of embedding vectors to the sixth set of embedding vectors to generate a second reconstructive loss; and further updating (i) the second model based on the second contrastive loss, (ii) the third model based on the second reconstructive loss, and (iii) the first model based on the second contrastive loss and the second reconstructive loss.

11. The method of claim 9, further comprising: obtaining an objective image; obtaining a set of text descriptions for a set of novel objects; applying, to the second model following training of the second model, the text descriptions for the novel objects to generate respective novel object identity embedding vectors in the image-text embedding space; segmenting, into an objective set of image tiles, the objective image; applying, to the fine-tuned model following training of the fine-tuned model, representations of the objective set of image tiles to generate an objective set of embedding ^{vectors in the image-text embedding space;} a_{pplying, to the detector model following training of the detector model, the objective} set of embedding vectors to generate a predicted objective location of an object in the objective image; based on the predicted objective location, selecting a subset of the objective set of embedding vectors that represent image tiles of the objective set of image tiles that depict the object in the objective image; based on the selected subset of the objective set of embedding vectors, determining an objective summary embedding vector in the image-text embedding space for the object in the objective image; and determining an identity of the object in the objective image by comparing the objective summary embedding vector to the novel object identity embedding vectors.

12. The method of claim 11, further comprising: applying, to the first model following training of the first model, representations of the objective set of image tiles to generate a frozen set of embedding vectors in the image-text ^{embedding space;} b_{ased on the predicted objective location, selecting a subset of the frozen set of} embedding vectors that represent image tiles of the objective set of image tiles that depict the object in the objective image; and based on the selected subset of the objective set of embedding vectors, determining a frozen summary embedding vector in the image-text embedding space for the object in the objective image, wherein determining the identity of the object in the objective image comprises comparing a combination of (i) the objective summary embedding vector and (ii) the frozen summary embedding vector to the novel object identity embedding vectors.

13. A method comprising: obtaining a target image; segmenting, into a target set of image tiles, the target image; and applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space.

14. The method of claim 13, wherein the first trained vision-language model is the first model as trained in any of claims 1-12.

15. The method of claim 13, further comprising: determining, based on the target set of embedding vectors, a target summary embedding vector for the target image; receiving a query text; applying, to a second trained language model, the query text to generate a query embedding vector in the image-text embedding space; determining a degree of similarity between the target summary embedding vector and the query embedding vector; and responsive to at least one of (i) determining that the degree of similarity is less than a threshold degree of similarity or (ii) determining that the degree of similarity is greater than at least one alternative degree of similarity between the query embedding vector and at least one target summary embedding vector for at least one alternative target image.

16. A method comprising: obtaining a target image; segmenting, into a target set of image tiles, the target image; applying, to a first trained vision-language model, representations of the target set of image tiles to generate a target set of embedding vectors in an image-text embedding space; applying, to a detector model, the target set of embedding vectors to generate a predicted target location of an object in the target image; based on the predicted target location, selecting a subset of the target set of embedding vectors that represent image tiles of the target set of image tiles that depict the object in the target image; and based on the selected subset of the target set of embedding vectors, determining a target summary embedding vector in the image-text embedding space for the object in the objective image.

17. The method of claim 16, wherein the first trained vision-language model is the fine-tuned model as trained in any of claims 9-12 and the detector model is the detector model as trained in any of claims 9-12.

18. The method of claim 16, further comprising: receiving a query text; applying, to a second trained language model, the query text to generate a query embedding vector in the image-text embedding space; determining a degree of similarity between the target summary embedding vector and the query embedding vector; and responsive to at least one of (i) determining that the degree of similarity is less than a threshold degree of similarity or (ii) determining that the degree of similarity is greater than at least one alternative degree of similarity between the query embedding vector and at least one target summary embedding vector for at least one alternative target image.

19. The method of claim 16, further comprising: determining an identity of the object in the target image by comparing the target summary embedding vector to a set of pre-defined object identity embedding vectors.

20. A method comprising: obtaining training data, wherein the training data comprises a plurality of images; and using the training data, iteratively training a model to receive tiles of an input image and generate therefor respective embedding vectors that represent the tiles of the input image in an embedding space, wherein iteratively training the model comprises: segmenting, into a first set of image tiles, a first image of the plurality of images; applying, to the model to generate a first set of embedding vectors in the embedding space, representations of the first set of image tiles that include positional embedding information about the location of respective image tiles of the first set of image tiles within the first image; based on the first set of embedding vectors, determining a first loss; segmenting, into a second set of image tiles, a second image of the plurality of images; applying, to the model to generate a second set of embedding vectors in the embedding space, representations of the second set of image tiles that do not include positional embedding information about the location of respective image tiles of the second set of image tiles within the second image; based on the second set of embedding vectors, determining a second loss; and updating the model based on the first loss and the second loss.

21. The method of claim 20, wherein the model comprises a vision transformer.

22. The method of any of claims 20-21, wherein updating the model based on the first loss and the second loss comprises: updating the model based on the first loss, wherein applying the representations of the second set of image tiles to the model to generate the second set of embedding vectors comprises applying the representations of the second set of image tiles to the model as updated based on the first loss; and further updating the first model based on the second loss.

23. The method of any of claims 20-21, further comprising: obtaining a target image; segmenting, into a target set of image tiles, the target image; and applying, to the model following training of the model, representations of the target set of image tiles to generate a target set of embedding vectors in the embedding space.

24. A method comprising: obtaining a target image; segmenting, into a target set of image tiles, the target image; and applying, to a trained model, representations of the target set of image tiles to generate a target set of embedding vectors in an embedding space, wherein the trained model is the model as trained in any of claims 20-23.

25. A non-transitory computer readable medium having stored thereon program instructions executable by at least one processor to cause the at least one processor to perform the method of any preceding claim.

26. A system comprising: at least one processor; and a non-transitory computer-readable medium, having stored therein instructions executable by the at least one processor to cause the system to perform the method of any of claims 1-24.