Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/776—Validation; Performance evaluation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
  • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
  • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Definitions

• the present disclosure relates generally to machine learning. More particularly, the present disclosure relates to the use of visual transformers in combination with sparse application of video kernels for joint image and video learning.
• Transformer models are a type of machine learning model that leverages a selfattention mechanism over a sequence of tokens or embeddings at each of a number of layers.
• One example “vanilla’ 7 transformer architecture is described in Vaswani et al., Attention is all you need. ⁇ n NeurlPS 2017.
• transformer models were originally applied to natural language settings, they have since been adapted for and widely applied to single image vision tasks such as image classification.
• Transformers applied to vision tasks can be referred to as visual transformers or vision transformers.
• One example transformer model configured to process single images is the ViT model described in Dosovitskiy et al., An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
• transformers As applied to vision tasks, transformers have become an ubiquitous backbone for visual representation learning, leading to many advances in image understanding, multimodal tasks, and self-supervised learning, etc.
• Video understanding is an essential computer vision task.
• adaptations of transformer models to video are both challenging and computationally intensive.
• One example aspect of the present disclosure is directed to a computer system for performing video processing tasks with improved computational efficiency, the computer system comprising: one or more processor and one or more non-transitory computer-readable media.
• the one or more non-transitory computer-readable media collectively store a machine-learned model comprising: a video kernel configured to be applied to a plurality of data samples from a set of video data to respectively generate a plurality of video tokens, wherein each data sample comprises at least a portion of multiple image frames included in the set of video data; and a visual transformer configured to process the plurality of video tokens to generate a model output.
• the one or more non-transitory computer-readable media collectively store instructions that, when executed by the one or more processors, cause the computer system to perform operations.
• the operations include: processing the set of video data with the machine-learned model to generate the model output; wherein processing the set of video data with the machine-learned model comprises sparsely applying the video kernel to the set of video data.
• Figure 1 depicts a graphical diagram of example machine learning models according to example embodiments of the present disclosure.
• Figure 2 depicts a graphical diagram of example machine learning models according to example embodiments of the present disclosure.
• Figure 3 depicts a graphical diagram of example machine learning models according to example embodiments of the present disclosure.
• Figures 4A-C depict block diagrams of example computing systems and devices according to example embodiments of the present disclosure.
• the present disclosure is directed to machine-learned models for performing video processing with improved efficiency.
• the machine-learned model can perform the sparse application of one or more video kernels to a set of video data to generate video tokens that can, for example, be provided as input to a visual transformer.
• example implementations of the present disclosure are directed to an approach which can turn a visual transformer (e.g., a ViT encoder) into an efficient video model.
• example implementations described herein can seamlessly work with both image and video inputs. Specifically, by sparsely sampling the inputs, the model is able to do training and inference from both inputs.
• the proposed model is easily scalable and can optionally be adapted to large-scale pre-trained visual transformers without requiring full finetuning.
• the present disclosure proposes an effective model, example implementations of which can be referred to as "‘TubeViT”.
• the proposed model can utilize existing visual transformer architectures (e.g., a standard ViT model) seamlessly for both image and videos.
• the proposed models can implement Sparse Video Tubes, which are a lightweight approach for video learning (e.g., joint image and video learning).
• Example implementations of the proposed techniques work by sparsely sampling one or more variously-sized 3D space-time tubes from the video to generate learnable tokens, which are used by the visual transformer.
• FIG. 1 shows an example model according to aspects of the present disclosure.
• a visual transformer 12 can be used to process image inputs 14, video inputs 16, and/or both inputs 14 and 16, thereby providing an efficient video backbone and more accurate performance.
• the input to the transformer can also include tokens generated from other modalities of data beyond the image and/or video tokens.
• tokens generated from other modalities of data beyond the image and/or video tokens For example, speech tokens, natural language tokens, and/or other tokens can be provided as input for other forms of multi-modal processing tasks.
• example implementations of the present disclosure are able to better share the weights learned for both images and videos. This is in contrast to prior works that either inflate kernels or add new temporal-specific layers. Further, due to the sparse sampling, the number of tokens remains low, which is also beneficial for both for reducing computational expenditure (e.g., in terms of FLOPs) and improving performance.
• Technical effects and benefits of the present disclosure therefore include both reduced consumption of computational resources and improved performance or functionality of computer systems with respect to various computer vision tasks.
• video-based computer vision tasks can include video classification, video recognition, object detection or classification in videos, embedding generation (e.g., for video representation), and/or other video processing or vision tasks.
• the present disclosure provides sparse video tubes, which can be obtained by sparsely sampling videos with various sized 3D space-time tubes or video kernels.
• example implementations of the present disclosure can accomplish some or all of the following: (1) a universal visual backbone which easily adapts a visual transformer (e.g., ViT) to videos; (2) joint image and video understanding which seamlessly uses either input; and/or (3) an easy-to-scale approach for video understanding, which can also leverage already trained (large) visual transformer models.
• a universal visual backbone which easily adapts a visual transformer (e.g., ViT) to videos
• joint image and video understanding which seamlessly uses either input
• an easy-to-scale approach for video understanding which can also leverage already trained (large) visual transformer models.
• the proposed techniques enable computationally efficient application of a single transformer model to a large set of video data.
• Example differences that enable this computational efficiency include: that the tubes are sparsely applied to the raw input, that the model consists of different shaped, possibly overlapping tubes, and that the model uses a single, shared backbone network. This leads to both more efficient and accurate models.
• the model can be entirely shared between the image and video modalities. This is a valuable distinction as it not only improves performance for both tasks, but is also more generally applicable to vision tasks.
• Another aspect of the present disclosure is directed to a technique for efficiently scaling from a small video model to a large video model using a pre-trained image model.
• video models are typically computationally expensive to train, and previous works have studied ways to leverage already trained models, such as using frozen ones or adapting them to videos.
• Example implementations of the present disclosure expand on these ideas, and use the Sparse Video Tubes to adapt much larger pre-trained visual transformer models to videos with lightweight training.
• powerful large video models can be created while consuming fewer computational resources such as processor usage, memory usage, network bandwidth, etc.
• a standard ViT architecture takes an image and converts it into patch embedding, for example, by using a 16 x 16 2D convolutional kernel, with a 16 x 16 stride. This results in a sequence of patches as the image representation, e g., 196 for a 224 x 224 input image.
• p rior applications of visual transformers either used the same, dense 2D patches independently for all frames (e.g.. densely applied 2D patches on a per-frame basis) or used completely dense 3D kernels (e.g., densely applied 3D patches to multiple frames with a temporal stride of one).
• ViVit As one example, ViVit (Amab et al., Vivit: A video vision transformer. In ICCV, 2021) densely applied 2 x 16 x 16 or 4 x 16 x 16 to all frames in the video. In both cases (dense application of 2D kernels or dense application of 3D kernels), this results in significantly more tokens, e.g., T * 196, where T is the number of frames.
• these tubes or patches are then linearly projected into an embedding space, z ( G d .
• This sequence of tokens is then processed by a transformer encoder, using standard components, e g., MSA - the multi-head self-attention and MLP - the standard transformer projection layer.
• MSA the multi-head self-attention
• MLP the standard transformer projection layer.
• a transformer can compute the representation y- and next token features z for all the z L tokens (LN denotes Layer Norm):
• a system can factorize the attention mechanism, to have a spatial and temporal attention or use multiple views with smaller, view level transformers.
• factorization of the attention mechanism does not directly resolve challenges associated with the number of tokens.
• example implementations of the present disclosure implement a simple and straightforward method which is seamlessly applicable to both images and videos.
• some example implementations can of the present disclosure can follows the standard ViT tokenization approach for images: a 2D convolution with a 16 x 16 kernel.
• example implementations build on the observation that sparseness is effective for videos. Therefore, rather than following the pnor works that densely tokenize the video, some example implementations instead use the same 2D kernel, but with a large temporal stride, for example, applied to every 16th frame.
• a large temporal stride for example, applied to every 16th frame.
• some example implementations can create sparse video tubes of different shapes (e.g., various 3D shapes). Specifically, video kernels having these shapes can be applied to video data sampled according to these shapes to generate video tokens. Kernels can also be referred to as filters.
• 3D shapes can generally refer to shapes having a temporal length (e.g., length in terms of number of frames), a spatial height, and a spatial width. These shapes can also be thought of as 4D shapes when a channel depth of the data is considered. Some example implementations can also optionally add an offset to the start location, so that the patches do not always start at (0,0,0) and this allows a reduction in the overlap between the tubes. This is illustrated in Figure 2.
• Tubes of various sizes are also used in the MultiView approach for video classification (Yan et al., Multiview transformers for video recognition. In CVPR, 2022.).
• MultiView the 3D tubes are densely sampled and respectively processed by multiple different view-specific transformers, resulting in a more computationally intensive approach.
• some example implementations of the present disclosure also allow for overlap between the tubes.
• example implementations of the present disclosure enable seamless fusion of the image- and video- visual information.
• the sparse spatial sampling allows sharing the image and frame tokens and the sparse video tubes create a low number of video-specific tokens. This enables better sharing of the visual transformer model between images and videos.
• Figure 2 illustrates an example application of the model 52 to a set of video data 54.
• the example machine-learned model 52 can include a video kernel 56 configured to be applied to a plurality of data samples from the set of video data 54 to respectively generate a plurality of video tokens 58. Each data sample can include at least a portion of multiple image frames included in the set of video data 54.
• kernel 56 is called out with reference to Figure 2, additional example video kernels 60 and 62 are illustrated, which can also be sparsely applied to generate additional video tokens.
• the model 52 can also include a visual transformer 64 configured to process the plurality of video tokens 58 (and any other tokens present) to generate a model output 66.
• the machine-learned model 52 can process the set of video data 54 to generate the model output 66. Specifically, processing the set of video data 54 with the machine-learned model 52 can include sparsely applying the video kernel 56 to the set of video data.
• the video kernel 56 has a spatial dimension size and sparsely applying the video kernel 56 to the set of video data 54 can include applying the video kernel 56 with a spatial stride greater than the spatial dimension size of the video kernel 56 to achieve spatial sparseness.
• the video kernel 56 has a temporal dimension size and sparsely applying the video kernel 56 to the set of video data 54 can include applying the video kernel 56 with a temporal stride greater than the temporal dimension size of the video kernel 56 to achieve temporal sparseness.
• the video kernel 56 can be directly applied to the set of video data 54 (e.g.. as opposed to linearly projecting or transforming the video data 54 first.
• the video kernel 56 can be directly applied to pixel values included in the set of video data 54.
• the machine-learned model 52 can further include one or more image kernels (e.g.. image kernel 68) configured to be applied to an individual image frame (e.g., frame 70) of the set of video data to generate a plurality of image tokens (e.g., tokens 74) from the individual image frame.
• image kernels e.g.. image kernel 68
• image tokens e.g., tokens 74
• the machine-learned model 52 can include a single visual transformer (e.g., the transformer 64) configured to jointly process both the plurality of video tokens (e.g., video tokens 58) and the plurality of image tokens (e.g., image tokens 74) to generate the model output 66.
• a single visual transformer e.g., the transformer 64
• the transformer 64 configured to jointly process both the plurality of video tokens (e.g., video tokens 58) and the plurality of image tokens (e.g., image tokens 74) to generate the model output 66.
• the transformer 64 configured to jointly process both the plurality of video tokens (e.g., video tokens 58) and the plurality of image tokens (e.g., image tokens 74) to generate the model output 66.
• these multiple transformer approaches are required due to the density of sampling combined with the quadratic cost of the attention mechanism.
• data samples to which different kernels are applied can be overlapping.
• video kernel 56 and video kernel 60 are applied to overlapping samples in the frames shown on the left-hand side of video data 54.
• the model output 66 can be or include a video classification output.
• other forms of outputs for other video understanding or computer vision tasks can be output as well, including, as examples, video recognition, object recognition, object detection, image classification, image recognition, video representation, image representation, and/or others.
• some example implementations of the present disclosure can use a fixed sine/cosine embedding.
• the stride, kernel shape and offsets of each tube can be taken into account when applying the positional embeddings. This ensures that the positional embedding of each patch and tube has the global spatio-temporal location of that tube.
• the embeddings can be computed as follows.
• T is a constant hyperparameter (e.g., 10,000).
• j from 0 to d//6 (d is the number of features)
• z L [t, x, y] represents the center of the tube, taking into account any strides and offsets used in the tube construction (the channel dimension is not show n here).
• some example implementations can concatenate all the tokens together and apply a standard transformer model. This simple structure lets the model share the majority of the weights between all inputs, which is quite beneficial.
• Any number of methods or shapes can be used to create any number of visual tubes.
• One example approach can include 2 tubes: the 1 x 16 x 16 x d tube used to tokenize the image and a 8 x 8 x 8 x d tube additionally used for the video, where d represents the channel depth. Both tubes can have strides of 16 x 16 x 16. This base tokenizer provides strong performance, any number of variations from this example are possible.
• Multi-Tube add multiple tubes of various sizes to the core approach.
• some example implementations can include temporally long and spatially small tubes, such as 16 x 4 x 4 to learn long actions, or more spatially focused tubes such as a 2 x 16 x 16 tube.
• temporally long and spatially small tubes such as 16 x 4 x 4 to learn long actions
• spatially focused tubes such as a 2 x 16 x 16 tube.
• tube shape and stride which are possible.
• any factor can be used that comports with the number of channels available.
• the model can leam one 3D kernel of a certain shape (e.g., 8 x 8 x 8). Tri-linear interpolation can then be performed to reshape the kernel any number of various different sizes, e.g., 4x16x16 or 32x4x4. etc. depending on the tube configuration. Any sized kernel can be created from this single kernel. This method has several advantages. (1) It reduces the number of learned parameters that are only used on the video stream. (2) It enables more flexible usage of the kernels, e.g., it can be made longer to handle longer videos, or spatially larger to find small objects.
• the proposed approach can seamlessly adapt to either image, video, or both inputs. While image+video joint inputs are rare, the ability to use them together while training is beneficial given that many datasets with valuable annotations (e.g., ImageNet, Kinetics) come from either image sources or video sources but not both.
• image+video joint inputs are rare, the ability to use them together while training is beneficial given that many datasets with valuable annotations (e.g., ImageNet, Kinetics) come from either image sources or video sources but not both.
• Some example implementations also leverage a more efficient way of scaling up the models (e.g., with an example illustrated in Figure 3). Training large visual transformer models is computationally expensive, especially for videos. Since nearly all the components of the proposed model are shared between the both images and videos, one example method is to utilize large models without having heavy fine-tuning.
• a smaller model can be trained jointly on images and videos. This gives a set of weights for the tubes (e.g., the values for the parameters of the video kernels) which are used to generate the video tokens. Then, a large pre-trained image ViT can be modified by further adding the learned tubes. These tubes can use the same kernel weights as the smaller model, and so some example implementations can avoid further training them (though alternatively they may optionally continue to be refined/leamed if desired). Since larger ViTs generally use more channel dimensions than smaller ones, the space-to-depth transform can be optionally used again here to create tokens with the proper channel dimensions without needing new weights.
• the scaling approach can include picking a point in the network and freeze all the layers before it, for example, the 26th of 32 layers in ViT-H (though alternatively the entire model can continue to be refined/leamed if desired)
• this gate has no effect on the representation, and thus the ViT is unchanged. However, it can learn to incorporate the raw tubes at this point and further refine the later weights.
• Figure 3 shows an example approach for scaling of TubeViT models.
• the proposed scaling approach can expand model capacity' for video models leveraging large pre-trained ViTs.
• TubeViT can easily train a small-scale model on both image and video data. Then, the sparse video tubes can be imported to a much larger image-only trained ViT, which can be mostly frozen and/or fine-tuned in part.
• Figure 4A depicts a block diagram of an example computing system 100 according to example embodiments of the present disclosure.
• the system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.
• the user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.
• a personal computing device e.g., laptop or desktop
• a mobile computing device e.g., smartphone or tablet
• a gaming console or controller e.g., a gaming console or controller
• a wearable computing device e.g., an embedded computing device, or any other type of computing device.
• the user computing device 102 includes one or more processors 112 and a memory 114.
• the one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected.
• the memory' 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.
• the memory 114 can store data 1 16 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.
• the user computing device 102 can store or include one or more machine-learned models 120.
• the machine-learned models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models.
• Neural networks can include feed-forward neural networks, recurrent neural networks (e g., long short-term memory’ recurrent neural networks), convolutional neural networks or other forms of neural networks.
• Some example machine-learned models can leverage an attention mechanism such as self-attention.
• some example machine-learned models can include multi-headed self-attention models (e.g., transformer models).
• Example machine-learned models 120 are discussed with reference to Figures 1-3.
• the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory' 114, and then used or otherwise implemented by the one or more processors 112.
• the user computing device 102 can implement multiple parallel instances of a single machine-learned model 120.
• one or more machine-learned models 140 can be included in or otherwise stored and implemented by 7 the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship.
• the machine-learned models 140 can be implemented by the server computing system 140 as a portion of a web service.
• one or more models 120 can be stored and implemented at the user computing device 102 and/or one or more models 140 can be stored and implemented at the server computing system 130.
• the user computing device 102 can also include one or more user input components 122 that receives user input.
• the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus).
• the touch-sensitive component can serve to implement a virtual key board.
• Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.
• the server computing system 130 includes one or more processors 132 and a memory’ 134.
• the one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected.
• the memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.
• the memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.
• the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.
• the server computing system 130 can store or otherwise include one or more machine-learned models 140.
• the models 140 can be or can otherwise include various machine-learned models.
• Example machine-learned models include neural networks or other multi-layer non-linear models.
• Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.
• Some example machine-learned models can leverage an attention mechanism such as self-attention.
• some example machine-learned models can include multi-headed self-attention models (e g., transformer models).
• Example models 140 are discussed with reference to Figures 1-3.
• the user computing device 102 and/or the server computing system 130 can train the models 120 and/or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180.
• the training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.
• the training computing system 150 includes one or more processors 152 and a memory 154.
• the one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected.
• the memory 154 can include one or more non-transi lory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.
• the memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations.
• the training computing system 150 includes or is otherwise implemented by one or more server computing devices.
• the training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and/or 140 stored at the user computing device 102 and/or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors.
• a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function).
• Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions.
• Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.
• performing backwards propagation of errors can include performing truncated backpropagation through time.
• the model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.
• the model trainer 160 can train the machine-learned models 120 and/or 140 based on a set of training data 162.
• the training data 162 can include, for example, video data with an associated label, image data with an associated label, and/or other forms of training data.
• the training examples can be provided by the user computing device 102.
• the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.
• the model trainer 160 includes computer logic utilized to provide desired functionality.
• the model trainer 160 can be implemented in hardware, firmware, and/or software controlling a general purpose processor.
• the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors.
• the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.
• the network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links.
• communication over the network 180 can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP. HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).
• TCP/IP Transmission Control Protocol/IP
• HTTP HyperText Transfer Protocol
• SMTP Session Init Protocol
• FTP e.g., HTTP, HTTP, HTTP, FTP
• encodings or formats e.g., HTML, XML
• protection schemes e.g., VPN, secure HTTP, SSL
• the machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.
• the input to the machine-learned model(s) of the present disclosure can be image data.
• the machine-learned model(s) can process the image data to generate an output.
• the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data. etc ).
• the machine-learned model(s) can process the image data to generate an image segmentation output.
• the machine- learned model(s) can process the image data to generate an image classification output.
• the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.).
• the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.).
• the machine-learned model(s) can process the image data to generate an upscaled image data output.
• the machine-learned model(s) can process the image data to generate a prediction output.
• the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output.
• the machine-learned model(s) can process the text or natural language data to generate a semantic intent output.
• the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.).
• the machine-learned model (s) can process the text or natural language data to generate a prediction output.
• the input to the machine-learned model(s) of the present disclosure can be speech data.
• the machine-learned model(s) can process the speech data to generate an output.
• the machine-learned model(s) can process the speech data to generate a speech recognition output.
• the machine- learned model(s) can process the speech data to generate a speech translation output.
• the machine-learned model(s) can process the speech data to generate a latent embedding output.
• the machine-learned model(s) can process the speech data to generate an encoded speech output (e.g., an encoded and/or compressed representation of the speech data, etc.).
• an encoded speech output e.g., an encoded and/or compressed representation of the speech data, etc.
• the machine-learned model(s) can process the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data, etc.).
• the machine-learned model(s) can process the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.).
• the machine- learned model(s) can process the speech data to generate a prediction output.
• the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e.g., a latent space representation of an input, etc.).
• the machine-learned model(s) can process the latent encoding data to generate an output.
• the machine-learned model(s) can process the latent encoding data to generate a recognition output.
• the machine-learned model(s) can process the latent encoding data to generate a reconstruction output.
• the machine-learned model(s) can process the latent encoding data to generate a search output.
• the machine-learned model(s) can process the latent encoding data to generate a reclustering output.
• the machine-learned model(s) can process the latent encoding data to generate a prediction output.
• the input to the machine-learned model(s) of the present disclosure can be statistical data.
• Statistical data can be. represent, or otherwise include data computed and/or calculated from some other data source.
• the machine-learned model(s) can process the statistical data to generate an output.
• the machine- learned model(s) can process the statistical data to generate a recognition output.
• the machine-learned model(s) can process the statistical data to generate a prediction output.
• the machine-learned model(s) can process the statistical data to generate a classification output.
• the machine-learned model(s) can process the statistical data to generate a segmentation output.
• the machine-learned model(s) can process the statistical data to generate a visualization output.
• the machine-learned model(s) can process the statistical data to generate a diagnostic output.
• the input to the machine-learned model(s) of the present disclosure can be sensor data.
• the machine-learned model(s) can process the sensor data to generate an output.
• the machine-learned model(s) can process the sensor data to generate a recognition output.
• the machine-learned model(s) can process the sensor data to generate a prediction output.
• the machine-learned model(s) can process the sensor data to generate a classification output.
• the machine-learned model(s) can process the sensor data to generate a segmentation output.
• the machine-learned model(s) can process the sensor data to generate a visualization output.
• the machine-learned model(s) can process the sensor data to generate a diagnostic output.
• the machine-learned model(s) can process the sensor data to generate a detection output.
• the machine-learned model(s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding).
• the task may be an audio compression task.
• the input may include audio data and the output may comprise compressed audio data.
• the input includes visual data (e.g. one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task.
• the task may comprise generating an embedding for input data (e.g. input audio or visual data).
• the input includes visual data and the task is a computer vision task.
• the input includes pixel data for one or more images and the task is an image processing task.
• the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class.
• the image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest.
• the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories.
• the set of categories can be foreground and background.
• the set of categories can be object classes.
• the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value.
• the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.
• the input includes audio data representing a spoken utterance and the task is a speech recognition task.
• the output may comprise a text output which is mapped to the spoken utterance.
• the task comprises encrypting or decrypting input data.
• the task comprises a microprocessor performance task, such as branch prediction or memory address translation.
• Figure 4A illustrates one example computing system that can be used to implement the present disclosure.
• the user computing device 102 can include the model trainer 160 and the training dataset 162.
• the models 120 can be both trained and used locally at the user computing device 102.
• the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.
• Figure 4B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure.
• the computing device 10 can be a user computing device or a server computing device.
• the computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model.
• Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.
• each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components.
• each application can communicate with each device component using an API (e.g., a public API).
• the API used by each application is specific to that application.
• Figure 4C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure.
• the computing device 50 can be a user computing device or a server computing device.
• the computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer.
• Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.
• each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).
• the central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 4C. a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.
• the central intelligence layer can communicate with a central device data layer.
• the central device data layer can be a centralized repository' of data for the computing device 50. As illustrated in Figure 4C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).
• an API e.g., a private API

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Software Systems (AREA)
• Evolutionary Computation (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Computing Systems (AREA)
• Medical Informatics (AREA)
• General Health & Medical Sciences (AREA)
• Databases & Information Systems (AREA)
• Health & Medical Sciences (AREA)
• Artificial Intelligence (AREA)
• Computational Linguistics (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=89542016

Family Applications (1)

Country Status (4)

Families Citing this family (1)

Family Cites Families (3)

• 2023
  • 2023-11-22 CN CN202380080607.2A patent/CN120239875A/zh active Pending
  • 2023-11-22 EP EP23838260.0A patent/EP4605902A1/de active Pending
  • 2023-11-22 WO PCT/US2023/080947 patent/WO2024112910A1/en not_active Ceased
  • 2023-11-22 US US18/577,051 patent/US20250005924A1/en active Pending

Also Published As

Similar Documents

Legal Events