WO2025166290A1

WO2025166290A1 - Memory consolidation for neural networks which process media elements

Info

Publication number: WO2025166290A1
Application number: PCT/US2025/014194
Authority: WO
Inventors: Ivana BALAZEVIC; Yuge SHI; Rahma CHAABOUNI; Skanda Kumar KOPPULA; Pinelopi PAPALAMPIDI; Olivier Jean Henaff
Original assignee: DeepMind Technologies Ltd; GDM Holding LLC
Current assignee: DeepMind Technologies Ltd; GDM Holding LLC
Priority date: 2024-02-01
Filing date: 2025-01-31
Publication date: 2025-08-07
Anticipated expiration: 2026-08-01

Abstract

A media element processing system, for media elements such as videos, comprises a memory which is augmented by a memory control unit in each of number of steps based on a corresponding current segment of the media element, by adding to it a memory element. The memory element is formed based on an embedding of the current segment of the media element, and comprises a smaller number of values than the embedding of the current segment. The system uses the memory to process current segments of the media element. The data stored in the memory may be informative about a large number of previous segments of the media element, while limiting the required size of the memory and the number of tunable parameters employed in the system.

Description

MEMORY CONSOLIDATION FOR NEURAL NETWORKS WHICH PROCESS MEDIA ELEMENTS CROSS REFERENCE TO RELATED APPLICATIONS [0001] This application claims priority to GR Patent Application No.20240100063, filed on February 1, 2024, the entire contents of which are hereby incorporated by reference. BACKGROUND [0002] This specification relates to processing media elements, such as videos or sound signals, using neural networks. [0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY [0004] This specification describes a system, implemented as computer programs on one or more computers in one or more locations, that processes media elements, such as videos and/or sound signals. The media elements may be ones captured from a real-world environment by sensors (e.g. still or video camera(s) and/or microphone(s)). [0005] The system may process the media element(s) of the network input as a sequence of segments (chunks) of the media element. The segments may be, e.g., non-overlapping, portions of the media element, such as portions which collectively include the whole of the media element, and having an order in the sequence which is the same as the order of the corresponding portions of the media element. [0006] For example, the media element may comprise, or be, a video, i.e., a plurality of images (frames), each comprising one or more intensity values for each of an array of pixels. The frames may be in a sequence, e.g., spanning an interval of time depicted in the video. If so, each segment may comprise one or more successive frames of the video, with the frame(s) of each segment, except the first segment, being later in the video than the frame(s) of the preceding segment(s) of the sequence. The video may be one having a smooth moving-

1 37923819-1 image appearance when played. However, the term “video” is not limited in this respect. The video could instead be a sequence of still pictures (frames) taken by a still camera at intervals (e.g., of at least a second) long enough that, if the images were viewed successively, motions depicted in the images would not be smooth. Furthermore, a given segment of the video may include images captured by multiple cameras, e.g., during a corresponding sub-interval of an interval of time spanned by the video. [0007] Alternatively, the media element may be a sound signal (audio signal) describing sound during a time interval, e.g., as sound amplitude values in the time domain, or the time- frequency domain. The sequence of segments may comprise audio data for a series of corresponding (e.g., non-overlapping) portions of the time interval. Each of the segments of the media element is embedded (encoded), e.g., by an embedding unit, to generate an embedded segment of the media element. [0008] In a further possibility, the media element may comprise both at least one video and at least one sound signal (e.g., a soundtrack associated with the media element). [0009] In general terms, the proposed system comprises a memory which is augmented by a memory control unit in each of number of steps based on a corresponding current segment of the media element, by adding to it a memory element which is formed based on an embedding of the current segment of the media element. The memory element comprises a smaller number of values (e.g., it is a smaller number of bits) than the embedding of the current segment. [00010] This allows the system to use the memory to process current segments based on data stored in the memory which is based on a large number of previous segments of the media element, while limiting the required size of the memory and the number of tunable parameters employed in the system. [00011] Each embedding of a given segment may include a number of components referred to as “activations”, or “activation components”, or media “tokens”. The activations are each one or more numerical values, e.g., the same number of numerical values for each activation, denoted d, where d is an integer which may be equal to one but in an implementation is greater than one. [00012] The values used to produce a memory element based on an embedding of the current segment may be a selected (proper) subset of the activations of the embedding of the current segment. The subset may be selected from the embedding of the current segment in any way, for example randomly.

2 37923819-1 [00013] Alternatively, the subset may be chosen by a different mechanism, e.g., to find activations which are more representative of the embedding of the current segment. For example, the subset may be selected iteratively, with the first activation being chosen (e.g., at random, or as some default choice for the activation), and each successive member of the subset being chosen based on the previously selected activation(s). For example, each successive member of the subset may be chosen as one of the activations which has not already been chosen and which is furthest away from the closest one of the activations which has already been chosen. Here the terms “furthest” and “closest” are defined according to a distance metric for measuring the distance between two activations, e.g., Euclidean distance. [00014] Alternatively, the activations of an embedding of the current segment which are used to produce a memory element may be obtained from the embedding by performing a clustering algorithm on the activations of the embedding, e.g., to define a number of clusters of the activations which is smaller than the number of activations in the embedding of the current segment. Then the values may be selected based on the clusters, e.g., as “cluster” activations which are averages of the activations of the corresponding clusters. [00015] The components of the memory element also may be called “activations” or “activation components”. Each may be a set of d numerical values. Similarly, the memory, which may be composed of memory elements generated as discussed above (i.e., the memory may be empty before the first memory element is added to it), may be composed of “activations” (or “activation components”). [00016] The memory may be employed in a neural network which is a media element processing system (e.g., a video processing system or audio processing system) configured to process a network input comprising the media element, to generate a network output. The network input may include portions received at successive times and each comprising a corresponding segment of the media element, or alternatively the media element processing system may comprise an input unit which receives a network input comprising the media element, and divides the media element into the segments, for subsequent processing by the processing system. [00017] The media element processing system may further comprise a processing sub- system, configured to generate the network output, or to generate data upon which the network output is based, e.g., by an output unit of the media element processing system which receives the output of the processing sub-system. The outputs of the processing sub- system thus form a latent representation of a media element.

3 37923819-1 [00018] The processing sub-system also sequentially processes the segments of the media element. Thus, the successive segments of the media element are used by the processing sub-system (i) to generate an output based on the current segment of the media element and the memory, and (ii) by the memory control unit to update the memory. [00019] The processing sub-system may have one or more processing layers, each typically implemented as a neural network. In an implementation, there is a stack of processing layers, i.e., an ordered plurality of processing layers (optionally with other sorts of layers interleaved with the stack), in which each processing layer but the first is configured to receive the output of the preceding processing layer. The network output is, or more generally is based on, the output of the last processing layer of the stack. That is, the network output may be the combined outputs of the last processing layer of the stack, or the network output may be formed from those outputs by the output unit (if any) of the media element processing system. [00020] Each processing layer may be configured to receive, successively, at successive respective time steps, embeddings of successive segments of the media element. The media element processing system may comprise an embedding unit for generating, from the successive segments of the media element, respective “initial” embeddings of the segments to form the embeddings which are input to a first of the processing layers. Each other processing layer of the stack successively receives a corresponding embedding of each segment from the preceding layer in the stack. For a given segment, each processing layer generates, from the corresponding embedding of that segment which it receives, a corresponding new (and possibly very different) embedding of the segment. The processing layer outputs the new embedding of the segment (e.g., to the next processing layer, if any; or, if there is no next processing layer, to be the network output of the processing sub-system). Thus, the processing sub-system receives at different corresponding times an embedding of each segment of the media element from the embedding unit, repeatedly generates new embeddings of each segment in each of the processing layers in turn (e.g., in successive corresponding time steps), and eventually generates a corresponding output. The corresponding outputs of the processing sub-system for all the segments of the media element, are used to form the network output. [00021] The memory may be provided as part of the processing sub-system. In particular, the memory may contain a plurality of portions, and each processing layer may use (e.g., include) at least one corresponding portion of the memory.

4 37923819-1 [00022] In one possibility, the processing sub-system comprises a respective portion (“memory portion”) for each of the processing layers, and each processing layer may use the respective memory portion to generate successively the new embeddings of the corresponding segments received by the processing layer in successive time steps. The memory control unit may be provided as a respective activation selection unit for each of the processing layers included in the processing sub-system. [00023] In alternative possibility, one or more of the memory portions may each be used by multiple corresponding ones of the processing layers. In a further possibility, one or more of the processing layers may generate new embeddings of the segments using the data from more than one corresponding one of the memory portions. However, for simplicity in much of the following it is assumed that each processing layer uses data stored in only one corresponding memory portion. [00024] The memory control unit may perform the augmentation of the memory by augmenting each of the memory portions for each received segment of the media element. Specifically, for a given memory portion and in each time step, the memory control unit (e.g. an activation selection unit for the corresponding processing layer) may form a memory element by the method discussed above using the current embedding received in the time step by the respective processing layer, and add the memory element to the corresponding memory portion (i.e. to be stored in the memory portion in addition to the data currently stored in the memory portion), for use by the corresponding processing layer in future time steps. [00025] For each given processing layer and in each time step, the received embedding of a current segment of the media element may be used by the processing layer, with the data currently stored in the corresponding memory portion (i.e., the memory elements generated from previously received segments, if any), to generate a new embedding of the current segment of the media element. [00026] Thus an alternative expression of a concept disclosed here is a computer- implemented method of processing a network input comprising a media element by a media element processing system to generate a network output, the media element processing system comprising a processing sub-system comprising an ordered plurality of processing layers, each processing layer being configured to receive a sequence of embeddings of segments of the media element (e.g. at corresponding time steps), and to process the embedding of each current segment of the media element using a corresponding portion of a

5 37923819-1 memory to generate a new embedding of the current segment, the network output being based on embeddings of the segments output by the last processing layer, the method comprising augmenting at least one of the memory portions by: forming a corresponding memory element based on an embedding of the current segment of the media element and having a smaller number of bits than the embedding of the current segment of the media element; and adding the memory element to the memory portion; wherein the processing of the received embedding of each current segment of the media element is by a cross attention mechanism based on the embedding of the current segment of the media element, and a compound data set comprising the previously added memory elements (if any; that is, the memory elements which were formed based on the embedding of any previous segments) and the embedding of the current segment of the media element. Thus, when the current segment is the first segment, the processing is based on a cross-attention of the received embedding of the current segment with itself (i.e. a self- attention), and when the current segment is the second or a later segment, the processing is based on a cross-attention of the embedding of the current segment with a compound data set comprising the one or more previously added memory elements (i.e. based on the embeddings of the one or more previous segments) and the embedding of the current segment. [00027] As discussed above, each embedding may be formed from a certain number of activations/activation components, and the number of numerical values in each activation/activation component is denoted d. In principle, the value of d for a given processing layer and corresponding memory portion (i.e., the number of numerical values in each activation process by that processing layer and memory portion), may be different for different ones of the processing layers. That is, one or more of the processing layers may output embeddings having activations composed of a different number of numerical values than the number of numerical values in each activation of the embeddings it receives. However, the value of d may alternatively be the same for all processing layers and memory portions. [00028] A processing layer may perform a cross attention mechanism between values based on an embedding of the current segment of the media element, and a compound data set comprising data stored in the memory (e.g. the memory portion corresponding to the layer) which was derived from received embeddings of any previous segments of the media element (i.e. any segments before the current segment of the media element), and the

6 37923819-1 embedding of the current segment of the media element. The attention mechanism emphasizes the portion of the current segment which is significant given the context defined by the compound data set. The process may be repeated multiple times for successive current segments of the media element, as the memory is gradually augmented. [00029] An explanation of cross-attention mechanisms is given below. The cross- attention mechanism used in a processing layer of the present system may be a query-key- value cross attention mechanism. The embedding of the current segment may be used as, or more generally to obtain, at least one query vector (typically a plurality of query vectors). A compound data set may be formed which comprises data stored in the memory and the embedding of the current segment of the media element. The compound data set may be used as, or more generally to obtain, one or more key vectors and one or more corresponding value vectors. The output of the attention mechanism may be based on a sum of the value vector(s) weighted by a corresponding weight value, which is based on a similarity measure of the corresponding key vector and the corresponding query vector. Since it based on the compound vector, it incorporates information about the entire context defined by earlier segments of the media element. [00030] The media element processing system may be configured to perform a media element processing task on the network input to generate the network output. In particular, the media element processing system may be configured to perform a multi-modal task on a network input comprising at least one media element, to generate a network output. That is, one or both of the network input and the network output may comprise data of a different modality from the modality of at least one of the media element(s). In an example, discussed below, the media element processing system may generate text tokens based on the network input. Examples of the usage of the present system in this case are given below. [00031] Although the explanation above assumes, for simplicity, that a single media element is contained in each network input, alternatively there may be plurality of media elements in a given network input, for example multiple videos (e.g., if the media element processing task is the multi-modal task of generating text describing differences between the videos), multiple sound signals, or a combination of video(s) and sound signal(s). [00032] Any known embedding unit may be used to form the initial embeddings of the segments of the media element (i.e., the embedding of the segments received by the first processing layer and used by the memory control unit to update the memory portion corresponding to the first processing layer) using the (raw, i.e. unmodified) segments of the media element.

7 37923819-1 [00033] Furthermore, the present disclosure may be defined as a method of defining a latent representation of a media element. The latent representation may be used (e.g., by as output unit, if any) to generate a network output of the media element processing system. [00034] For example, an alternative expression of a concept disclosed here is a method of processing a media element to generate a latent representation of the media element. This method includes (e.g., for a certain processing layer of the processing sub-system) the set of steps: obtaining a first embedding corresponding to a first segment of the media element and comprising first activations; processing the first embedding to generate first latent activations corresponding to the first segment of the media element (i.e., a new embedding of the first segment of the media element); processing the first activations to obtain first consolidated activations (in the terminology used above, this is a first memory element), wherein the number of first consolidated activations is less than the number of first activations (this step may for example be performed by selecting some of the first activations, e.g. randomly or based on the distance metric as described above, as the first consolidated activations, or by forming the first consolidated activations as the centroids of corresponding clusters of activations defined based on the first activations); obtaining a second embedding corresponding to a second segment of the media element and comprising second activations; processing the first consolidated activations and the second embedding to generate second latent activations corresponding to the second segment of the media element, i.e. a new embedding of the second segment of the media element; this is done by applying a cross- attention mechanism between values based on the second activations and combined activations comprising at least the second activations and the first consolidated activations (that is, a compound data set comprising the second activations and the first consolidated activations, i.e. the first memory element);

8 37923819-1 processing the second activations to obtain second consolidated activations (i.e., a second memory element), wherein the number of second consolidated activations is less than the number of second activations; obtaining a third embedding corresponding to a third segment of the media element (and comprising third activations); and processing the first consolidated activations, the second consolidated activations and the third embedding to generate third latent activations corresponding to the third segment of the media element (i.e. a new embedding of the third segment of the media element obtained using the third embedding and the first and second memory elements generated respectively from the first and second segments of the media element). This may be done by applying a cross-attention mechanism between values based on the third activations and combined activations comprising the third activations and the first and second consolidated activations (that is, a compound data set comprising the third activations and the first and second consolidated activations, i.e., the first and second memory elements). [00035] The latent representation of the media element is then generated based on at least the third latent activations. The latent representation may indeed be the third latent activations. Alternatively, it may be generated by processing the first latent activations, second latent activations and third latent activations (and latent activations for any further segments of the media element) successively using one or more further processing layers of the processing subsystem. [00036] For each of the following processing layer(s) of the sequence, if any, the same set of steps is performed. The first latent activations, second latent activations and third latent activations obtained in a given set of steps (i.e., for a given processing layer) are used as the first, second and third embeddings obtained in the next set of steps (i.e. for the next successive processing layer of the sequence). [00037] The present concepts may be expressed as methods. Alternatively, they may be expressed as computer systems which perform the methods, or as computer program products (e.g., tangible recording media) storing program instructions operative to control a computer system to perform the method. BRIEF DESCRIPTION OF THE DRAWINGS

9 37923819-1 [00038] FIG.1 illustrates a media element processing system for generating a network output based on a network input. [00039] FIG.2, which is composed of FIGS.2(a)-2(c), illustrates a principle employed by the system of Fig.1. [00040] FIG.3 shows the steps of a process for augmenting a memory. [00041] FIG.4 shows the steps of a process for generating a latent representation of a media element. [00042] FIG.5 is composed of FIG.5(a) which shows experimental results from a process which is an example of the present disclosure, and FIGS.5(b) and 5(c) which show results from other processes. [00043] FIG.6 is composed of FIGS.6(a) to 6(c), and shows experimental results from a process which is an example of the present disclosure and from other processes. [00044] FIG.7 compares the accuracy of results from three examples of the present disclosure. [00045] FIG.8 is a table of experimental data comparing various algorithms for processing media elements. [00046] FIG.9 shows experimental results from processes which are examples of the present disclosure and from other processes. [00047] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION [00048] An example of the present disclosure will now be described. The example is a media element processing system which comprises a memory. The media element processing system receives segments of the media element successively, embeds each segment, and processes each embedded segment based on a memory. During the processing of each current segment, the memory is augmented by a memory control unit based on the current segment of the media element, by adding to it a memory element based the embedding of the current

10 37923819-1 segment of the media element. The memory element is smaller (that is, it comprises a smaller number of values, i.e., it has a fewer bits, e.g. fewer than 20% of the number of bits, or fewer than 10% the number of bits) than the embedding of the current segment. Thus, the memory accumulates a precis of previous segments of the media element. Since only a precis of earlier segments is accumulated, the size of the memory is limited, and the processing of each current segment is simplified. Thus, the media element processing system can extend its context far into the past. As discussed below, this can give excellent scaling behavior when learning, for example, from media elements which are long videos. [00049] FIG.1 shows an example neural network media element processing system 100. The media element processing system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. [00050] The media element processing system 100 is for processing a network input which is (or which comprises) a media element which may itself comprise (e.g., be) video(s) and/or sound signal(s). For simplicity, the case is considered in which that media element is a video having a number T of frames (images), where T is a positive integer. The media element may be partitioned into a number of segments s, where s is a positive integer, so that each segment is S=T/s frames. In FIG.1, the media element is denoted 1, and is composed of s segments labelled 11, 12,…,1s. Thus, any given segment can be denoted 1^^, where ^^ is an integer variable taking corresponding values (e.g.1,…, s) which label respective segments of the media element. [00051] The media element comprises an embedding unit 101 for forming an embedding of a given segment. The embedding comprises a number of vectors called “activations”. The embedding unit 101 partitions each segment into a number t of time patches (i.e., the interval of the video corresponding to the segment is partitioned into sub- intervals, and each of the time patches corresponds to a respective sub-interval). Thus, each time patch is S/t frames. Each image may be composed of one or more intensity values for _{each of an ^^ ൈ ^^ rectangular array of pixels, where W and H are integers. To generate the} initial embeddings of the segments of the media element, each image may be partitioned into an array of (non-overlapping) spatial patches of substantially equal size. For example, each _{may be ℎ ൈ ^^ rectangular array of pixels, where h and w are integers. For a given segment, a} given activation (media token) may be generated for each time patch and for each of the spatial patches. Thus, the total number of media tokens N per segment is ௌ _{ு ^} ௧ _∙ ^ _∙ ௪.

11 37923819-1 [00052] The activations of the initial embedding may be formed in a manner similar to that described in A. Arnab, et al., “Vivit: A video vision transformer”, in Proceedings of the IEEE/CVR international conference on computer vision, pp 6836-6846, 2021. Each activation is produced by forming a linear projection of the corresponding spatial patch in each frame of the corresponding time patch of the segment. The linear projection may be defined by a learnable projection layer E to which is optionally added a learnt vector p of additional position embeddings. A layer normalization may optionally be performed to the result. Denoting the number of numerical values in each activation by d, E may be a matrix _{with a size ^^ ൈ ^^^ℎ^^^, and p may have d components.} [00053] The activations generated by the embedding unit 101 are processed successively by a processing sub-system 102. The output of the processing sub-system 102 for a given media element 1 is referred to as a latent representation of the media element 1. It is processed by an optional output unit 104 of the media element processing system 100, to generate a network output, such as data informative of the content of the media element 1. Note that in some embodiments the output unit 104 may be omitted (i.e., the network output is the latent representation of the media element 1 generated by the processing sub-system 102), e.g. because the latent representation of the media element is itself of use, e.g. as a compressed version of the media element 1. [00054] The processing sub-system 102 includes a stack (ordered sequence) of L processing layers, where L is an integer which is at least one. The output of each processing layer (except the last of the order sequence) is an input to the next processing layer of the sequence. Each processing layer is labelled by an integer index l, which runs from 1 to L, where L is the number of processing layers. Thus, each processing layer can be thought of a receiving, in successive time steps, embeddings of corresponding ones of the segments 1^^ composed of “activations”, and generating a modified embedding of the corresponding segment also composed of activations. When a single processing layer is being considered below, the activations which the processing layer generates for a given current segment are called “latent activations”, since they are latent values encoding features of the corresponding segment. The latent representation of the media element is composed of the latent activations generated by the last processing layer of the sequence. [00055] The embedding of the current segment received by each given layer of the media element processing system is denoted ^^^{^} ఛ . For a given processing layer l, ^^ has a mapping to the time step in which an embedding of that segment is received. ^^^{^} ఛ is composed

12 37923819-1 of a number of activations (activation components/tokens) denoted by the integer value N, and each activation has d components. [00056] The media element processing system also comprises a memory. The memory has a respective portion 113l for (included in) each processing layer l. The state of the l-th memory portion when the corresponding processing layer receives ^^^{^} ఛ at time step ^^ is denoted ^^^{^} ఛ having a number of activations denoted M. [00057] The memory control unit may be implemented as a respective activation selection unit 111l for (included in) each processing layer l. The activation control unit 111l of the respective processing layer l receives the ^^^{^} ఛ and from it generates a memory element denoted ^^^^{^} ఛ which is added to the corresponding memory portion, to generate an augmented memory portion denoted ^^ ^{^} ఛ_ା^ = [^^^{^} ఛ ; ^^^{^} ఛ ]. where [..;..] denotes concatenation. [00058] The process of augmenting a given memory portion by the activation selection unit 111l is now summarized. For simplicity, the subscript l is omitted, so the received embedding of the current segment is just denoted ^^_ఛ and the current state of the memory portion (i.e., the data currently stored in the memory portion before the augmentation of the memory portion based on ^^_ఛ) is denoted ^^_ఛ. [00059] The memory element based on the current segment 1^^ which is added to the _{memory portion is denoted ^^^^^ which is composed of K activations denoted ^̂^ఛ,^ where ^^ ∈} 1,^^ is an integer variable taking corresponding values which label respective ones of K activations of the memory element. [00060] The K activations ^^^_ఛ may be any subset of the activations ^^_ఛ of the embedding of the current segment received by the processing layer. [00061] For example, it may be a subset selected at random. Thus, ^_{^^^^ ൌ ^^^ఛ,^|^^ ∈ ℐ^ ∈ ℝ^ൈௗ (1)} _{where ℐ ∈ ℕ^ is a set of} _{1 to N.} [00062] Alternatively, the K activations ^^^_ఛ may be chosen as a subset of activations which are maximally far from each other according to a criterion. For example, they may be selected sequentially according to a “greedy coreset” algorithm, by iteratively selecting the most distant activation to the one(s) already selected. One iteration of this algorithm is defined as: ^^^^^^^^^^^^ ^^^^^^ ଶ ^^^∗ ^_{^ ^^ ^^} (2) 37923819-1 ℳ^∗ , which is the set of activations to be chosen, is initially empty, but in each iteration it is _{set to ℳ∗ ∪ ^^^∗^ . Here ^^ ∈ ℝே is the set of indices of all the segment activations in ^^ఛ. The} greedy coreset selection algorithm is run for K iterations to produce the memory element for the segment 1^^. Due to its nature, the greedy coreset selection algorithm becomes increasingly computationally expensive as the size of the memory increases. [00063] Alternatively, the K activations may be formed in another way, e.g., by clustering the N activations ^^_ఛ into K clusters, and choosing the K activations as “cluster activations” which are a mean (centroid) of the activations of respective ones of the clusters. This may be done by initializing K cluster centroids (e.g., at random using Eqn. (1)), and then performing a number (e.g.5) of iterations of k-means clustering to compute updated centroids. [00064] There is now a discussion of how the processing sub-system 102 may be implemented to generate successive embeddings of a current segment using the memory portions 113l. Apart from the activation selection units 111l and the memory portions 113l, the processing sub-system 102 may be any known system for processing a media element using a memory. It, and the output unit 121l if any, may be chosen in view of the media element processing task which the overall media element processing system is to perform. [00065] In one implementation of the processing sub-system 101, each processing layer is a transformer-based processing layer, which generates the network output by a sequence of one or more transformer layers. Each of the transformer layer(s) performs an attention mechanism 103l, such as a multi-head cross-attention (MCA) mechanism. Specifically, one or more of the transformer layer(s) may perform a cross-attention mechanism 103l based on an embedding of a current segment which is input to the transformer layer and the memory elements stored in the corresponding memory portion 113l in respect of previous received segments (if any). The output of the attention mechanism may be added to the input to the processor layer. An information flow path for this possibility is indicated as 107l. [00066] An example of such a system is that described by Dai Z., et al., “Transformer- XL: Attentive language models beyond a fixed-length context”, arXiv: 1901.02860, 2019, which was proposed to implement a language model. _{[00067] A compound data set can be defined which is denoted ^^^^} _{ఛ ൌ ^^^^ ^} _{ఛ ; ^^ఛ ^, where} [..;..] denotes concatenation. The compound data set has a number of activations N+M, denoted ^^^_ఛ,^ where i is an integer index ranging from 1 to N+M. Optionally, a layer-norm (LN) operation may be used to generate the compound data set, thus the compound dataset is _^^^^^^^^^ _{ఛ ^; ^^^^^^^^} _{ఛ ^^.}

14 37923819-1 [00068] The output of the attention mechanism is denoted by ^^_ఛ ^{^} given by: (3) where ^{^}∙;∙^{^} again denotes a a head cross attention and LN denotes a layer-norm operation. As discussed below, here ^^^^^^^^{^} ఛ ^ is used to produce the query vectors by, for each head, multiplying it by a corresponding query _{matrix ^^ொ,^, and ^^^^^^^^^} _{ఛ ^; ^^^^^^^^} _{ఛ ^^ is used to produce both the key vectors and the value} vectors by multiplying it by corresponding key and value matrices ^^^{^,^}, ^^^{^,^} . This allows the activations ^^^{^} ఛ of the current segment (i.e. the ^^-th segment) to attend both to themselves, and to memories ^^^{^} ఛ of past segments. The quadratic complexity is limited to N+M. The corresponding activations ^^^{^} ఛ are added to ^^_ఛ ^{^} (separately for each activation), as shown by the information path 107l. A normalization operation may be performed on the result. [00069] Note that ^^^{^} ఛ (rather than ^^ ^{^} ఛ_ା^ ) is used by unit 103l, for processing ^^^{^} ఛ to generate the processing layer’s latent activations for the segment. Thus, the operation of the activation selection unit 111l for a given segment may be performed in parallel with, or before or after, the operation of the unit 103l to process ^^^{^} ఛ . [00070] One or more of the transformer layers may process the output of the attention mechanism using a corresponding adaptive system of the transformer layer such as a multi- layer perception (MLP) 105l. Optionally, the input to the MLP may be added to the output of the multilayer perceptron. An information flow path for this possibility is indicated as 109l. [00071] Thus, the operation of the MLP 105l and information path 109l may be given by: ^_{^^ା^ ^ ^ ^ ^} _{ఛ ൌ ^^^^^^^^^^^ ^^ఛ ^ ^ ^^ఛ . (4)} This provides an output for of the segment. The memory augmentation process for the next layer (if any) may be performed using the new embedding, as described above. Note that in the case of l=L, ^^^{^} ఛ^{ା^} denotes the output of the last processing layer of the processing sub-system 102, which is the input to the output unit 104 (if any), or the network output otherwise. [00072] Thus, the overall algorithm performed by the media element processing system 100 may be represented by the following pseudo-code:

15 37923819-1

[00073 m 102, one or more, or all, of the processing layers may not employ an attention mechanism 103l. For example, in such a processing layer a clustering operation, such as a k-means operation, may be performed on the data in the corresponding memory portion, to form groups (“pools”) of remembered activations with similar features. The k-means operation can enforce mutual exclusivity of the activations. The processing layer may be configured to generate an output for the processing layer by processing the embedding it received for the current segment, and the output of the clustering operation, e.g. to generate an output based on one of the pools which is determined to be most similar to the received embedding of the current segment (e.g. according to a pre-determined similarity measure, such as a Euclidean distance). [00074] Turning to FIG.2, the media processing system 100 is contrasted with two comparative examples. [00075] FIG.2(a) shows the attention mechanism of a single processing layer of a comparative example media element processing system (“Streaming ViT”) which has the same structure as the media element processing system 100 of FIG.1, but which omits the activation selection unit 111l and the memory portion 113l. Reference numeral 21 denotes a plurality of activations (as illustrated, four activations) received by the attention mechanism for the first segment. The attention mechanism processes these by self-attention (shown by solid lines). [00076] FIG. 2(b) shows the attention mechanism of a single processing layer of a comparative example media element processing system (“memory augments ViT”) which has

16 37923819-1 the same structure as the media element processing system 100 of FIG.1, but which omits the activation selection unit 111l. The attention mechanism processes the activations 21 of the first segment in the same way as the example of FIG.2(a), but also places all the activations 21 into a memory portion 113l. Reference numeral 22 denotes a plurality of activations (as illustrated, four activations) received by the attention mechanism for the second segment. The attention mechanism processes these by cross-attention (shown by dashed lines) using the activations 21 for the first segment stored in the memory portion 113l. The activations 22 are added to the memory, so that in processing an embedding of a third segment, the attention mechanism will do this by cross-attention to both the activations 21 and the activations 22. While this is successful for a video having a small number of segments, as the number of segments which have been received increases, the required size of the memory and the number of parameters of the attention mechanism become large, so that no more than a few segments can be handled in this way. [00077] By contrast, as shown in Fig.2(c), in an example of the present disclosure the attention mechanism 103l of a single processing layer of the media element processing system 100 shown in Fig.1, processes the received activations 21 by self-attention (as in Figs.2(a) and 2(b), and places only a selection of them into the memory portion 201 for the processing layer. Upon receiving the activations 22 of the second segment, the attention mechanism 103l processes the activations 22 by cross attention, referencing the selected ones of the activations 21; it also places a selection of the activations 22 into the memory portion 201. Upon receiving the activations 23 of the third segment, the attention mechanism 103l processes the activations 22 by cross attention, referencing the selected ones of the activations 21 and 22; it also places a selection of the activations 23 into the memory portion 201. Upon receiving the activations 24 of the fourth segment, the attention mechanism 103l processes the activations 22 by cross attention, referencing the selected ones of the activations 21, 22, 23 stored in the memory 201; it also places a selection of the activations 24 into the memory portion 201. [00078] This example of the present disclosure is here referred to as “memory- consolidated ViT” (or “MC-ViT”). Compared to the comparative example of Fig.2(a), for a given size of the memory portion 201 the attention mechanism 103l can use data from segments further into the past to process a current segment, and do so using fewer trainable parameters. [00079] FIG.3 shows an exemplary method 300 according to the disclosure. It is an example of a method which can be performed by one or more computers in one or more

17 37923819-1 locations. For example, a processing layer of a media element processing system 100 as shown in FIG.1 can perform the method 300. [00080] In step 301, from a received embedding of a current segment comprising a plurality of activations, a memory element having a small number of bits is formed. This may be done by the activation selection unit 111l of the processing layer, such as by any of the methods described above with reference to FIG.1. Whereas the received embedding may include N activations (of a given size), the memory element may include only K activations (of the same size) where K is less than N. [00081] Optionally, this may be performed by processing layer selecting a subset of the activations of the received embedding of the current segment. If so, the activation selection unit 111l forms a memory element based on the selected activations. The memory element may simply be a concatenation of the selected activations. It has a smaller size (e.g., a smaller number of bits) than the received embedding. [00082] Alternatively, if a clustering method is used as explained above, then no selection of a subset of the activations of the received embedding of the current segment may be needed. That is, all the activations of the received embedding may be allocated to a corresponding one of K clusters, and the memory element may be defined having K activations based on the respective centroids of the clusters. [00083] In step 302, the memory element is added to the memory portion 113l. [00084] Thus, by repeatedly performing the method 300 in respect of successive segments, the memory portion 113l is gradually augmented. [00085] As mentioned, the output of a given processing layer for a given current segment can be considered a latent encoding of the current segment of the media element. The output is a set of activations which can be called “latent activations”. FIG.4 shows an exemplary method 400 according to the disclosure. It is an example of a method which can be performed by one or more computers in one or more locations. For example, a processing layer of a media element processing system 100 as shown in FIG. 1 can perform the method 400. The method 400 comprises processing received embeddings of three successive segments. Each of a plurality of processing layers of the media processing system 101 can perform respective instances of the method 400, so that embeddings of the three segments generated successively by the embedding unit 101 are successively refined, layer-by-layer, to generate respective sets of latent activations, which are passed to the output unit 104 (if any) or used as the network output.

18 37923819-1 [00086] Step 401 comprises obtaining a first embedding corresponding to a first segment of a media element. It may for example be received as the output of a preceding one of the processing layers (if any), or, in the case of the first processing layer, it is the output of the embedding unit 101. The plurality of activations of the first embedding are referred to as “first activations”. [00087] Step 402 is processing the first embedding to generate an output from the processing layer. The output, referred to as a plurality of “first latent activations”, is an updated embedding of the first segment of the media element. In the case of the media element processing system 100 of Fig.1, the first latent activations may be as defined by Eqns. (3) and Eqn. (4) above, though the memory portion ^^^{^} ఛ is zero at this stage (e.g., ^^_^ ^{^} is a default value). [00088] In step 403, which may be performed in parallel with step 402 or before or after it, the first activations are also processed to obtain “first consolidated activations”. This is by selecting certain ones of the first activations. Thus, the number of first consolidated activations is less than the number of first activations. In total the first consolidated activations have a smaller number of bits than the total number of bits of the first activations, e.g., fewer than 20% of the number of bits, or fewer than 10% the number of bits. [00089] In step 404, the processing layer receives a second embedding corresponding to a second segment of a media element. It may for example be received as the output of a preceding one of the processing layers (if any), or, in the case of the first processing layer, it is the output of the embedding unit 101. The plurality of activations of the second embedding are referred to as “second activations”. [00090] In step 405, the second embedding and the first consolidated activations are processed to generate an output from the processing layer. The output, referred to as a plurality of “second latent activations”, is an updated embedding of the second segment of the media element. In the case of the media element processing system 100 of Fig. 1, the second latent activations may be as defined by Eqns. (3) and (4) above. [00091] In step 406, which may be performed before or after step 405 or in parallel with it, the second activations are also processed to obtain “second consolidated activations”. This is by selecting certain ones of the second activations. Thus, the number of second consolidated activations is less than the number of second activations. In total the second consolidated activations have a smaller number of bits than the total number of bits of the second activations, e.g., fewer than 20% of the number of bits, or fewer than 10% the number of bits [00092] In step 407, the processing layer receives a third embedding corresponding to a third segment of a media element. It may for example be received as the output of a preceding

19 37923819-1 one of the processing layers (if any), or, in the case of the first processing layer, it is the output of the embedding unit 101. The plurality of activations of the third embedding are referred to as “third activations”. [00093] In Step 408, the third embedding and the first and second consolidated activations are processed to generate an output of the processing layer. The output, referred to as a plurality of “third latent activations”, is an updated embedding of the third segment of the media element. In the case of the media element processing system 100 of Fig. 1, the third latent activations may be as defined by Eqns. (3) and (4) above. [00094] In step 409, a network output is generated based on the third latent embeddings, and optionally also the first and/or second latent embeddings. For example, step 409 may be performed by the output unit 104 (if any). If no output unit 104 is present, and if the processing layer which performs method 400 is the last processing layer of the stack, then the network output may be the third latent embeddings, optionally also including the first and/or second latent embeddings. [00095] The operation of the media element processing system 100 is defined by a plurality of trainable numerical parameters. These may comprise numerical parameters defining the operation of each processing layer, e.g., the function which each processing layer performs on a received embedding of a segment and the data stored in the memory (e.g., in the corresponding memory portion), to generate the new embedding of the segment. For example, if the processing layer is a transformer layer which implements an attention mechanism 103l, e.g., followed by processing using a MLP (multi-layer perceptron) 105l, the numerical parameters defining the operation of the processing layer may include key, query and/or value matrices defining the attention mechanism and/or numerical parameters defining the MLP. [00096] Furthermore, the trainable numerical parameters defining the media element processing system 100 may comprise numerical parameters defining the operation of the embedding unit 101 and/or the output unit 104. [00097] Some or all of the trainable numerical parameters defining the media element processing system 100 may be trained (e.g., before the media processing system 100 is used to process new network inputs), e.g., based on a training database of example network inputs and corresponding target network outputs. A loss function may be defined comprising a loss term indicative of a measure of the difference between the outputs of the media element processing system upon receiving the network inputs, and the target network outputs. The numerical parameters defining the media element processing system may be updated to

20 37923819-1 reduce the value of this loss function. The updates may be performed by any known algorithm (e.g., a backpropagation algorithm), and may be performed jointly (i.e., updates to different ones of the numerical parameters defining the media element processing system may be performed substantially simultaneously, or may be interleaved with each other). [00098] Note that the training database of example network inputs and target network outputs may not be fixed. For example, the training may use contrastive learning, in which the media element processing system is trained jointly with a text encoding network (i.e., with updates to the media element processing system being simultaneous with, or interleaved with, updates to the text encoding network), using a training database of media elements and corresponding text items. The training is such as to increase the similarity between an output of the media processing system upon receiving and processing one of the media elements of the training database, and an output of the text encoding network upon receiving the corresponding text item. Alternatively, the media element processing system may play the role of the text encoding network, i.e., such that is trained to increase the similarity between a latent representation which it generates of a media element and a latent representation it generates of a text item which it receives. [00099] In some examples, one or more components of the media element processing system may be pre-trained, prior to the training procedure described above. For example, the processing sub-system may be one which has been pre-trained without using a memory, or using a memory of a different sort than that disclosed herein. The processing sub-system may be fine-tuned during the training procedure within the media element processing system as described above. [000100] Note that the training procedure may be carried out (e.g., by a first party) before the trained media element processing system is used (e.g., by a second party) to process media elements. Thus, potentially confidential information contained in the media elements need not be made available to the first party. [000101] A number of experimental results will now be described with reference to FIGS.5 to 9. Examples of the system of FIG.1 are referred to as “MC-ViT”, and the experiments implemented the activation selection unit using the clustering method described above. The parameters of the processing network were initialized using those of an example of the ViViT system (Arnab et al., 2021, referenced above) based on short (16 frame) video clips using contrastive learning (Xu H. et al, “VideoCLIP: |Contrastive pre-training for zero- short video-text understanding”, 2021). The media element processing system was then fine- tuned using datasets composed of video-text pairs, with multimodal contrastive learning. The

21 37923819-1 training was done using a loss function which, for each training example (denoted i), included a respective loss term: in the batch. [000102] The evaluation was carried out using the zero-shot transfer paradigm from CLIP (Radford A. et al., “Learning transferable visual models from natural language supervision”, 2021) to perform all downstream tasks. In all cases, a test video is provided with multiple possible captions, only one of which was correct. For action recognition, the captions are class names (i.e., the media element processing system is configured to choose one of the predetermined classes to assign a given media element to). For video question answering, captions are question-answer pairs constructed from a set of multiple-choice answers. The media element processing system was used to compute caption embedding ^^_^ ^௧௫௧, and this was compared with a video embeddings ^^_^ ^{௩^ௗ}. The model’s prediction is the caption i for which ^^_௩^ௗ _{^ . ^^^} ^௧௫௧ was highest. [000103] FIG.5 shows the accuracy for the exemplary system of FIG.1 (Fig.5(a). The lines show the number of frames used for the fine tuning (16, 32, 54 or 128 frames). Fig.5(b) shows comparable results from a comparative example using a joint space-time attention mechanism which does not perform steaming (i.e., processing of segments individually). This system could not be employed for videos longer than 128 frames, due to its large complexity and memory footprint. Fig.5(c) shows processing by a comparative example using streaming but no memory (ST-ViT, as in Fig.2(a)). It can be used for longer videos but does not benefit from training on longer videos and has low accuracy. [000104] FIG.6 compares the accuracy of the exemplary system of FIG.1 with four comparative examples, as a function of the number of test frames (FIG.6(a)), the peak memory footprint (FIG.6(b)) and total FLOPS (Fig.6(c)). The exemplary system of FIG.1 corresponds to the lines marked 6. The comparative examples respectively employ joint- space time attention, joint space time attention with masking, ST-ViT, and factorized attention. The exemplary system can be seen to have the highest performance, and for example reached the same performance as joint space-time attention with 10 times less memory and FLOPs.

22 37923819-1 [000105] FIG.7 shows how the example of the present disclosure makes efficient use of finite-length context. The three lines show implementations of the activation selection unit using the activation selection options described above: clustering using K-means (line 71), coreset (line 72) and random (line 73), with different number of test memories per segment. For comparison, accuracy results are given for a joint space-time attention model, ST-ViT and MeMViT (C.-Y. Wu, et al., “MeMViT: Memory augmented multiscale vision transformer for efficient long-term video recognition”, 2022), which is shown as the point with an accuracy 86 at 128 text memories. MeMViT employs a parametric approach to memory compression, requiring a convolutional module to be trained alongside the network. The K-means and coreset options of MC-ViT surpass all other methods at a 16x compression rate with 128 activations per segment. Even using random activation selection, MC-ViT outperforms all baselines at 4x compression with 512 activations. [000106] FIG.8 is a table showing fine-grained action classification for the standard problem “Diving48”. The examples of the present disclosure are shown as “MC-ViT-B” (with 200M parameters) and “MC-Vit-L” (with 313M parameters). Results for 9 other known algorithms are also given. The second column of the table shows the number of parameters used, and the third column shown the number of frames which can be used. Performance is calculated as a percentage correct on multiple-choice video question answering using evaluation standards called EgoSchema (“EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding”, K. Mangalam et al, 2023, arXiv:2308.09126), Perception Test (“Perception Test: A Diagnostic Benchmark for Multimodal Video Models”, V. Patraucean et al., 2023, arXiv:2305.13786) and Next-QA (“NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions”, J. Xaio et al, 2021, arXiv:2105.08276). It will be seen that the examples of the present disclosure are almost always the most accurate algorithms out of those listed, despite having fewer parameters. [000107] This is summarized by Fig.9 which shows, along the two axes, the video question-and-answer (VQA) accuracy for EgoSchema (a long-form multiple-choice video question answering dataset) and Perception Test (a question answering dataset inspired by assessment in developmental psychology and which evaluates a model’s grasp of physics, reasoning, memory and semantic extraction). The two examples of the present disclosure (MC-ViT-B and MC-ViT-L) are compared with seven other known algorithms, demonstrating higher performance, despite having fewer parameters. [000108] The subject matter described in this specification can be implemented in particular embodiments so as to realize the advantage that information can be extracted from

23 37923819-1 media elements with high accuracy despite the technique using fewer trainable parameters, and thus using less computer resources (memory and computing operations). [000109] As has been demonstrated experimentally, an implementation of the present system can perform media element processing tasks with extremely high success rate compared to existing methods, especially on media elements which are long videos. Quantitative results have been obtained on videos of hundreds of frames. Furthermore, success has been found when studying longer videos, e.g., videos spanning a quarter of an hour, or even longer. [000110] This is achieved with a number of trained numerical parameters which is lower (e.g., two orders of magnitude lower) than in some existing systems, even ones which are not as successful at performing the media element processing tasks. Furthermore, the computational burden in operating the trained media element processing system is much lower, e.g., over an order of magnitude lower, than in some known systems. [000111] There is now a discussion of some of the uses of the media element processing system disclosed herein. [000112] As mentioned above, the media element processing system may be a neural network which performs a multi-modal task on a network input comprising at least one media element, to generate a network output, where one or both of the network input and the network output comprise data of a different modality from the modality of at least one of the media element(s). [000113] For example, the media element processing system (e.g. the system depicted in FIG.1) may implement a multi-modal language model, “MMLM” (e.g., a visual language model, “VLM”). The multi-modal language model may be configured to generate the network output by selecting one or more text tokens from a vocabulary of text tokens. The network output comprises (or consists of) a token output comprising the selected text tokens, and representing a text response to the network input. The updates to the numerical parameters defining MMLM during the training described above may such as to reduce the value of a multi-modal loss function for the multi-modal task that quantifies a difference (e.g. for a batch of example network inputs in the training database) between the token output which is generated by the MMLM upon receiving a training network input comprising a media element, and corresponding target token output(s) (target network outputs) in the training database. For example, the multi-modal loss function may measure respective likelihoods assigned to text tokens of the target token output by score distributions over the

24 37923819-1 text tokens in the vocabulary of text tokens generated by the trained language model neural network as a result of processing the training network input. [000114] Alternatively or additionally, the network input may comprise, in addition to the media element(s), a token input comprising one or more text tokens from a vocabulary of text tokens (e.g. the same vocabulary of text tokens). The text tokens of the token input are typically selected from the same vocabulary as the text tokens of the token output, though in some implementations the vocabularies may be different (e.g. the input text tokens may be selected from the Roman alphabet, and the output tokens may be selected from a Japanese character set, such that the MMLM performs a translation operation, conditioned on the media element(s) of the network input). The tokens in the vocabulary/vocabularies can be any appropriate text tokens, e.g., words, word pieces, punctuation marks, and so on that represent elements of text in one or more natural languages and, optionally, numbers and other text symbols that are found in a corpus of text. For example, to generate the input text tokens the system can tokenize a given sequence of words by applying a tokenizer, e.g., the SentencePiece tokenizer or another tokenizer, to divide the sequence into tokens from the vocabulary. The vocabulary(-ies) may also include pre-defined tokens labelling positions in a text portion of the network input, e.g., the input text tokens may include a text token indicating the start/end of a phrase formed by other input text tokens, or indicating that a given input text token is the last input text token of the network input. Note that the token input of the network input, or an embedding thereof, may be provided to, and processed by, any one (or more) of the embedding unit 101, the processing sub-system 102 or the output unit 104. [000115] Note that if the network input contains no input text tokens (e.g., it just contains media element(s)), then the multi-modal neural network may be configured to generate the token output based on the media element(s) alone, e.g., as a textual description of the content of the media element(s). [000116] In some implementations, in which the media element(s) of each network input are sound signal(s), the multi-modal task may be to obtain information, of a type defined by input text tokens of the network input, from the sound signal(s) of the network input. For example, the input text tokens may define the multi-modal task as determining the topic of conversation in the sound signal, or determining a regional accent or other characteristics a speaker in the sound signal. [000117] In some implementations, in which the media elements are videos, the multi- modal processing task is a video captioning task. The token output for each network input

25 37923819-1 may represent at least a portion of at least one predicted text caption for the input image(s) of the input network input. The caption(s) describe action(s) and/or object(s) (people, animals, or inanimate objects) depicted in the videos, and thus the system constitutes an action/object recognition system. Especially if, as discussed below, the caption(s) relate to respective sub- sections (portions) of the media element, the caption(s) can be used to recognize at least one portion of the media item containing an action and/or object, and a further processing system (e.g., a VLM) may be used to perform further processing (e.g., more in-depth processing) on the recognized portion(s) of the media item. [000118] One such implementation may for example be as shown in Fig.1, with the output unit 104 generating a network output 15 indicating one or more captions for a media element which is at least one video. The output unit 104 generates the network output 15 based on at least one of the outputs of the processing sub-system 102 upon the system processing the respective segments of the at least one input video. [000119] For example, the output unit 104 may generate the network output 15 based on the respective outputs of the processing sub-system 102 upon processing the respective embeddings of one or more of the last segments of the at least one input video; in this case, the caption may indicate content (e.g., objects and/or actions) depicted in any frames of the media element, e.g., content which is present throughout the media element. [000120] Alternatively, the output unit 104 may generate the network output 15 based on the respective outputs of the processing sub-system 102 upon processing the respective embeddings of a portion of the input video having a particular characteristic, e.g., a portion of the input video which depicts a certain action and/or a certain object. The network output 15 may indicate which portion of the input video the caption describes. [000121] Alternatively, the output unit 104 may generate the network output including a portion for each of a plurality of corresponding portions of the media element (e.g., with each portion comprising one or more of the segments). Each portion of the network output specifies a respective caption indicative of content (e.g., objects and/or actions) depicted in one or more frames of the corresponding portion of the media element; in this case, the output unit 104 may generate the respective portion of the network output 15 for each portion of the media element based on the respective outputs of the processing sub-system 102 upon processing the respective embeddings of one or more of the segments of the corresponding portion of the input video (e.g. one or more last segments of the portion of the at least one input video), and each network output may indicate (e.g. be) the caption of the corresponding portion of the media element.

26 37923819-1 [000122] In one case, the output unit 104 may have access to a set of pre-defined captions. Based on the at least one of the outputs of the processing sub-system 102 upon the system processing the segments of the at least one input video, the output unit 110 may determine a respective score for each of the pre-defined captions, and select one of the captions based on the respective scores, e.g., as the pre-defined caption for which the respective score is highest. The output unit 104 may generate the scores based also on a token input the media processing system 100 (e.g., the output unit 104) receives, e.g., an instruction specifying a property of the caption (e.g. that they are in Spanish). [000123] In another case, the output unit 104 may include a language model (large language model) such as an auto-regressive language model, so that the media element processing system 100 as a whole implements a VLM. The language model may be conditioned on the at least one of the respective outputs of the processing sub-system 102 upon the system processing the segments of the at least one input video. It may be configured to generate the caption(s) as the token output of the language model. The language model may generate text tokens of the token output successively, based on any text tokens the language model has previously generated, and optionally based on a token input (e.g., specifying a property of the captions). The token output defines the caption(s) generated by the output unit 104. [000124] In some implementations, each network input comprises one or more input text tokens which represent a question about a media element contained in the network input, and the token output represents a predicted text answer to the question about the input image. [000125] In some implementations, the network input comprises: a plurality of media elements; and one or more input text tokens which represent a classification of a (proper) subset of the media elements of the network input (i.e., an indication that the content of the subset of media elements is in a corresponding one of a set of predetermined categories). The token output may be a predicted classification for a media element of the network input other than the subset of the media elements. For example, the network input may comprise a video or sound recording of a cat, and a video or sound recording of a dog, and input text tokens “cat”. In this case, the token output may comprise text tokens spelling “dog”. [000126] In some implementations, each media element comprises a video, and one or more input text tokens that represent a question when asked about the video, and the token output represents a predicted answer to the question. For example, the media element may comprise a video of a cat eating successive items, and input text tokens may be “what is the

27 37923819-1 third thing the cat eats”, and the token output may comprise text tokens indicating a third thing the cat eats in the video. [000127] In some implementations, each media element comprises a video, and one or more input text tokens that represent an answer to a question when asked about a (proper) subset of the images of the video, and the token output represents a predicted answer to the question when asked about an input image of the network input other than the subset of the input images. For example, the media element may comprise a video having a first portion which depicts a cat and a second portion which depicts and dog. The input text tokens corresponding to segments of the video depicting the cat may be “miaow”, and the token output for a second portion of the video may comprise the text tokens “woof”. [000128] A trained media element processing system as disclosed herein has a number of technical applications. [000129] For example, in the case that the media element(s) are sound signals, the sound signal(s) of a network input may be sound signal(s) captured by a microphone, and the token output may be text describing the content of the sound signal(s). For example, a characteristic of the speaker and/or the content of speech encoded by the sound signal(s). [000130] Alternatively, in the case that the media element(s) are videos, the video may be a video of the real world captured by a camera, and the token output may be text describing the real world. In particular, the token output may answer a question about the real world posed by text included in the network input. Thus, one technical application of the system is to be used by a visually-impaired user to obtain information about the real world based on the output of a video camera. For example, the media element processing system may be part of a computer system which obtains a video (e.g., using a camera of the computer system). The user can supply input text tokens (e.g., by controlling a data input device, or by speaking a command which is converted into text by the computer system, e.g. a string of letters or tokens representing phonemes). The media element processing system, upon receiving a network input comprising the video and the input text tokens, can then generate a token output, e.g., as a string of tokens which are converted into a sound signal which is broadcast to the user by a speaker device. A useful feature of this computer system is its flexibility, since the user may freely define the information which the media element processing system should extract from the image(s). [000131] Optionally, the computer system may be configured to allow a user (a person) to generate a series of multiple network inputs, each including the same media element(s) but with different respective text. For example, the first network input may be a video and the

28 37923819-1 text of the first network input may be “What is shown in the video?”. If the media element processing system generates, in response to the network input, a corresponding token output (e.g., “A bus”), then the person may be able to define a second network input including the same video and with new text (e.g., “What destination is on the front of the bus?”). [000132] Optionally, each network input in the series of network inputs, except the first network input of the series, may include the input text tokens and the token output from one or more of the previous network inputs in the series. In this way, the person may engage in a coherent discussion with the media element processing system about the media element, in which at each stage the media element processing system generates a token output which is a sensible continuation of the earlier part of the discussion. [000133] Another technical application of the media element processing system is a captioning system for a video. The text of the network input may define a video processing task such as “provide a caption explaining what is happening”, and in this case the token output may comprise a caption describing events or actions depicted in the video. [000134] In another technical application, a video included in the network input comprises text, and the media element processing system might comprise an OCR (optical character recognition) system for extracting the text from the video and from it generating text tokens to be included in the network input. For example, if the video depicts a scene comprising a sign with instructions (e.g., the video may be images of a road including a road sign including text), the OCR may generate the text of the input text tokens of the network input based on the instructions in the image. In this case, the token output may indicate the significance of the instructions, given the rest of the scene (e.g., in the case of a video which is an image of an empty parking bay and a sign indicating that parking is allowable at the present time, the token output may be “Please park here”). [000135] In another technical application, the media element processing system may be configured, or be instructed by text tokens in the network input, to generate, as the token output, code which is executable by a processor of a computer system, and which is conditioned on the media element(s) in the network input. For example, the network input may comprise a video and input text tokens including text which reads “Generate HTML code which when executed generates a moving image resembling the object shown in the video”. [000136] In another technical application, in a case that that the media element is a video, the media element processing system may analyze a sequence of video frames to detect objects in the video frames and provide information relating to the detected objects in

29 37923819-1 response to a question defined by the input token string. The questions may comprise, for example, a request for a prediction of a future event or state relating to one or more of the objects (e.g. “will objects X and Y collide?”), or a request for conditional or counterfactual information relating to one or more of the objects (e.g. “what event would [not] happen if object X is modified, moved or absent?”), or a request for analysis of the video frames to determine a property or characteristic of one or more of the objects (e.g. “how many objects of type Z are moving?”). [000137] The response to the question given by the token output may, for example, be in the form of a yes/no answer, or may define a probability distribution over a set of possible answers; or the response may define the location of an object. Implementations of the system are flexible in the types of question that can be defined, and a single implementation of the system can handle multiple different types of question. Merely by way of example, the system can be used to predict whether or not two objects will collide, or how this may be avoided. The response to the query may be used by a human or computer system in many ways. For example, the analysis may be useful in itself, or it may be used to provide a warning and/or to control motion of one or more of the objects. [000138] In another application, the media element processing system may generate output data that categorizes the content of the media element into one or more of a plurality of categories, e.g., by defining a score for each category of a plurality of possible categories for the data items. As another particular application, the media element processing system may perform a multi-modal task which is to provide output data that defines (e.g., as a score) whether the content of the media element is described by text comprised in text tokens of the network input. [000139] The media element may be one (e.g., a video) having segments which are captured, or otherwise obtained, while the media element processing system is processing earlier portions of the media element. For example, in one application, the media element processing system may be configured to guide a human user to perform a task based on a media element which is a video showing an environment containing the user, or with which the user interacts. As the user is performing the task, and media element processing system generates outputs (e.g., token outputs) which are instructions (which may be converted into aural instructions and broadcast to the user as sound, or displayed on a screen) to perform the task. For example, the task may be a task (e.g. a cooking task) in which the user, who acts as an agent of the media processing system, should take actions (e.g. add a certain amount of an ingredient to a dish being prepared) which depend on actions the user has taken, or other

30 37923819-1 events in the environment, earlier (e.g. how much of another ingredient has already been added to the dish earlier, as shown by earlier segments of the video). For example, the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real- world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g., a monitoring system such as a video camera or sound capture system, to capture visual observations of the user performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task. [000140] The term “attention mechanism” as used in this document refers to a process which determines a relationship between two ordered data sets (sequences). When the data sets are different, this is called a cross-attention mechanism; a self-attention mechanism is configured to relate different positions in the same ordered data set to determine a transformed version of the ordered dataset as an output. An “attention layer”, also called here a “transformer layer”, is a processing layer which performs an attention mechanism, and optionally other operations also. [000141] For example, the transformer layer input may comprise a vector for each element of the input sequence. These vectors provide an input to the self-attention mechanism and are used by the self-attention mechanism to determine a new representation of the same sequence for the transformer layer output, which similarly comprises a vector for each element of the input sequence. An output of the self-attention mechanism may be used as the transformer layer output, or it may be processed by one or more feed-forward layers (a multi-layer perceptron), and optionally other operations (e.g., normalization operations), to provide the layer output. [000142] In some implementations the attention mechanism is a query-key-value self- attention mechanism configured to apply each of a query transformation e.g. defined by a “query” matrix ^^^ொ, a key transformation e.g. defined by a “key” matrix ^^^{^}, and a value transformation e.g. defined by a “value” matrix ^^^{^}, to the transformer layer input for each element denoted by a vector x of the input sequence X (where X is a matrix in which each row is one of the elements x of the sequence; note that the number of rows of X may be limited to a value N+1, so that the output of the attention mechanism for a given input x may only be based on that input x and the N preceding inputs) to derive a respective query vector _{matrix (formed of query vectors) ^^ ൌ ^^^^ொ, a respective key vector matrix (formed of key}

31 37923819-1 _{vectors) ^^ ൌ ^^^^^, and a respective value vector matrix (formed of value vectors) ^^ ൌ} ^^^^^{^}, which are used determine an attended sequence for the output. For example, the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine respective weights for each value vector, then combining the value vectors using the respective weights to determine the transformer layer output for each element of the input sequence. The transformer layer output may be scaled by a scaling factor e.g., by the square root of the dimensions of the queries and keys, to implement scaled dot product attention. Thus, for example, an output of the attention _{mechanism may be determined as ^^^^^^^^^^^^^^ ^} ^^^^^{^} √^ௗ ^V where d is a dimension of the key (and vectors included in V is _{assumed here, weighted by the respective values ^^^^^^^^^^^^^^ ^} √ௗ ^. In the case that the attention mechanism is not a self-attention mechanism, transformer layer input x is applied to another data matrix Y having rows composed vector elements y, the key matrix and value matrix are multiplied by Y rather than X, to give the corresponding matrix of key vectors and value vectors. In another implementation the attention mechanism may comprise an “additive attention” mechanism that computes the compatibility function using a feed-forward network with a hidden layer. As previously mentioned, output of the attention mechanism may be further processed by one or more fully-connected, feed forward neural network layers. [000143] The attention mechanism may implement multi-head attention. That is, it may apply multiple different attention mechanisms in parallel. The outputs of these may then be combined, e.g., concatenated, with a learned linear transformation applied to reduce to the original dimensionality if necessary. [000144] Examples of suitable architectures for the transformer neural network subsystem are described in Ashish Vaswani et al., “Attention is all you need”, Advances in Neural Information Processing Systems, pp.5998–6008, 2017; arXiv:1810.04805 Devlin et al. (BERT); and arXiv:1901.02860 Dai et al. (Transformer-XL). [000145] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with

32 37923819-1 new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions. [000146] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure. [000147] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized

33 37923819-1 programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics. [000148] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics. [000149] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors. [000150] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent

34 37923819-1 execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases. [000151] Computers capable of executing a computer program can be based on general- purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The essential elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the AI model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage. [000152] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid- state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

35 37923819-1 [000153] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction. [000154] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models. [000155] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience. [000156] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of

36 37923819-1 network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP/IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities. [000157] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. [000158] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. [000159] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one

37 37923819-1 example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. What is claimed is:

38 37923819-1

Claims

WHAT IS CLAIMED IS: 1. A method of augmenting a memory of a media element processing system, the media element processing system being configured to process a network input comprising a media element to generate a network output, and the media element processing system comprising the memory and at least one processing layer, the processing layer being configured to receive a sequence of embeddings of corresponding segments of the media element at corresponding time steps, and to process the received embedding of each current segment of the media element using the memory, to generate a new embedding of the current segment, the method comprising augmenting the memory, for each current segment of the media element, by: forming a memory element based on an embedding of the current segment of the media element, the memory element having a smaller size than the embedding of the current segment; and adding the memory element to the memory. 2. The method of claim 1 in which the processing of the received embedding of each current segment of the media element is by a cross attention mechanism between values based on an embedding of the current segment of the media element, and a compound data set comprising data stored in the memory and the embedding of the current segment of the media element. 3. The method of claim 2 in which the cross-attention mechanism uses a query vector based on the current embedding. 4. The method of claim 2 or 3, in which the cross-attention mechanism uses a key vector based on the compound data set. 5. The method of any preceding claim in which the cross-attention mechanism uses a value vector based on the compound data set.

39 37923819-1

6. The method of any preceding claim in which wherein the embedding of the current segment comprises a plurality of activations, and the forming of the memory element comprises: selecting a subset of the plurality of activations of the embedding of the current segment, and forming, for each selected activation of the embedding of the current segment, a corresponding activation of the memory element 7. The method of claim 6 in which the subset of the activations is selected randomly. 8. The method of claim 6, in which the subset of the activations are selected successively from the activations of the embedding of the current segment, each of the subset of activations except the first being selected as the one of the previously unselected components of the current segment which is furthest, according to a distance measure, from the closest one of the previously selected activations. 9. The method of any of claims 1 to 5, in which the embedding of the current segment comprises a plurality of activations, the method further comprises performing a clustering algorithm on the plurality of activations to define a number of clusters which is smaller than the number of activations components, and the forming of the memory element comprises forming, for each cluster, a corresponding cluster activation based on the activations of the cluster. 10. A computer-implemented method of processing a network input comprising a media element to generate a network output, the method employing a plurality of processing layers, each processing layer being configured to receive embeddings of segments of the media element, and to process the embedding of each current segment of the media element using a corresponding portion of a memory to generate a new embedding of the current segment, the network output being based on embeddings of the segments output by the last processing layer, the method comprising augmenting at least one of the memory portions by a plurality of times:

40 37923819-1 forming a corresponding memory element based on an embedding of the current segment of the media element and having a smaller number of bits than the embedding of the current segment of the media element; and adding the memory element to the memory portion; wherein the processing of the received embedding of each current segment of the media element is by a cross attention mechanism based on the embedding of the current segment of the media element, and a compound data set comprising the added memory elements formed based on the embedding of any previous segments and the embedding of the current segment of the media element. 11. The method of claim 10, in which said memory includes a respective memory portion for each of the processing layers, and the augmenting of the memory is performed at each time step by forming a corresponding memory element for each processing layer using the corresponding embedding of the current segment of the media element received or generated by the processing layer and the corresponding memory portion, and adding the corresponding memory element to the corresponding memory portion. 12. The method of claim 10 or 11 in which the cross-attention mechanism uses a query vector based on the embedding of the current segment. 13. The method of any of claims 10 to 12, in which the cross-attention mechanism uses at least one of a key vector and a value vector which are based on the compound data set. 14. The method of any of claims 10 to 13, in which a value vector of the cross-attention mechanism is based on the embedding of the compound data set and comprises a plurality of components, and the cross-attention mechanism forms, for each component of the value vector, a respective weight value using a query vector based on the current embedding and a key vector based on the compound data set, the output of the cross-attention mechanism being a sum of the components of the value vector weighted by the corresponding weight values. 15. The method of any of claims 10 to 14, in which the embedding of the current segment comprises a plurality of activations, and the forming of each memory element comprises:

41 37923819-1 selecting a subset of the plurality of activations of the embedding of the current segment, and forming, for each selected activation of the embedding of the current segment, a corresponding activation of the memory element. 16. The method of claim 15 in which the subset of the activations is selected randomly. 17. The method of claim 15 in which the subset of the activations are selected successively from the activations of the embedding of the current segment, each of the subset of activations except the first being selected as the one of the previously unselected components of the current segment which is furthest, according to a distance measure, from the closest one of the selected activations. 18. The method of any of claims 10 to 14, in which the embedding of the current segment comprises a plurality of activations, the method further comprises performing a clustering algorithm on the plurality of activations to define a number of clusters which is smaller than the number of activations components, and the forming of the memory element comprises forming, for each cluster, a corresponding cluster activation based on the activations of the cluster. 19. The method of any of claims 10 to 18, in which at least one of the processing layers includes a multi-layer perceptron which receives the output of the attention mechanism, the multi-layer perception being configured to generate the new embedding of the current segment based on the output of the attention mechanism. 20. The method of any of claims 10 to 19, in which the media element processing task is performing a multi-modal task on a network input comprising a media element, to generate a token output comprising text tokens from a vocabulary of text tokens and representing a text response to the media element. 21. A computer-implemented method of processing a media element to generate a latent representation of the media element, comprising:

42 37923819-1 obtaining a first embedding corresponding to a first segment of the media element and comprising first activations; processing the first embedding to generate first latent activations corresponding to the first segment of the media element; processing the first activations to obtain first consolidated activations, wherein the number of first consolidated activations is less than the number of first activations; obtaining a second embedding corresponding to a second segment of the media element and comprising second activations; processing the first consolidated activations and the second embedding to generate second latent activations corresponding to the second segment of the media element, the processing comprising applying a cross-attention mechanism between values based on the second activations and combined activations comprising at least the second activations and the first consolidated activations; processing the second activations to obtain second consolidated activations, wherein the number of second consolidated activations is less than the number of second activations; obtaining a third embedding corresponding to a third segment of the media element; processing the first consolidated activations, the second consolidated activations and the third embedding to generate third latent activations corresponding to the third segment of the media element; and generating a latent representation of the media element based on at least the third latent activations. 22. The method of claim 21, wherein the latent representation of the media element comprises the third latent activations. 23. The method of any preceding claim, in which the media element comprises a video. 24. The method of claim 23, in which each embedding of each segment of the video comprises a number of activations which is independent of the number of segments in the video.

43 37923819-1

25. The method of any preceding claim, in which the media element comprises a sound signal. 26. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1-25. 27. A computer-program product containing instructions that when executed by one or more computers cause the one or more computers to perform the method of any one of claims 1-25.

44 37923819-1