EP4713833A1 - Extending multi-task neural network systems to new modalities - Google Patents

Abstract

Methods and systems for generating a final neural network for performing a multi-modal task. The final neural network generally includes (i) one or more trained neural networks from a set of trained neural networks that generates represents of inputs of the first modality, (ii) a trained decoder neural network that is configured to receive a decoder input and to process the decoder input to generate the output that comprises data of a second modality, and (iii) a connector neural network that maps the representations generated by the one or more trained neural networks to the decoder input. To generate the final neural network, a search is performed over a space of configurations of candidate neural networks that each include the trained decoder neural network, the connector neural network, and a respective subset of one or more of the trained neural networks from the set of trained neural networks.

Description

EXTENDING MULTI-TASK NEURAL NETWORK SYSTEMS TO NEW MODALITIES

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application No. 63/523,910, filed on June 28, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

BACKGROUND

This specification relates to processing inputs using neural networks.

Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

This specification describes a system implemented as computer programs on one or more computers that performs a multi-modal task. That is, the system processes a first modality input and generates a second modality output that includes data of a different modality than the input.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

Multi-modal learning systems, i.e., systems that use neural networks to perform a multi-modal task, are useful for a variety of purposes that require processing data across different modalities. Examples of such purposes include medical imaging diagnostics, video analysis, video rendering from text, and so on.

A valuable multi-modal learning system needs to perform well across a range of inputs while also not being too costly in terms of size and time to train.

Some existing methods attempt to improve performance by training larger models using special-purpose or otherwise improved hardware. While improved hardware can reduce the computational resource requirement of larger neural networks, achieving performance gains by training larger neural networks on larger data remains expensive and inefficient. This specification, on the other hand, describe techniques that utilize insights from previously trained neural networks in order to more efficiently generate a high-performing multi-modal neural network. By making use of previously trained neural networks as described in this specification, the training of the multi-modal neural network can be made significantly more computationally-efficient while still yielding a trained neural network that achieves performance that is comparable to or exceeds that of existing approaches that are significantly more computationally expensive to train.

More specifically, this specification describes a multi-modal learning system that includes: (i) an encoder neural network that includes a subset of one or more trained neural networks from a larger set of trained neural networks that process data of a first modality , (ii) a trained decoder neural network configured to receive a decoder input and to process the decoder input to generate an output that includes data of a second, different modality, and (iii) a connector neural network that maps the representations generated by the one or more trained neural networks to the decoder input.

The system can perform a search through the space of possible configurations of the multi-modal neural networks, i.e., a search through a space of candidate neural networks that each include a respective subset of the trained neural networks in the larger set of trained neural networks. Because to evaluate a given candidate, the system only needs to train the relatively computationally inexpensive connector neural network, the system can perform this search in a computationally efficient manner while still yielding a high-performing final neural network.

More specifically, by performing the search, the system leverages the knowledge from trained neural networks that has been acquired from solving uni-modal tasks to form an efficient-to-train relatively sparse neural network that has excellent quality /size trade off properties, i.e., that successfully achieves a target balance between computationally efficiency / model size and the quality of the outputs generated by the trained neural network.

As a result, by using the described techniques to generate the final neural network, the system can consume much less energy, e.g., electricity, than systems that attempt to develop a final neural network for a given multi-modal task from scratch.

Because the same set of trained neural networks can be used to generate final neural networks for multiple different tasks by performing a new round of search, the techniques described in this specification enable the repeated extension of modular multitask ML systems with new modalities. This enables a large-scale and potentially boundless extension of the set of learned tasks to include any input output modality pair while leveraging the properties of modular multitask systems such as knowledge transfer across tasks with different modalities.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 shows an example multi-modal learning system.

FIG. 2 shows an example connector neural network for an image-to-text multi-modal machine learning task.

FIG. 3 shows a flow diagram of an example process for generating a final neural network of the system using an evolutionary algorithm.

FIG. 4 shows a flow diagram of an example process for generating a candidate neural network.

FIG. 5 shows a flow diagram of an example process for training a generated candidate neural network.

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 shows an example multi-modal learning system 100. The multi-modal learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

The multi-modal learning system 100 is a system that performs a multi-modal task. That is. the multi-modal learning system 100 processes a first modality input 102 and generates a second modality output 118 that includes data of a different modality than the input. Optionally, the first modality input 102 can also include additional data of one or more additional modalities, e.g., the same modality as in the second modality output 118 or one more additional modalities in addition to the modality in the second modality output 118. Similarly, optionally, the second modality output 118 can also include additional data of one or more additional modalities, e.g., the same modality as in the first modality input 102 or one more additional modalities in addition to the modality in the first modality input 102.

Generally, the system 100 can be configured to perform any of a variety of multimodal tasks, e.g., where one of the first or second modalities is any of text, audio, images, videos, or other sensor measurements, e.g., Lidar data, EEG data, EKG data, and so on.

For example, the multi-modal task can be a task that requires processing an input that includes one or more images to generate an output that includes text, i.e.. so that the first modality is images and the second modality is text. For example, the task can be an image captioning task that requires processing an image to generate a text caption for the image, e.g. describing the obj ects present in the image.

As another example, the multi-modal task can be a task that requires processing an input that includes text to generate an output that includes an image that is described by the text in the input, i.e., so that the first modality is text and the second modality’ is images. For example, the text can specify the objects that are to be present in the image.

As another example, the multi-modal task can be a task that requires processing an input that includes text to generate an output that includes audio, i.e., so that the first modality is text and the second modality is audio. For example, the task can be a text to speech task that generates speech that verbalizes text or a music generation task that generates music that is described by the text.

As another example, the multi-modal task can be a task that requires processing an input that includes audio to generate an output that includes text, i.e., so that the first modality is audio and the second modality’ is text. For example, the task can be a speech recognition task where the text is a transcript of the audio.

For each of the above tasks, an image may comprise pixels values and the task can take as input the pixel values or output pixel values accordingly. Audio can comprise digital samples of an audio signal or an encoding of the audio signal to be provided as input or generated as output accordingly.

Generally, the system 100 performs the multi-modal task using a final neural network that includes: (i) an encoder neural network 104 that includes a subset of one or more of trained neural networks, (ii) a trained decoder neural network 1 16 that is configured to receive a decoder input and to process the decoder input to generate the output that comprises the data of the second modality, and (iii) a connector neural network 114 that maps the representations generated by the one or more trained neural networks to the decoder input. To generate the final neural network, the system 100 performs, using training data, a search over a space of configurations of candidate neural networks. Each candidate neural network includes the trained decoder neural network, the connector neural network, and the encoder neural network with a respective subset of one or more of the trained neural networks from the set of trained neural networks. That is, while each candidate neural network has a configuration that includes the trained decoder neural network and the connector neural network, different candidate neural networks can include, as part of the encoder, different trained neural networks from the set of trained neural networks.

Optionally, this search can also include configurations that assign different values to a set of one or more hyperparameters of an architecture of the connector neural network. Hyperparameters for the architecture of the connector neural network will be described below.

Further optionally, the search can also include the set of hyperparameters for an optimizer that trains the candidate neural network. For example, the hyperparameters can include the type of optimizer, e.g.. adamw, momentum, rmsprop, etc. that is used for the training and, in some cases, corresponding parameters for the optimizer type, e.g., learning rate, learning rate schedule, weight decay, and so on.

Details of performing a search to generate a final neural network of the system 100 to perform a multi-modal machine learning task will be provided in the detailed descriptions of FIG. 3. FIG. 4. and FIG. 5.

The encoder 104 includes a subset of trained neural networks 1 10A-C from a set of trained neural networks that are each configured to receive a respective input, that includes data of the first modality, and to process the input to generate a representation of the respective input for a corresponding task.

While only three trained neural networks 110A-C for three tasks (i.e. Task A, Task B, Task C) are shown in FIG. 1, in practice the encoder 104 can include many more trained neural networks for many more machine learning tasks. In some cases, the encoder 104 can include more than one trained neural network for one or more of the tasks.

The machine learning tasks that the set of trained neural networks have been trained to perform can be any appropriate task that takes as input the first modality input 102.

For example, when the first modality input 102 includes one or more images, the machine learning tasks can be any of image classification, object detection, image segmentation, image resolution enhancement, and so on. As another example, when the first modality input 102 includes text, the machine tasks can be any of text classification, sentiment analysis, text summarization, text generation, text translation, and so on.

As yet another example, when the first modality input 102 includes audio, the machine tasks can be any of speaker identification, speech emotion recognition, sound cloning, sound event detection, and so on.

The neural networks in the set of trained neural networks 110A-C can generally have any appropriate neural network architecture that maps an input of the first modality to a representation of the first modality'. In this specification, a “representation” of an input generally refers to a vector, matrix, or higher-order tensor of numeric values, e g., of floatingpoint values or other numeric values.

Each of the trained neural networks in the set of trained neural networks has been trained on a respective task that requires processing the representation generated by the neural network by a corresponding task output head, for the respective task to generate an output for the respective task. Because of the transferability of features learned by the trained neural network on the original task, the representation produced by the trained neural network (the first modality representations, e.g., first modality representations 112A-C) enable performance generalization across unseen data.

As one example, the set of trained neural networks can include neural networks trained as part of a modular, multi-task machine learning system and some or all of the trained neural networks can share modules (or “layer blocks”) with others of the trained neural networks.

Examples of training such a modular, multi-task system are described in ArXiv, 2209.07326, ArXiv, 2209.14745, ArXiv, 2302.02721. ArXiv. 2205.10937, ArXiv, 2205. 12755, and US Patent Application 18/199,901, TRAINING NEURAL NETWORK SYSTEMS TO PERFORM MULTIPLE MACHINE LEARNING TASKS, filed on 5/19/2023, all of which are herein incorporated by reference in their entirety. As another example, the trained neural networks can include neural networks that have been trained independently on the corresponding task(s). For example, the trained neural networks can include publicly-available pre-trained neural networks that generate representations of inputs of the first modality.

The connector neural network 114 can be any neural network with an appropriate architecture that can process one or more first modality representations 112A-C to generate an output that serves as the input for the decoder 116. Further details of an example connector neural network 114 will be provided in the detailed description of FIG. 2.

The decoder neural network 116 can be any neural network that processes the output of the connector neural network 114 as an internal representation input to generate a second modality output 118.

For example, the decoder neural network 116 can have been pre-trained on a uni- modal task for the second modality or a multi-modal task that requires generating outputs of the second modality. For example, the decoder neural network 116 can be an auto-regressive language model neural network or an auto-regressive multi-modal neural network that generates outputs sequences of tokens. For example, the decoder neural network 116 can be a decoder-only or encoder-decoder Transformer neural network that includes one or more self-attention layers.

Using a pretrained decoder instead of training from random initialization of parameters or initial trained parameters offer the advantages of quicker convergence of during training (due to training fewer parameters) and reduction in the likelihood of overfitting (also due to fewer parameters). The use of the pre-trained decoder also allows the system 100 to leverage the ability of the decoder to effectively generate outputs of the second modality.

FIG. 2 shows an example connector neural network 200 for an image-to-text multimodal machine learning task.

The example connector 200 processes the first modality representations to produce a decoder input 208 for the decoder 116 to process, and includes components such as a shape adapter layer 202, one or more resampling layers 204. and a soft-prompt prefix 206.

The shape adapter layer 202 processes the first modality representations 112A-C. i.e., image representations in this example, as a two-dimensional tensor having a sequence dimension and a hidden dimension to generate a single representation for further processing.

That is, the image representations, i.e., first modality representations, are concatenated along a sequence dimension, then processed by the shape adapter 202, e.g.. a fully connected layer or a multilayer perceptron, to generate a single output, i.e. a single representation, that reflects all the concatenated image representations. More generally, the shape adapter 202 maps the hidden dimension of the concatenated representation to the size of a hidden state vector the decoder 116 requires as input.

Optionally, the one or more resampling layers 204 can then be applied to compress the sequence dimension to a fixed size. These resampling layers 204 can both add representational capacity to the connector, i.e., the ability to learn more hierarchical features, and control the size of the final output of the connector 204, i.e., to improve the computational efficiency of the system after training. For example, the one or more resampling layers 204 can include one or more Perceiver layers or other types of layers that map a variable-length input to a fixed-length sequence.

The soft-prompt prefix 206 is an optional element that prepends learnable parameters of fixed size along the sequence dimension of the last resampling layer output. Because these parameters are the same for all first modality inputs 102, the parameters assist the system to generalize its performance across all first modality inputs 102.

That is, in the example of FIG. 2, the decoder input 208 includes (i) a soft-prompt that is a learned input that is independent of the first modality representations 112A-C followed by (ii) a sequence of inputs that is generated from the first modality representations 112A-C.

Overall, the connector neural network 114 enhances and standardizes the insights derived from the first modality representations 112A-C through knowledge compilation, leading to improved performance of the multi-modal task. Additionally, by using the connector as the only trainable component of the system, the system can evaluate a given candidate neural network with a short training time and while mitigating overfitting by maintaining the generation of the first modality representations intact.

As described above, the system 100 performs a search to generate a final neural network for performing the multi-modal task. One example of performing a search is described below.

FIG. 3 shows a flow diagram of an example process 300 for generating a final neural network of the system 100 using an evolutionary algorithm. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-modal learning system, e.g., the multi-modal learning system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 3OO.The system initializes a population of trained candidate neural networks for the multi-modal task (step 302). For example, the system can initialize the population to be empty, i.e., to not include any candidates. As another example, the system can initialize the population with a user selected root-model, i.e., an initial candidate neural netw ork from which subsequent candidate neural netw orks are generated from. The system then performs steps 304-310 at each of one or more generation steps. As the system performs the generation steps, the population grows to include model architectures previously sampled as candidate neural networks during the model architecture search.

The multi-modal learning system generates one or more candidate neural networks for the multi-modal task (step 304).

As described above, each candidate neural network includes (i) an encoder neural network that includes a subset of one or more of the trained neural networks in a set of trained neural networks, (ii) a trained decoder neural network that is configured to receive a decoder input and to process the decoder input to generate the output that includes the data of the second modality, and (iii) a connector neural network that maps the representations generated by the one or more trained neural networks to the decoder input. Moreover, each candidate will generally include a connector neural network and the same trained decoder neural network, but can include different ones of the trained neural networks in the set.

Generally, to generate a given candidate neural network, the system selects a parent neural network from the population and then generates the candidate neural network by modifying the parent neural network.

The details of generating new candidate neural networks to grow the population will be provided in the detailed description of FIG. 4.

The system trains each of the one or more candidate neural networks (step 306). The details of training each of the candidate neural networks will be provided in the detailed description of FIG. 5.

The system determines a performance score for each trained candidate neural network (step 308).

The performance score can be defined to optimize a mixture of a variety of factors. Examples of such factors include validation quality, inference latency, training compute or model size.

For example, the performance score of a candidate can be the product of validation quality, a scaling factor, and the number of parameters included in the candidate, summarized as performance_score(m) = q(jn)*s* i/#params), w here m denotes the candidate model of interest, q(m) denotes the validation quality, v denotes the scaling factor, and #params denotes number of model parameters. This described performance score considers validation quality and model size. The validation qualify generally measures the performance of the candidate neural network on a validation data set for the multi-modal task and can be any appropriate function for the multi-modal task. As another example, the performance score function can further include a term (“walltime") that measures, summarized as performance_score(m) = < (m~)*s*(].l#params)*(Ylwalltime). For example, walltime can denote the actual elapsed time that it takes to execute inference on the validation set. This performance score metric considers validation quality, model size, and additionally inference latency.

In some cases, the inference latency can be targeted to a specific set of hardware devices, e.g.. a specific set of hardware devices on which the multi-modal task will be performed after training. For example, the specific set of hardware devices can be a set of special-purpose hardware accelerators, e.g., GPUs, TPUs, or other ASICs that will be used at inference. Thus, the search can be targeted to generate a final neural network that is optimized for a specific set of hardware devices. Optionally, the performance score function can, instead or in addition to the latency, include one or more additional terms that are targeted to the specific set of hardware devices, e.g., throughput or memory access measures.

The system determines whether to add each of the trained candidate neural networks to the population (step 310).

The system can make this determination for any given candidate based on the performance score of the candidate. By prioritizing parent neural networks with higher performance scores, the convergence to the best final neural network can be accelerated.

For example, the criteria to add the trained candidate neural networks to the population can be that the trained candidate neural network has a better performance score than a parent candidate neural network that was used to generate the candidate neural network, as will be described in more detail below.

As another example, the criteria to add the trained candidate neural networks to the population can be that the trained candidate neural network has a better performance score than a threshold performance score. The threshold performance score can be fixed for all generation steps or can be increased according to a schedule across the generation steps.

The process 300 then repeats steps 304 through step 310 until a stopping criterion is met, such as reaching a maximum number of iterations, reaching a maximum population size, or any candidate achieving a satisfactory performance score.

After the criterion is met, the system selects, from the final population, the trained candidate neural network that has the best performance score as the final neural network (step 312).

FIG. 4 shows a flow diagram of an example process 400 for generating a candidate neural network. The system determines an ordering of the trained candidate neural networks in the population according to the respective performance scores for the trained candidate neural networks (step 402).

If the population is empty, then a candidate neural network with default hyperparameters and a connector with a random architecture and randomly initialized parameter values is added to the set.

When the population set is greater than one. the ordering can be determined based on one or more criteria. For example, the ordering of the population can be in decreasing order of the performance score.

Starting at an initial trained candidate neural network in the ordering of the population, the system evaluates the current initial trained candidate neural network (404).

The system determines according to a random sample, whether to select the current trained candidate neural network (step 406).

That is, the system can assign the current trained neural network a probability and can select the current trained neural network with the assigned probability.

For example, the probability can depend on the number of times that the particular neural network has been selected as the parent neural network when training a candidate neural network. For example, the probability for the particular neural network m can be equal to or proportional to 0.5^A#selections(m) wherein #selections(m) identifies a number of times that the particular neural network m has been selected as the parent neural network when training a candidate neural network.

In response to determining to select the trained candidate neural network, the system selects the trained candidate neural network as the parent neural network (step 408).

In response to determining not to select the trained candidate neural network, the system continues to evaluate the next trained candidate neural network in the order (step 410), returning to step 406.

After step 408, the system applies one or more mutation operations from a set of mutation operations to the parent neural network to generate the candidate neural network (step 412). These mutations are generally operations that change the configuration of the parent neural network to create a new candidate neural network whose architecture still lies within the search configuration space.

For example, the set of mutations can include one or more operations that each modify the one or more trained neural networks from the subset that are included in the parent neural network. As one example, a first mutation operation can remove one of the trained neural networks from the subset that are included in the parent neural network. As another example, a second mutation operation can add a new trained neural network to the subset that are included in the parent neural network.

As another example, the set of mutations can include one or more mutations that each modify one or more hyperparameters of an architecture of the connector neural netw ork. These connector architecture hyperparameters can include hyperparameters that define any of the number, types, or parameterization of the layers and blocks that make up the connector neural network. Considering the example 200 connector previously described, some specific parameters would be the shape adapter’s architecture type, e.g., a fully connected layer or a multilayer perceptron; the number of resampling layers: and the size of the learnable prompt prefix.

As another example, the set of mutations can include one or more mutations that each modify⁷ the hyperparameters of the training of the connector neural network.

As one example, one of these mutations can modify the type of optimizers used for the training of the candidate neural network.

As one example, one of the mutations can modify the learning rate of the optimizer used for the training. For example, the learning rate a of the optimizer may be updated in an incremental fashion such as a = a + A, where A is the change in value applied by the mutation.

In some implementations, the system determines whether to apply each mutation operation in the set of mutations operations in accordance with a respective probability for the mutation operation.

As one example, the probability for the mutation operation may be parameterized as a global mutation probability constant.

As another example, the probability for a mutation operation may be parameterized per candidate neural network and tuned by an evolutionary method during the search, thus enabling the selection of distinct mutation probabilities for each candidate that are adapted through time.

As described above, using evolutionary algorithms as a search method for a final neural network allows the system to systematically and efficiently explore strong candidate neural networks by iteratively refining the search space through a balance of exploring new candidates while exploiting information of already strong performing candidates.

FIG. 5 shows a flow diagram of an example process 500 for training a generated candidate neural network. The system selects training data (step 502). The sources and types of training data vary depending on the multi-modal task. For example, for describing plant health, an image captioning task where images are the input and natural language text is the output, the training data would include images of pants along with corresponding natural language text that may include plant species names, and descriptors such as ‘bug-infested’, ‘infected’, ‘wilted’ etc.

For each candidate neural network that has a parent neural network that has a trained connector neural network, the system initializes parameter values for at least a portion of the parameters of the connector neural network using the trained values of the trained connector neural network of the parent neural network (step 504). For example, the system can initialize parameters values for each component that is shared between the connector neural network in the candidate and the connector neural network in the parent using the trained values of the trained connector neural network of the parent neural network. Initializing a parameter “using” another parameter can refer to setting the value of the parameter equal to the value of the other parameter.

By using preinitialized parameter values, the training can reach convergence quickly due to the fact that parameters can already capture relevant features and incremental improvements may require only a small amount of training.

The system trains the connector neural network while holding the one or more trained neural networks in the candidate neural network and, optionally, the trained decoder neural network fixed (step 506).

The system can generally use any appropriate optimizer to train on the training data on any of a variety of appropriate loss functions for the multi-modal task. For the example task of captioning an image of a plant regarding its health mentioned above, the loss function may be the cross-entropy loss function, hinge loss function, focal loss function, and so on. In addition, the loss functions may also include regularization terms for the learnable parameters, such as the L-l (LASSO regularization) and L-2 (ridge regression regularization) variants of the L-p norm.

This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be. or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

In this specification, the term “database’’ is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory' or a random access memory' or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry'. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g , a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g.. a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machine learning framework, .e.g., a TensorFlow' framework or a Jax framework.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end. middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication netw ork. Examples of communication networks include a local area network (LAN) and a wide area netw ork (WAN), e.g.. the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: receiving training data for training a neural network to perform a first multi-modal task that requires receiving an input that comprises data of a first modality and generating an output that comprises data of a different, second modality; maintaining data specifying a set of trained neural networks that are each configured to receive a respective input that includes data of the first modality and to process the input to generate a representation of the respective input; and generating a final neural network for performing the first multi-modal task, wherein the final neural network comprises (i) one or more of the trained neural networks, (ii) a trained decoder neural network that is configured to receive a decoder input and to process the decoder input to generate the output that comprises the data of the second modality, and (iii) a connector neural network that maps the representations generated by the one or more trained neural networks to the decoder input, the generating comprising: performing, using the training data, a search over a space of configurations of candidate neural networks that each include the trained decoder neural network, the connector neural network, and a respective subset of one or more of the trained neural netw orks from the set of trained neural networks.

2. The method of claim 1, wherein the first modality is images and the second modality is text.

3. The method of any preceding claim, wherein each trained neural network has been trained on a respective task that requires processing the representation generated by the neural network by a task output head for the respective task to generate an output for the respective task.

4. The method of claim 3, wherein, for one or more of the trained neural networks, the respective task is a uni-modal task for the first modality.

5. The method of any preceding claim, wherein performing the search comprises: initializing a population of trained candidate neural netw orks for the first multi-modal task; and at each of one or more generation steps: generating one or more candidate neural networks for the first multi-modal task; training each of the one or more candidate neural networks on at least a portion of the training data to generate a trained candidate neural network; determining a performance score for each trained candidate neural network; and determining whether to add each of the trained candidate neural networks to the population.

6. The method of claim 5, wherein training each of the one or more candidate neural networks comprises training the connector neural network while holding the one or more trained neural networks in the candidate neural network and, optionally, the trained decoder neural network fixed.

7. The method of claim 5 or 6, wherein performing the search comprises, after performing the one or more generation steps, selecting, from the population, the trained candidate neural network that has a best performance score as the final neural network.

8. The method of any one of claims 5-7, wherein generating one or more candidate neural networks for the first multi-modal task, comprises for each candidate neural network: selecting a parent neural network for the candidate neural network; and applying one or more mutation operations from a set of mutation operations to the parent neural network to generate the candidate neural network.

9. The method of claim 8, wherein determining whether to add each of the trained candidate neural networks to the population comprises: for each trained candidate neural network, determining to add the trained candidate neural networks to the population when the trained candidate neural network has a better performance score than the parent candidate neural network.

10. The method of claim 8 or claim 9, wherein selecting a parent neural network for the candidate neural network comprises: determining an ordering of the trained candidate neural networks in the population according to the respective performance scores for the trained candidate neural networks; and starting at an initial trained candidate neural network in the ordering, evaluating each of one or more of the respective trained candidate neural networks, comprising: determining, according to a random sample, whether to select the trained candidate neural network; in response to determining to select the trained candidate network, ending the evaluation and selecting the trained candidate neural network as the parent neural network; and in response to determining not to select the trained candidate neural network, continuing to evaluate the next trained candidate neural network in the order.

11. The method of claim 10, wherein a particular neural network m is selected with a probability that depends on a number of times that the particular neural network has been selected as the parent neural network when training a candidate neural network.

12. Th method of claim 11, wherein the probability for the particular neural network m equal to or proportional to:

Q ^ttselections^m.t) wherein #selections(m,t) identifies a number of times that the particular neural network m has been selected as the parent neural network when training a candidate neural network.

13. The method of any one of claims 8-12, wherein the set of mutation operations includes: one or more mutation operations that each modify the one or more trained neural networks from the set that are included in the parent neural network.

14. The method of claim 13, wherein the one or more mutation operations that each modify the one or more trained neural networks from the set that are included in the parent neural network comprise: a first mutation operation that removes one of the trained neural networks from the subset that are included in the parent neural network; and a second mutation operation that adds a new trained neural network from the subset that are included in the parent neural network.

15. The method of any one of claims 8-14, wherein the set of mutation operations includes: one or more mutation operations that each modify one or more hyperparameters of an architecture of the connector neural network.

16. The method of any one of claims 8-15, wherein training each of the one or more candidate neural networks on at least a portion of the training data to generate a trained candidate neural network comprises: for each candidate neural network that has a parent neural network that has a trained connector neural network, initializing parameter values for at least a portion of the parameters of the connector neural network using the trained values of the trained connector neural network of the parent neural network.

17. The method of any preceding claim, wherein the space of configurations of candidate neural networks includes configurations that assign different values to a set of one or more hyperparameters of an architecture of the connector neural network.

18. The method of any preceding claim when dependent on claim 8, wherein applying one or more mutation operations from a set of mutation operations to the parent neural network to generate the candidate neural network comprises: determining whether to apply each mutation operation in accordance with a respective probability for the mutation operation.

19. The method of any preceding claim, wherein the trained decoder neural network has been pre-trained on a uni-modal task for the second modality.

20. A method comprising: receiving a network input for the first multi-modal task; and processing, using the final neural network of any preceding claim, the network input to generate a network output.

21. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-15.

22. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-15.