EP4627579A1

EP4627579A1 - Optimizing viral capsids using machine learning

Info

Publication number: EP4627579A1
Application number: EP23828943.3A
Authority: EP
Inventors: Sourav Roy CHOUDHURY; Eugenia LYASHENKO; Yu Qiu; Jason Wu
Original assignee: Sanofi SA
Current assignee: Sanofi SA
Priority date: 2022-12-02
Filing date: 2023-11-28
Publication date: 2025-10-08
Also published as: CN120615216A; WO2024118631A1; JP2026500138A

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a fitness score for a capsid monomer. In one aspect, a method comprises: receiving data defining an amino acid sequence of a monomer of a capsid; processing the data defining the amino acid sequence of the monomer of the capsid using an embedding neural network, in accordance with values of a set of embedding neural network parameters, to generate an embedding of the monomer of the capsid; processing the embedding of the monomer of the capsid using a fitness prediction machine learning model to generate a fitness score that characterizes a predicted quality of the capsid; and outputting the fitness score that characterizes the predicted quality of the capsid.

Description

OPTIMIZING VIRAL CAPSIDS USING MACHINE LEARNING

BACKGROUND

[0001] This specification relates to optimizing viral capsids using machine learning.

[0002] A virus can refer to a submicroscopic infectious agent that can replicate inside living cells of an organism, and a capsid can refer to a protein shell of a virus that encloses the genetic material of the virus.

[0003] A capsid can include an assembly of repeating protein structural units referred to as “capsid monomers” or simply “monomers.” A capsid can include any appropriate number of monomers, e.g., 20 monomers, 50 monomers, 100 monomers, etc. A capsid monomer can include a sequence (chain) of amino acids, where each amino acid is selected from a set of possible amino acids, e.g., the standard set of 20 alpha amino acids, e.g., glycine, alanine, valine, etc. The amino acid sequence of a capsid monomer can have any appropriate length (i.e., where the “length” of amino acid sequence refers to the number of positions, occupied by amino acids, in the amino acid sequence), e.g., a length of 10. 20. 30, etc.

[0004] An "adeno-associated virus” (AAV) is a small virus that infects humans but that causes a mild immune response and is not currently known to cause disease. AAVs have properties making them appropriate candidates for creating viral vectors for gene therapy, e.g., their apparent non-pathogenicity, their ability to infect non-dividing cells, and their capacity to stably integrate into host cell genomes at specific sites. AAV capsids can include a mixture of capsid monomers VP1 , VP2, and VP3, totaling 60 monomers, which jointly assemble to form a capsid with icosahedral symmetry.

SUMMARY

[0005] This specification describes a fitness prediction system implemented as computer programs on one or more computers in one or more locations that can process data defining an amino acid sequence of a capsid monomer to generate a fitness score that characterizes a predicted quality of the capsid. The quality of a capsid can refer to any appropriate property' of the capsid, e.g., the manufacturability of the capsid, the capacity of viruses that include the capsid to avoid neutralization, the immunoreactivity of the capsid, etc., as will be described in more detail below.

[0006] Throughout this specification, an “embedding” can refer to an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values.

[0007] Throughout this specification, a “block” in a neural network can refer to a collection of neural network layers in the neural network. [0008] According to a first aspect, there is provided a method comprising: receiving data defining an amino acid sequence of a monomer of a capsid; processing the data defining the amino acid sequence of the monomer of the capsid using an embedding neural network, in accordance with values of a set of embedding neural network parameters, to generate an embedding of the monomer of the capsid; processing the embedding of the monomer of the capsid using a fitness prediction machine learning model to generate a fitness score that characterizes a predicted quality of the capsid: and outputting the fitness score that characterizes the predicted quality of the capsid.

[0009] In some implementations, the method further comprises processing a representation of the monomer of the capsid using an error prediction machine learning model to generate a fitness error, wherein the fitness error is an estimate of an error in the fitness score generated by the fitness prediction machine learning model.

[0010] In some implementations, the method further comprises updating the fitness score by combining: (i) the fitness score generated by the fitness prediction machine learning model, and (ii) the fitness error generated by the error prediction machine learning model.

[0011] In some implementations, updating the fitness score comprises summing the fitness score generated by the fitness prediction machine learning model and the fitness error generated by the error prediction machine learning model.

[0012] In some implementations, the fitness prediction machine learning model has a different model architecture than the error prediction machine learning model.

[0013] In some implementations, processing a representation of the monomer of the capsid using the error prediction machine learning model comprises processing a representation of the amino acid sequence of the monomer of the capsid using the error prediction machine learning model.

[0014] In some implementations, the error prediction machine learning model processes a different model input than the fitness prediction machine learning model.

[0015] In some implementations, processing the data defining the amino acid sequence of the capsid using the embedding neural network comprises: instantiating a respective amino acid embedding corresponding to each position in the amino acid sequence of the monomer of the capsid; processing the amino acid embeddings by one or more neural network layers of the embedding neural network to update each amino acid embedding; and generating the embedding of the monomer of the capsid protein based on one or more of the updated amino acid embeddings. [0016] In some implementations, generating the embedding of the monomer of the capsid protein based on one or more of the updated amino acid embeddings comprises generating the embedding of the monomer of the capsid protein by combining the updated amino acid embeddings.

[0017] In some implementations, processing the amino acid embeddings by one or more neural network layers of the embedding neural network to update each amino acid embedding comprises processing the amino acid embeddings using one or more self-attention neural network layers of the embedding neural network.

[0018] In some implementations, processing the data defining the amino acid sequence of the monomer of the capsid using the embedding neural network comprises: instantiating a sequence of amino acid embeddings that comprises a respective amino acid embedding corresponding to each position in the amino acid sequence of the monomer of the capsid; and sequentially processing each amino acid embedding in the sequence of amino acid embeddings using a recurrent neural network to generate an updated hidden state of the recurrent neural network; and determining the embedding of the monomer of the capsid based on the updated hidden state of the recurrent neural network.

[0019] In some implementations, the embedding neural network has been trained, on a set of training proteins, to perform a token unmasking task.

[0020] In some implementations, training the embedding neural network to perform the token unmasking task comprises, for each training protein: generating a masked amino acid sequence by masking one or more positions in an amino acid sequence of the training protein; processing data defining the masked amino acid sequence using the embedding neural network to determine, for each masked position in the amino acid sequence of the training protein, a respective probability distribution over a set of possible amino acids; and training the embedding neural network to optimize an objective function that, for each masked position in the amino acid sequence of the training protein, measures an error between: (i) the probability distribution generated for the masked position, and (ii) an identity' of an amino acid located at the masked position in the training protein.

[0021] In some implementations, the embedding neural network has been trained, on a set of training proteins, to perform a next token prediction task.

[0022] In some implementations, training the embedding neural network to perform the next token prediction task comprises, for each training protein: generating, for each of one or more target positions in an amino acid sequence of the training protein, a respective probability distribution over a set of possible amino acids using the embedding neural network, wherein for each target position, the probability distribution generated for the target position only depends on amino acids at preceding positions in the amino acid sequence of the training protein; and training the embedding neural network to optimize an objective function that, for each target position in the amino acid sequence of the training protein, measures an error between: (i) the probability distribution generated for the target position, and (ii) an identity of an amino acid located at the target position in the training protein.

[0023] In some implementations, the fitness prediction neural network has been trained on a set of training examples, wherein each training example corresponds to a respective training monomer of a capsid and includes: (i) an embedding of the training monomer of the capsid, and (ii) a target fitness score for the training monomer of the capsid.

[0024] In some implementations, training the fitness prediction neural network on the set of training examples comprises rebalancing the set of training examples based on the target fitness scores specified by the training examples.

[0025] In some implementations, the fitness prediction machine learning model is a parametric machine learning model.

[0026] In some implementations, the fitness prediction machine learning model comprises one or more of: a neural network model or a random forest model or a support vector regression model.

[0027] In some implementations, the fitness prediction machine learning model is a nonparametric model.

[0028] In some implementations, the fitness prediction machine learning model comprises a k- nearest neighbor model.

[0029] In some implementations, the predicted quality⁷ of the capsid characterizes a predicted manufacturability of the capsid.

[0030] In some implementations, the manufacturability of the capsid is defined based on a number of instances of the capsid that are generated when a plasmid coding for the capsid is transfected into one or more cells.

[0031] In some implementations, the number of instances of the capsid that are generated when a plasmid coding the capsid is transfected into one or more cells is normalized by a number of generated wild-type sequences.

[0032] In some implementations, the predicted quality of the capsid characterizes a predicted capacity of viruses that include the capsid to avoid neutralization.

[0033] In some implementations, the predicted quality of the capsid characterizes a predicted immunoreactivity of the capsid. [0034] In some implementations, the predicted quality of the capsid characterizes a predicted capacity of viruses that include the capsid to penetrate a target tissue.

[0035] In some implementations, the predicted quality of the capsid characterizes a packing capacity of the capsid.

[0036] In some implementations, the predicted quality of the capsid characterizes a predicted capacity of viruses that include the capsid to integrate into host genomes.

[0037] In some implementations, the capsid corresponds to a capsid of an adeno-associated virus (AAV).

[0038] In some implementations, the method further comprises: selecting a capsid monomer from a set of possible capsid monomers using the fitness prediction machine learning model; and physically generating one or more viruses that include a protein having the selected capsid protein monomer.

[0039] In some implementations, selecting a capsid monomer from a set of possible capsid monomers using the fitness prediction machine learning model comprises: generating a respective fitness score for each capsid monomer in the set of possible capsid monomers using the fitness prediction machine learning model; and selecting a capsid monomer based on the fitness scores.

[0040] In some implementations, selecting a capsid monomer based on the fitness scores comprises selecting a capsid monomer having a highest fitness score from among the capsid monomers included in the set of possible capsid monomers.

[0041] In some implementations, the method further comprises applying the generated viruses to a subject to achieve a therapeutic effect in the subject.

[0042] According to another aspect, there is provided a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of methods described herein.

[0043] According to another aspect, there are provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the methods described herein.

[0044] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0045] The fitness prediction system described in this specification leverages machine learning models to predict the quality of capsids. [0046] Generally, the accuracy and robustness of a machine learning model can be limited by the amount and quality of training data that is available for training the machine learning model. In many cases, generating capsid fitness training data for training the fitness prediction system involves running laboratory experiments, e.g., by measuring the number of instances of capsids that are generated by transfecting cells with plasmids coding for the capsids. These laboratory experiments can be expensive and time consuming, and can generate “noisy” (e.g., inaccurate or inconsistent) data.

[0047] Moreover, the quality (e.g., manufacturability) of a capsid can result from complex interactions of biology, chemistry, and mechanics, and thus in order to predict capsid quality, a machine learning model must perform sophisticated implicit reasoning. The reasoning capacity of a machine learning model can depend on the number of parameters of the model, e.g., such that increasing the number of parameters of the machine learning model can increase its reasoning capacity. Certain machine learning models that perform complex prediction tasks can include large numbers of parameters, e.g., millions or billions of parameters. However, increasing the number of parameters of a machine learning model can increase the likelihood of the machine learning model “overfitting” during training, in particular, by capturing some of the residual variation (noise) in the training data as if that variation represented underlying structure in the training data. The risk of overfitting can be exacerbated when the training data is limited in quantity and subject to noise, as is the case with capsid fitness training data, as described above.

[0048] The fitness prediction system described in this specification implements various innovations to address these issues.

[0049] To generate a fitness score for a capsid, the fitness prediction system can process data defining an amino acid sequence of a monomer of the capsid using an embedding neural network to generate an embedding of the capsid monomer. The fitness prediction system can then process the embedding of the capsid monomer using a fitness prediction machine learning model to generate the fitness score for the capsid.

[0050] The fitness prediction system can train the embedding neural network on large quantities of protein data for which labels (such as capsid fitness scores) are unavailable using unsupervised learning techniques. For instance, the fitness prediction system can train the embedding neural network to perform a token unmasking task or a next token prediction task, as will be described in more detail below. Training the embedding neural network to perform unsupervised tasks causes the embedding neural network to learn to generate capsid monomer embeddings that compactly encode biological/chemical/mechanical properties of capsid monomers which can be leveraged by the fitness prediction machine learning model to generate accurate fitness scores. Moreover, training the embedding neural network on unsupervised tasks does not rely on capsid fitness training data, and therefore is not affected by the limited quantity and quality of the available capsid fitness training data.

[0051] After generating a fitness score for a capsid using the fitness prediction machine learning model, the fitness prediction system can use a separate error prediction machine learning model to generate a fitness error that predicts an error in the fitness score. The fitness prediction system can then update the fitness score using the fitness error, e.g., to correct errors in the fitness score, which can increase the overall accuracy of the fitness prediction system. More specifically, the fitness prediction system can train the error prediction machine learning model to learn to correct systematic errors in fitness scores generated by the fitness prediction machine learning model. The error prediction machine learning model can detect and avoid certain systematic errors of the fitness prediction machine learning model, e.g., by having a different model architecture, or by processing a different form of input than the fitness prediction machine learning model, as will be described in more detail below.

[0052] The fitness prediction system can rebalance the capsid fitness training data to increase the accuracy and learning rate of the machine learning models included in the fitness prediction system. For instance, without rebalancing the training data, the number of training examples with low capsid fitness scores may be significantly higher than the number of training examples with high capsid fitness scores. Training on unbalanced training data can limit the ability of training examples with high capsid fitness scores to impact the parameter values of the machine learning models and therefore reduce the capacity of the machine learning models to learn characteristics of capsid monomers that result high capsid fitness scores. The fitness prediction system can rebalance the training data to increase the impact of training examples with high capsid fitness scores. For instance, the fitness prediction system can resample the training data to increase the proportion of training examples w ith high capsid fitness scores, or can modify the objective function used during training to increase the penalty on prediction errors for training examples corresponding to high capsid fitness scores.

[0053] After being trained, the fitness prediction system can be used to screen large libraries of capsid monomers (e.g., that includes millions or billions of possible capsid monomers) to identify capsid monomers that result in capsids with high fitness scores, e.g., capsids with high manufacturability. That is, the fitness prediction system enables an automated search through the space of possible capsid monomers to identify capsid monomers with desirable properties. The fitness prediction system can enable exploration of orders of magnitude more capsids than could be physically generated and tested in laboratory experiments. Moreover, the fitness prediction system operates rapidly (e.g., requiring 1 second or less to generate a predicted fitness score) while being less expensive and consuming relatively fewer operational resources than laboratory experiments.

[0054] The fitness prediction system can be used to identify capsids for viral vectors for gene therapies. Gene therapy has the potential to transform healthcare, e g., by enabling diseases (such as hemophilia, leukemia, melanoma, etc.) to be addressed with single-dose treatments of therapeutic viral vectors (e.g., AAV vectors). However, gene therapy treatments are often expensive, in some cases costing $1 million or more to treat a single patient. High costs are a major barrier to the full realization of the potential transformative effects of gene therapy. A major contributor the cost of gene therapy is the cost of manufacturing capsids for viral vectors. The fitness prediction system described in this specification can enable the discovery of viral capsids with high levels of manufacturability which could significantly reduce the cost of gene therapy.

[0055] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0056] FIG. 1 shows an example environment for screening a capsid monomer library using a fitness prediction system.

[0057] FIG. 2 shows an example fitness prediction system.

[0058] FIG. 3 is a flow diagram of an example process for generating a fitness score that characterizes the predicted qualify of a capsid.

[0059] FIG. 4 is a flow diagram of an example process for generating an embedding of a capsid monomer using an embedding neural network having a recurrent neural network architecture. [0060] FIG. 5 is a flow diagram of an example process for generating an embedding of a capsid monomer using an embedding neural network having a neural network architecture that includes a sequence of update blocks.

[0061] FIG. 6 is a flow diagram of an example process for training an embedding neural network to perform a next token prediction task. [0062] FIG. 7 is a flow diagram of an example process for training an embedding neural network to perform a token unmasking task.

[0063] FIG. 8 provides an illustration of the accuracy achieved by the fitness prediction system described in this specification on the task of predicting fitness scores that characterize capsid manufacturability' .

[0064] FIG. 9 provides a comparison of the accuracy achieved by the fitness prediction system with and without the error prediction machine learning model.

[0065] FIG. 10 provides a comparison of the prediction accuracies achieved by the fitness prediction system for different choices of model architecture of the fitness prediction machine learning model.

[0066] FIG. 11 provides a comparison of the prediction accuracies achieved by the fitness prediction system when training data rebalancing is not performed and when training data rebalancing is performed.

[0067] FIG. 12 provides a comparison of the prediction accuracies achieved by the fitness prediction machine learning model when the embedding neural network generates capsid monomer embeddings with 1900 dimensions and when the embedding neural network generates capsid monomer embeddings with 256 dimensions.

[0068] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0069] FIG. 1 shows an example environment 100 for screening a capsid monomer library 102 using a fitness prediction system 200.

[0070] The capsid monomer library 102 defines a set of capsid monomers, where each capsid monomer is represented by a respective sequence of amino acids. The capsid monomer library 102 can include any appropriate number of capsid monomers, e.g., 1 thousand, 1 million, or 1 billion capsid monomers.

[0071] The capsid monomer library 102 can be generated in any of a variety of possible ways. For instance, some or all of the capsid monomers in the capsid monomer library 102 can be mutations of an "original’’ capsid monomer, e.g., a VP1, VP2, or VP3 capsid monomer of an AAV. More specifically, each capsid monomer in the capsid monomer library 102 can be generated by modifying the identity of a respective amino acid at one or more positions in the amino acid sequence of the original capsid monomer. Positions in the amino acid sequence of the original capsid monomer can be selected for mutation in any appropriate way. e.g., through random selection or through selection in accordance with a predefined rule. The identities of new amino acids substituted into positions in the amino acid sequence of the original capsid monomer can be selected in any appropriate way, e.g.. randomly selected from a probability distribution over a set of possible amino acids. The respective amino acid sequence of each capsid monomer in the capsid monomer library can differ from the amino acid sequence of the original capsid monomer in any appropriate number of positions, e.g., 1 position, 3 positions, or 10 positions.

[0072] The fitness prediction system 200 is configured to process a capsid monomer (e.g., from the capsid monomer library 102) to generate a fitness score 104 for the capsid monomer that characterizes a predicted quality of the corresponding capsid. A fitness score 104 for a capsid monomer can characterize any appropriate quality of the corresponding capsid. A few examples of possible fitness scores 104 are described next.

[0073] In some implementations, the fitness prediction system 200 generate a fitness score 104 for a capsid monomer that characterizes a manufacturability of the corresponding capsid. The manufacturability' of a capsid characterizes a rate at which the capsid can be generated when plasmids coding the capsid are transfected into one or more cells. (More specifically, the manufacturability of a capsid can be based at least in part on a number of instances of the capsid that are generated when plasmids coding for the capsid are transfected into one or more cells). That is, the manufacturability of a capsid characterizes how efficiently the capsid can be generated. The manufacturability of a capsid indirectly characterizes various structural and biochemical properties of the capsid that affect its manufacture, e.g., how readily the capsid monomers of the capsid can assemble into the three-dimensional structure of the capsid, the stability of the capsid, etc.

[0074] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes a predicted capacity of a virus that includes the corresponding capsid to avoid neutralization in an organism. Virus neutralization refers to a process by which the capacity of a virus to infect a cell is reduced, e.g., through the binding of antibodies to epitopes on the surface of the virus.

[0075] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes a predicted immuno reach vity of the corresponding capsid. The immunoreactivity of a capsid refers to a measure of immune reaction caused by the capsid.

[0076] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes a predicted capacity of viruses that include the capsid to penetrate a target tissue in an organism. The target tissue can be, e.g., tissue corresponding to a particular organ, e.g., liver tissue, brain tissue, eye tissue, etc.

[0077] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes a predicted packing capacity of the corresponding capsid. The packing capacity of a capsid refers to a measure of the amount of genetic material (e.g., deoxyribonucleic acid (DNA) or ribonucleic acid (RNA)) that can be contained by the capsid.

[0078] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes a predicted capacity of viruses that include the corresponding capsid to integrate into host genomes.

[0079] The fitness prediction system 200 can screen the capsid monomer library 102 to identify capsid monomers that correspond to capsids with desirable properties. More specifically, the fitness prediction system 200 can generate a respective fitness score 104 for each capsid monomer in the capsid monomer library 102. The fitness prediction system 200 can designate a proper subset of the capsid monomers in the capsid monomer library 102 as being "‘target’" capsid monomers 106 based at least in part on the fitness scores 104. For each target capsid monomer 106, viruses that include the capsid corresponding to the target capsid monomer 106 can then be manufactured 108, i.e., physically generated, using appropriate manufacturing techniques.

|0080| The generated viruses can be used in any of a variety of applications, e.g., the generated viruses can be applied as a therapeutic 1 10 to a subject 1 12 to achieve a therapeutic effect in the subject. For instance, the generated viruses can be used as viral vectors to implement gene therapies, as described above. The generated viruses can be applied to any appropriate subject, e.g., a mouse, a cat, a dog, a pig, or a human, to achieve any appropriate therapeutic effect, e.g., treatment of a disease, e.g., hemophilia, leukemia, melanoma, etc.

[0081] In a particular example, viral vectors (e.g., AAV vectors, e.g., recombinant AAV vectors) that include target capsids selected using the fitness prediction system can be generated and administered to subjects for gene therapy, as described above.

[0082] The fitness prediction system 200 can select a proper subset of the capsid monomers in the capsid monomer library to be designated as target capsid monomers in any of variety of possible ways. For instance, the fitness prediction system 200 can designate any capsid monomer having a fitness score 104 that satisfies a predefined threshold as being a target capsid monomer. As another example, the fitness prediction system 200 can designate a predefined number of capsid monomers having the highest fitness scores 104 as being target capsid monomers.

[0083] The fitness prediction system 200 can designate any appropriate number of capsid monomers from the capsid monomer library 102 as being target capsid monomers 106, e.g., 10 capsid monomers, 100 capsid monomers, or 1000 capsid monomers. In some cases, the fitness prediction system 200 designates only a small fraction of the total number of capsid monomers in the capsid monomer library (e.g., <1%. <0. 1%, or less than <0.01% of the total number of capsid monomers in the capsid monomer library) as being target capsid monomers.

[0084] FIG. 2 shows an example fitness prediction system 200. The fitness prediction system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0085] The fitness prediction system 200 is configured to process data defining an amino acid sequence of a capsid monomer 202 to generate a fitness score 104 for the capsid monomer 202. The fitness score 104 characterizes a predicted quality of the capsid corresponding to the capsid monomer 202. For instance, the fitness score 104 can characterize a predicted manufacturability of the capsid, a predicted capacity of viruses that include the capsid to avoid neutralization, a predicted immunoreactivity of the capsid, a predicted capacity' of viruses that include the capsid to penetrate target tissues, a predicted packing capacity of the capsid, or a predicted capacity of viruses that include the capsid to integrate into host genomes, as described above with reference to FIG. 1. Fitness scores 104 generated by the fitness prediction system 200 can be used to screen a library of capsid monomers to identify target capsid monomers having desirable properties, as described above with reference to FIG. 1.

[0086] The fitness prediction system 200 includes an embedding neural network 204, a fitness prediction machine learning model 208, and an error prediction machine learning model 212, which are each described in more detail next.

[0087] The embedding neural network 204 is configured to process data defining the amino acid sequence of the capsid monomer 202 to generate an embedding 206 of the capsid monomer 202.

[0088] The embedding neural network 204 can be configured to process any appropriate representation of the amino acid sequence of the capsid monomer 202. For instance, the embedding neural network 204 can process a representation of the amino acid sequence of the capsid monomer 202 as a sequence of amino acid embeddings. The amino acid embedding at each position in the sequence of amino acid embeddings can represent the identify of the amino at the corresponding position in the amino acid sequence of the capsid monomer. An amino acid embedding representing an amino acid can be, e.g.. a one-hot embedding, or any other appropriate predefined embedding.

[0089] The embedding neural network 204 can have any appropriate neural network architecture that enables the embedding neural network 204 to perform its described functions. In particular, the embedding neural network 204 can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, recurrent layers, attention layers, etc.) in any appropriate number (e.g., 5 layers, 10 layers, 50 layers, etc.) and connected in any appropriate configuration (e.g., as a linear sequence of layers).

[0090] In some implementations, the embedding neural network 204 can be a recurrent neural network that is configured to sequentially process each amino acid embedding in a sequence of amino acid embeddings representing the amino acid sequence of the capsid monomer 202. An example process for generating an embedding 206 of a capsid monomer 202 using an embedding neural network 204 having a recurrent neural network architecture is described in more detail below with reference to FIG. 4.

[0091] In some implementations, the embedding neural network 204 can include a sequence of update blocks, wherein each update block is configured to process a current set of embeddings representing the capsid monomer to generate an updated set of embeddings representing the capsid monomer. Each update block can include, e.g., one more self-attention neural network layers. An example process for generating an embedding 206 of a capsid monomer 202 using an embedding neural network 204 that includes a sequence of update blocks is described in more detail below with reference to FIG. 5.

[0092] The fitness prediction system 200 can train the embedding neural network 204 on a set of training examples, where each training example corresponds to a respective protein and defines an amino acid sequence of the protein. The fitness prediction system 200 can train the embedding neural network, on the set of training examples, using unsupervised learning techniques, i.e., training techniques that do not rely on the training examples being labeled with fitness scores. A few example techniques for training the embedding neural network 204 on a set of training examples using unsupervised learning techniques are described next.

[0093] In some implementations, the fitness prediction system 200 trains the embedding neural network 204 to perform a token unmasking task. More specifically, for each training example, the fitness prediction system 200 can “mask"’ the identity of the amino acid at one or more positions in the amino acid sequence of the protein. (Masking the identity of an amino acid can refer to replacing an amino acid embedding representing the amino acid by a default embedding, as will be described in more detail below with reference to FIG. 5). The fitness prediction system 200 can then train the embedding neural network 204 to process the masked amino acid sequence to predict the identities of the amino acids at the masked positions in the amino acid sequence. The token unmasking task requires the embedding neural network 204 to leam to infer the identities of masked amino acids from the contextual information provided the remaining, unmasked portions of the amino acid sequence. An example process for training an embedding neural network to perform a token unmasking task is described in more detail below with reference to FIG. 7.

[0094] In some implementations, the fitness prediction system 200 trains the embedding neural network to perform a next token prediction task. More specifically, for each training example, the fitness prediction system 200 can designate one or more positions in the amino acid sequence of the protein as being “target” positions. For each target position, the fitness prediction system 200 can train the embedding neural network 204 to leam to predict the identity of the amino acid at the target position based on only the identities of amino acids at preceding positions in the amino acid sequence. The next token prediction task requires the embedding neural network 204 to leam to infer the identity of an ammo acid at a position in an amino acid sequence based on the identities of the amino acids at preceding positions in the amino acid sequence. An example process for training an embedding neural network to perform a next token prediction task is described in more detail below with reference to FIG. 6.

|0095| Training the embedding neural network 204 using unsupervised learning techniques, e.g., to perform tasks such as token unmasking, next token prediction, or both, can cause the embedding neural network 204 to leam to generate embeddings densely encode rich information characterizing proteins, e.g., capsid monomers.

[0096] In some cases, in combination with or as an alternative to training the embedding neural network 204 using unsupervised learning techniques, the fitness prediction system 200 can train the embedding neural network 204 using supervised learning techniques on an auxiliary set of training examples. Each training example can correspond to a respective protein and can define: (i) the amino acid sequence of the protein, and (ii) a label associated with the protein. The label can represent any appropriate feature of the protein, e.g., the structure of the protein, the stability' of the protein, etc.

[0097] The fitness prediction system 200 can jointly train the embedding neural network and a “projection” neural network on the set of supervised training examples. The projection neural network can be configured to process a protein embedding, generated by the embedding neural network, to generate a prediction for the label associated with the protein. [0098] To train the embedding neural network on a supervised training example, the fitness prediction system 200 can process the amino acid sequence of the protein corresponding to the training example using the embedding neural network 204 to generate a protein embedding. The fitness prediction system 200 can then process the protein embedding using the projection neural network to generate the predicted label associated with the protein. The fitness prediction system 200 can evaluate a supervised objective function that measures an error in the predicted label generated for the protein, and backpropagate gradients of the supervised objective function through the projection neural network and into the embedding neural network 204. After the training, the projection neural network can be discarded, and the fitness prediction system 200 can use the trained embedding neural network 204 to generate embeddings for use in generating fitness scores 104.

[0099] The supervised objective function can measure the error in predicted labels generated by the embedding neural network, e.g., as a cross-entropy error, a squared-error, or in any other appropriate manner. Backpropagating gradients of an objective function through a neural network can refer to determining gradients of the objective function with respect to the parameters of the neural network, and then using the gradients to adjust the values of the parameters of the neural network, e.g., using an appropriate gradient descent optimization technique, e.g., Adam or RMSprop.

[0100] The fitness prediction machine learning model 208 is configured to process the embedding 206 of the capsid monomer 202 to generate a fitness score 104 characterizing a predicted quality of the capsid corresponding to the capsid monomer 202.

[0101] The fitness prediction machine learning model 208 can have any appropriate machine learning model architectures that enables the fitness prediction machine learning model 208 to perform its described functions. For instance, the fitness prediction machine learning model can be a parametric machine learning model, such as a neural network model, a random forest model, or a support vector regression model. As another example, the fitness prediction machine learning model can be a non-parametric model, such as a k-nearest neighbor model.

[0102] The fitness prediction system 200 can train the fitness prediction machine learning model on a set of training examples. Each training example corresponds to a respective capsid monomer and defines: (i) an embedding of the amino acid sequence of the capsid monomer, generated using the embedding neural network; and (ii) a target fitness score associated with the capsid monomer. The target fitness score for a capsid monomer represents the prediction target for the fitness prediction system, i.e., the "‘ground truth’" fitness that should be generated the fitness prediction system for the capsid monomer. [0103] The fitness prediction system 200 can train the fitness prediction machine learning model 208 on the set of training examples and their associated target fitness scores using any appropriate supervised machine learning training technique. More specifically, for each training example, the fitness prediction system 200 can train the fitness prediction machine learning model to process the capsid monomer embedding included in the training example to generate a predicted fitness score that matches the target fitness score specified by the training example. The fitness prediction system 200 can train the fitness prediction machine learning model 208 to optimize an objective function that, for each training example, measures an error between: (i) the target fitness score, and (ii) the predicted fitness score generated by the fitness prediction machine learning model 208. The objective function can measure the error in any appropriate manner, e.g., as a squared error or as an absolute error.

[0104] During the training of the fitness prediction machine learning model 208, the fitness prediction system 200 can rebalance the set of training examples based on the target fitness scores associated with the training examples. In particular, the set of training examples may include significantly more training examples associated with low fitness scores than training examples associated with high fitness scores. Training the fitness prediction machine learning model 208 without rebalancing the set of training examples can reduce the accuracy of the fitness prediction machine learning model 208. The fitness prediction system 200 can rebalance the set of training examples in any of a variety of possible ways. A few example techniques for rebalancing the set of training examples are described next.

[0105] In some implementations, the fitness prediction system 200 can rebalance the set of training examples by duplicating training examples associated with high fitness scores (e.g., fitness scores that exceed a threshold) until at least a threshold fraction of the set of training examples are associated with high fitness scores.

[0106] In some implementations, the fitness prediction system 200 can rebalance the set of training data by determining a respective weight factor associated with each training example as a function of the fitness score associated with the training example. For instance, the fitness prediction system 200 can generate the weight factor for a training example by processing the fitness score for the training example using a sigmoid function. The weight factor for a training example can control the impact of the training example on the parameter values of the fitness prediction machine learning model during training, e.g., such that training examples with higher weight factors have a greater impact on the fitness prediction machine learning model. For instance, if the fitness prediction machine learning model is a neural network model, then as part of training the fitness prediction machine learning model on a training example, the fitness prediction system 200 can scale gradients of the objective function by the weight factor. Thus the weight factors can define individualized learning rates for the respective training examples during training of the neural network implementing the fitness prediction machine learning model.

[0107] In some implementations, the fitness prediction system 200 can rebalance the set of training data by determining a respective penalty factor associated with each training example as a function of the fitness score associated with the training example. For instance, the fitness prediction system 200 can generate the penalty factor for a training example by processing the fitness score for the training example using a sigmoid function. The penalty factor for a training example can control the penalty imposed on the fitness prediction machine learning model during training for incorrectly predicting the fitness score for the training example. For instance, the penalty’ factor for a training example can multiplicatively scale a term in the objective function that measures an error in the predicted fitness score generated by the fitness prediction machine learning model 208 for the training example.

[0108] The target fitness scores for the capsid monomers of the training examples can be generated in any appropriate manner, e.g., by physical experiments. A few example techniques for generating target fitness scores for capsid monomers are described next.

[0109] In some implementations, the fitness score for a capsid monomer characterizes the manufacturability of the corresponding capsid. To generate a target fitness score for a capsid monomer, plasmids coding for the capsid can be transfected into one or more cells. Experimental measurements can then be performed to determine: (i) a number of instances of the capsid that are generated by the one or more cells, and (ii) a number of instances of wildtype capsids that are generated by the one or more cells. The fitness score for the capsid can be defined, e.g.. as the ratio of: (i) the number of generated instances of the capsid, and (ii) the number of generated instances of the wild-type capsid.

[0110] In some implementations, the fitness score for a capsid monomer characterizes the capacity of viruses that include the corresponding capsid to avoid neutralization. To generate a target fitness score for a capsid monomer, viruses that include the corresponding capsid can be introduced into an organism, and experimental measurements can be performed to determine number of viruses that are subsequently neutralized. The fitness score for the capsid can be defined, e.g., based on the ratio of: (i) the number of instances of the virus (having the capsid) that are neutralized, and (ii) the total number of instances of the virus (having the capsid).

[oni] In some implementations, the fitness score for a capsid monomer characterizes the immunoreactivity of the capsid. To generate a target fitness score for a capsid monomer, the corresponding capsid can be introduced into an organism and experimental measurements can be performed to determine the resulting immune reaction of the organism. The target fitness score for the capsid can then be defined based on the measured immune reactions.

[0112] In some implementations, the fitness score for a capsid monomer characterizes the capacity of viruses that include the corresponding capsid to penetrate a target tissue. To generate a target fitness score for a capsid monomer, viruses that include the corresponding capsid can be introduced into an organism, and experimental measurements can be performed to determine the number of viruses that penetrate the target tissue. The target fitness score for the capsid monomer can then be defined based on a ratio of: (i) the number of instances of the virus (having the capsid) that penetrate the target tissue, and (ii) the total number of instances of the virus (having the capsid) that were introduced into the organism.

[0113] In some implementations, the fitness score for a capsid monomer characterizes the packing capacity of the corresponding capsid. To generate a target fitness score for a capsid monomer, one or more instances of the corresponding capsid can be generated, and the quantity of genetic material included in the generated capsids can be experimentally measured. The target fitness score for the capsid monomer can then be defined based on the experimentally measured quantities of genetic material included in the corresponding capsids.

[0114] In some implementations, the fitness score for a capsid monomer characterizes the capacity of viruses that include the corresponding capsid to integrate into host genomes. To generate a target fitness score for a capsid monomer, one or more instances of viruses that include the corresponding capsid can be introduced into an organism. Experimental measurements can be performed to determine the number of instances of the virus that successfully integrate into host genomes. The target fitness score for the capsid monomer can then be defined based on a ratio of: (i) the number of instances of the virus (having the capsid) that successfully integrate into host genomes in the organism, and (ii) the total number of instances of the virus (having the capsid) that were introduced into the organism.

[0115] The error prediction machine learning model 212 is configured to process a representation of the amino acid sequence of the capsid monomer 202 to generate a fitness error 214. The fitness error 214 is an estimate of an error in the fitness score 104 generated by the fitness prediction machine learning model 208.

[0116] The fitness prediction system 200 updates the fitness score 104 generated by the fitness prediction machine learning model 208 by combining: (i) the fitness score 104, and (ii) the fitness error 214. The fitness prediction system 200 can combine the fitness score 104 and the fitness error 214, e.g., by summing the fitness score 104 and the fitness error 214. The fitness prediction system 200 can then output the updated fitness score 104 as the predicted fitness score for the input capsid monomer 202.

[0117] In some implementations, the error prediction machine learning model 212 can have a different model architecture than the fitness prediction machine learning model 208. For instance, the error prediction machine learning model 212 may be a linear regression model, while the fitness prediction machine learning model 208 may be a random forest model.

[0118] In some implementations, the error prediction machine learning model 212 can be configured to process a different input (e.g., having a different format) than the fitness prediction machine learning model 208. For instance, the error prediction machine learning model 212 may process a representation of the capsid monomer as a sequence of amino acid embeddings, while the fitness prediction machine learning model 208 may process an embedding 206 of the capsid monomer generated using the embedding neural network 204.

[0119] Configuring the error prediction machine learning model 212 to have a different architecture, or to process a different input, than the fitness prediction machine learning model 208 can encourage these two machine learning models to be “orthogonal’" to one another. More specifically, configuring the models differently can cause each model to perform implicit reasoning that is independent and distinct from the other model, thus diminishing the mutual information between their outputs and maximizing the information gain achieved by combining their outputs to generate the updated fitness score 104.

|0120| The error prediction machine learning model 212 can have any appropriate machine learning model architecture that enables the error prediction machine learning model 212 to perform its described functions. For instance, the error prediction machine learning model can be a parametric machine learning model, such as a neural network model, a random forest model, or a support vector regression model. As another example, the fitness prediction machine learning model can be a non-parametric model, such as a k-nearest neighbor model.

[0121] The fitness prediction system 200 can train the error prediction machine learning model on a set of training examples. Each training example corresponds to a respective capsid monomer and defines: (i) the amino acid sequence of the capsid monomer; and (ii) a target fitness error associated with the capsid monomer. The target fitness error for a capsid monomer represents the prediction target for the error prediction machine learning model, i.e., the “ground truth” fitness error that should be generated the error prediction system for the capsid monomer. The fitness prediction system 200 can generate the target fitness error for a training example by determining an error between: (i) the target fitness score for the training example, and (ii) the predicted fitness score generated by the (trained) fitness prediction machine learning model 208 for the training example.

[0122] The fitness prediction system 200 can train the error prediction machine learning model 212 based on the set of training examples and their associated target fitness errors using any appropriate supervised machine learning training technique. More specifically, for each training example, the fitness prediction system 200 can train the error prediction machine learning model to process data defining the amino acid sequence of the capsid monomer included in the training example to generate a predicted fitness error that matches the target fitness error specified by the training example. The fitness prediction system 200 can train the error prediction machine learning model 212 to optimize an objective function that, for each training example, measures an error between: (i) the target fitness error, and (ii) the predicted fitness error generated by the error prediction machine learning model 212. The objective function can measure the error in any appropriate manner, e.g., as a squared error or as an absolute error.

[0123] FIG. 3 is a flow diagram of an example process 300 for generating a fitness score that characterizes the predicted quality of a capsid. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a fitness prediction system, e.g., the fitness prediction system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 300.

[0124] The system receives data defining an amino acid sequence of a monomer of the capsid (302).

[0125] The system processes the data defining the amino acid sequence of the monomer of the capsid using an embedding neural network, in accordance with values of a set of embedding neural network parameters, to generate an embedding of the monomer of the capsid (304).

[0126] The system processes the embedding of the monomer of the capsid using a fitness prediction machine learning model to generate a fitness score that characterizes a predicted quality of the capsid (306).

[0127] The system processes a representation of the monomer of the capsid using an error prediction machine learning model to generate a fitness error (308). The fitness error is an estimate of an error in the fitness score generated by the fitness prediction machine learning model. [0128] The system updates the fitness score by combining: (i) the fitness score generated by the fitness prediction machine learning model, and (ii) the fitness error generated by the error prediction machine learning model (310).

[0129] The system outputs the updated fitness score (312).

[0130] FIG. 4 is a flow diagram of an example process 400 for generating an embedding of a capsid monomer using an embedding neural network having a recurrent neural network architecture. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a fitness prediction system, e.g., the fitness prediction system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 400.

[0131] The system instantiates a sequence of amino acid embeddings representing the amino acid sequence of the capsid monomer (402). The sequence of amino acid embeddings includes a respective amino acid embedding corresponding to each position in the amino acid sequence of the capsid monomer. An amino acid embedding corresponding to a position in the amino acid sequence of the capsid monomer can represent the identity of the amino acid at the position, e.g., by a one-hot embedding.

[0132] The system sequentially processes each amino acid embedding in the sequence of amino acid embeddings using the embedding neural network having the recurrent neural network architecture to generate an updated hidden state of the embedding neural network (404). More specifically, prior to processing the first amino acid embedding in the sequence of amino acid embeddings, the system initializes the hidden state of the embedding neural network, e.g., to a default hidden state. The embedding neural network then sequentially processes each amino acid embedding in the sequence, starting from the first amino embedding and in accordance with the ordering of the amino acid embeddings in the sequence of amino acid embeddings. Each time the embedding neural network processes an amino acid embedding, the embedding neural netw ork updates the hidden state of the embedding neural network using the amino acid embedding.

[0133] The embedding neural network includes one or more recurrent neural network layers, and the hidden states of the recurrent neural network layers collectively define the hidden state of the embedding neural network. Each recurrent neural network layer can have any appropriate architecture, e.g., a long short-term memory⁷ (LSTM) architecture or a gated recurrent unit (GRU) architecture.

[0134] The system determines the embedding of the capsid monomer of the capsid based on the updated hidden state of the embedding neural network after processing the last amino acid embedding in the sequence of amino acid embeddings (406). For instance, the system can generate the embedding of the capsid monomer by processing the updated hidden state of the embedding neural network using one or more neural network layers, e.g., fully connected layers.

[0135] FIG. 5 is a flow diagram of an example process 500 for generating an embedding of a capsid monomer using an embedding neural network having a neural network architecture that includes a sequence of update blocks. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a fitness prediction system, e.g., the fitness prediction system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 500. [0136] The system instantiates a sequence of amino acid embeddings representing the amino acid sequence of the capsid monomer (502). The sequence of amino acid embeddings includes a respective amino acid embedding corresponding to each position in the amino acid sequence of the capsid monomer. An amino acid embedding corresponding to a position in the amino acid sequence of the capsid monomer can represent the identity of the amino acid at the position, e.g., by a one-hot embedding. Optionally, the system can combine each amino acid embedding with a positional embedding representing the corresponding position in the amino acid sequence of the capsid monomer.

[0137] The system processes the sequence of amino acid embeddings using a sequence of one or more blocks, referred to as update blocks, included in the embedding neural network (504). Each update block is configured to process a sequence of amino acid embeddings, using one or more neural network layers included in the update block, to generate an updated sequence of amino acid embeddings. The first update block can be configured to receive the initial sequence of amino acid embeddings, and each subsequent update block can receive the updated sequence of amino acid embeddings generated by the preceding update block. Each update block can have any appropriate neural network architecture. In some implementations, some or all of the update blocks include one or more self-attention neural network layers.

[0138] The system generates the embedding of the capsid monomer based on one or more of the updated amino acid embeddings generated by a final update block in the sequence of update blocks (506). For instance, the system can designate the updated amino acid embedding corresponding to the last amino acid in the sequence of amino acids of the capsid monomer as being the embedding representing the capsid monomer. As another example, the system can combine (e.g., average or sum) the updated amino acid embeddings generated by the final update block to generate the embedding of the capsid monomer. [0139] FIG. 6 is a flow diagram of an example process 600 for training an embedding neural network to perform a next token prediction task. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a fitness prediction system, e.g., the fitness prediction system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 600.

[0140] The system receives a set of training proteins (602).

[0141] The system performs steps 604 - 608 for each training protein. For convenience, steps 604 - 608 will be described with reference to a particular training protein from the set of training proteins.

[0142] The system designates one or more positions in the amino acid sequence of the training protein as being target positions (604). The system can select the target positions, e.g., randomly, or in accordance with a predefined selection rule.

[0143] The system uses the embedding neural network to generate, for each target positions in the amino acid sequence of the training protein, a respective probability’ distribution over a set of possible amino acids (606). For each target position, the probability distribution generated for the target position only depends on amino acids at preceding positions in the amino acid sequence of the training protein. To generate the probability distribution for a target position, the system can use the embedding neural network to generate an embedding for the target position, and then process the embedding using one or more neural network layers to generate the probability distribution. In some implementations, the embedding neural network is a recurrent neural network, and to generate an embedding for a target position, the embedding neural network sequentially processes amino acid embeddings corresponding to each position preceding the target position to generate an updated hidden state. The system can then designate the updated hidden state of the embedding neural network as being the embedding for the target position.

[0144] The system trains the embedding neural network to optimize an objective function that, for each target position in the amino acid sequence of the training protein, measures an error between: (i) the probability- distribution generated for the target position, and (ii) an identity of an amino acid located at the target position in the training protein (608). The objective function can measure the error, e.g., as a cross-entropy error.

[0145] FIG. 7 is a flow diagram of an example process 700 for training an embedding neural network to perform a token unmasking task. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a fitness prediction system, e.g., the fitness prediction system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 700. [0146] The system receives a set of training proteins (702).

[0147] The system performs steps 704 - 708 for each training protein. For convenience, steps 704 - 708 will be described with reference to a particular training protein from the set of training proteins.

[0148] The system generates a masked amino acid sequence by masking one or more positions in an amino acid sequence of the training protein (704). Masking a position in the amino acid sequence of the training protein can refer to replacing an amino acid embedding representing the identity of the amino acid at the position by a default embedding. The default embedding can be. e.g., a predefined embedding, e.g., an embedding with each entry having value zero. The system can determine which positions to mask in the amino acid sequence, e.g., by randomly selecting positions to be masked, or by selecting positions to be masked in accordance with a predefined rule.

[0149] The system processes data defining the masked amino acid sequence using the embedding neural network to determine, for each masked position in the amino acid sequence of the training protein, a respective probability distribution over a set of possible amino acids (706). To generate the probability' distribution for a masked position, the system can use the embedding neural network to generate an embedding for the masked position, and then process the embedding using one or more neural network layers to generate the probability’ distribution. In some implementations, the embedding neural network includes one or more update blocks that are each configured to process an input sequence of embeddings to generate an updated sequence of embeddings. In these implementations, to generate an embedding for a masked position, the embedding neural network can process the masked sequence of amino acids using the sequence of update blocks of the embedding neural network to generate an updated sequence of amino acid embeddings. Then, for each masked position, the system can designate the updated amino acid embedding for the masked position as being the embedding that is processed to generate the probability’ distribution for the masked position.

[0150] The system trains the embedding neural network to optimize an objective function that, for each masked position in the amino acid sequence of the training protein, measures an error between: (i) the probability' distribution generated for the masked position, and (ii) an identity of an amino acid located at the masked position in the training protein (708). The objective function can measure the error, e.g., as a cross-entropy error. [0151] FIG. 8 provides an illustration of the accuracy achieved by the fitness prediction system described in this specification on the task of predicting fitness scores that characterize capsid manufacturability. The horizontal axis of the plot represents the experimental (ground truth) fitness scores, and the vertical axis of the plot represents the predicted fitness scores. It will be appreciated that the experimental fitness scores and the predicted fitness score are strongly correlated.

[0152] FIG. 9 provides a comparison of the accuracy achieved by the fitness prediction system with and without the error prediction machine learning model. More specifically, the bar on the left represents prediction accuracy when the fitness prediction system does not use the error prediction machine learning model, and the bar on the right represents the prediction accuracy when the fitness prediction system does use the error prediction machine learning model. The accuracy is measured by the Pearson correlation coefficient between the experimental fitness scores and the predicted fitness scores.

[0153] FIG. 10 provides a comparison of the prediction accuracies achieved by the fitness prediction system for different choices of model architecture of the fitness prediction machine learning model. The accuracy is measured by the Pearson correlation coefficient between the experimental fitness scores and the predicted fitness scores.

[0154] FIG. 11 provides a comparison of the prediction accuracies achieved by the fitness prediction system when training data rebalancing is not performed (the bar on the left) and when training data rebalancing is performed (the bar on the right). The accuracy is measured by the Pearson correlation coefficient betw een the experimental fitness scores and the predicted fitness scores.

[0155] FIG. 12 provides a comparison of the prediction accuracies achieved by the fitness prediction machine learning model when the embedding neural network generates capsid monomer embeddings with 1900 dimensions (the bar on the left) and when the embedding neural network generates capsid monomer embeddings with 256 dimensions (the bar on the right). The accuracy is measured by the Pearson correlation coefficient between the experimental fitness scores and the predicted fitness scores.

[0156] This specification uses the term “configured’⁷ in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. [0157] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0158] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0159] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0160] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0161] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0162] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e g., a universal serial bus (USB) flash drive, to name just a few.

[0163] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way' of example semiconductor memory' devices, e g., EPROM, EEPROM, and flash memory' devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0164] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g.. a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0165] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0166] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework.

[0167] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0168] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0169] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0170] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0171] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order show n, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:

1. A method comprising: receiving data defining an amino acid sequence of a monomer of a capsid: processing the data defining the amino acid sequence of the monomer of the capsid using an embedding neural network, in accordance with values of a set of embedding neural network parameters, to generate an embedding of the monomer of the capsid; processing the embedding of the monomer of the capsid using a fitness prediction machine learning model to generate a fitness score that characterizes a predicted quality of the capsid; and outputting the fitness score that characterizes the predicted uality of the capsid.

2. The method of claim 1 , further comprising: processing a representation of the monomer of the capsid using an error prediction machine learning model to generate a fitness error, wherein the fitness error is an estimate of an error in the fitness score generated by the fitness prediction machine learning model.

3. The method of claim 2, further comprising: updating the fitness score by combining: (i) the fitness score generated by the fitness prediction machine learning model, and (ii) the fitness error generated by the error prediction machine learning model.

4. The method of claim 3, wherein updating the fitness score comprises: summing the fitness score generated by the fitness prediction machine learning model and the fitness error generated by the error prediction machine learning model.

5. The method of any one of claims 2-4, wherein the fitness prediction machine learning model has a different model architecture than the error prediction machine learning model.

6. The method of any one of claims 2-5, wherein processing a representation of the monomer of the capsid using the error prediction machine learning model comprises: processing a representation of the amino acid sequence of the monomer of the capsid using the error prediction machine learning model.

7. The method of claim 6, wherein the error prediction machine learning model processes a different model input than the fitness prediction machine learning model.

8. The method of claim 1, wherein processing the data defining the amino acid sequence of the capsid using the embedding neural network comprises: instantiating a respective amino acid embedding corresponding to each position in the amino acid sequence of the monomer of the capsid; processing the amino acid embeddings by one or more neural network layers of the embedding neural network to update each amino acid embedding; and generating the embedding of the monomer of the capsid protein based on one or more of the updated amino acid embeddings.

9. The method of claim 8, wherein generating the embedding of the monomer of the capsid protein based on one or more of the updated amino acid embeddings comprises: generating the embedding of the monomer of the capsid protein by combining the updated amino acid embeddings.

10. The method of any one of claims 8-9, wherein processing the amino acid embeddings by one or more neural network layers of the embedding neural network to update each amino acid embedding comprises: processing the amino acid embeddings using one or more self-attention neural network layers of the embedding neural network.

11. The method of any one of claims 1-7, wherein processing the data defining the amino acid sequence of the monomer of the capsid using the embedding neural network comprises: instantiating a sequence of amino acid embeddings that comprises a respective amino acid embedding corresponding to each position in the amino acid sequence of the monomer of the capsid; and sequentially processing each amino acid embedding in the sequence of amino acid embeddings using a recurrent neural network to generate an updated hidden state of the recurrent neural network; and determining the embedding of the monomer of the capsid based on the updated hidden state of the recurrent neural network.

12. The method of any preceding claim, wherein the embedding neural network has been trained, on a set of training proteins, to perform a token unmasking task.

13. The method of claim 12, wherein training the embedding neural network to perform the token unmasking task comprises, for each training protein: generating a masked amino acid sequence by masking one or more positions in an amino acid sequence of the training protein; processing data defining the masked amino acid sequence using the embedding neural network to determine, for each masked position in the amino acid sequence of the training protein, a respective probability distribution over a set of possible amino acids; and training the embedding neural network to optimize an objective function that, for each masked position in the amino acid sequence of the training protein, measures an error between: (i) the probability distribution generated for the masked position, and (ii) an identity of an amino acid located at the masked position in the training protein.

14. The method of any one of claims 1-11, wherein the embedding neural network has been trained, on a set of training proteins, to perform a next token prediction task.

15. The method of claim 14, wherein training the embedding neural network to perform the next token prediction task comprises, for each training protein: generating, for each of one or more target positions in an amino acid sequence of the training protein, a respective probability distribution over a set of possible amino acids using the embedding neural network, wherein for each target position, the probability distribution generated for the target position only depends on amino acids at preceding positions in the amino acid sequence of the training protein; and training the embedding neural network to optimize an objective function that, for each target position in the amino acid sequence of the training protein, measures an error between: (i) the probability distribution generated for the target position, and (ii) an identity of an amino acid located at the target position in the training protein.

16. The method of any preceding claim, wherein the fitness prediction neural network has been trained on a set of training examples, wherein each training example corresponds to a respective training monomer of a capsid and includes: (i) an embedding of the training monomer of the capsid, and (ii) a target fitness score for the training monomer of the capsid.

17. The method of claim 16, wherein training the fitness prediction neural network on the set of training examples comprises rebalancing the set of training examples based on the target fitness scores specified by the training examples.

18. The method of any preceding claim, wherein the fitness prediction machine learning model is a parametric machine learning model.

19. The method of claim 18, wherein the fitness prediction machine learning model comprises one or more of: a neural network model or a random forest model or a support vector regression model.

20. The method of any one of claims 1-17, wherein the fitness prediction machine learning model is a non-parametric model.

21. The method of claim 20, wherein the fitness prediction machine learning model comprises a k-nearest neighbor model.

22. The method of any one of claims 1-21, wherein the predicted quality of the capsid characterizes a predicted manufacturability of the capsid.

23. The method of claim 22, wherein the manufacturability of the capsid is defined based on a number of instances of the capsid that are generated when a plasmid coding for the capsid is transfected into one or more cells.

24. The method of claim 23, wherein the number of instances of the capsid that are generated when a plasmid coding the capsid is transfected into one or more cells is normalized by a number of generated wild-type sequences.

25. The method of any one of claims 1-21, wherein the predicted quality⁷ of the capsid characterizes a predicted capacity of viruses that include the capsid to avoid neutralization.

26. The method of any one of claims 1-21, wherein the predicted quality of the capsid characterizes a predicted immunoreactivity⁷ of the capsid.

27. The method of any one of claims 1-21, wherein the predicted quality of the capsid characterizes a predicted capacity of viruses that include the capsid to penetrate a target tissue.

28. The method of any one of claims 1-21, wherein the predicted quality of the capsid characterizes a packing capacity of the capsid.

29. The method of any one of claims 1-21, wherein the predicted quality of the capsid characterizes a predicted capacity of viruses that include the capsid to integrate into host genomes.

30. The method of any preceding claim, wherein the capsid corresponds to a capsid of an adeno-associated virus (AAV).

31. The method of any preceding claim, further comprising: selecting a capsid monomer from a set of possible capsid monomers using the fitness prediction machine learning model; and physically generating one or more viruses that include a protein having the selected capsid protein monomer.

32. The method of claim 31, wherein selecting a capsid monomer from a set of possible capsid monomers using the fitness prediction machine learning model comprises: generating a respective fitness score for each capsid monomer in the set of possible capsid monomers using the fitness prediction machine learning model; and selecting a capsid monomer based on the fitness scores.

33. The method of claim 32, wherein selecting a capsid monomer based on the fitness scores comprises: selecting a capsid monomer having a highest fitness score from among the capsid monomers included in the set of possible capsid monomers.

34. The method of any one of claims 31-33, further comprising: applying the generated viruses to a subject to achieve a therapeutic effect in the subject.

35. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1 -34.

36. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-34.