EP4677600A1 - System and method for generating protein sequences - Google Patents

Abstract

The present invention relates to systems and methods for generating protein sequences using machine learning model. The training method including: training an encoder artificial neural network to map amino-acid-level representations into latent variables in a latent space, the amino-acid-level representations generated by a pre-trained protein language model using received input training samples as input; training a decoder artificial neural network to reconstruct protein sequences using the mapped latent variables as input, the decoder artificial neural network and the encoder artificial neural network trained using determination of a loss between the reconstructed protein sequences and the naturally-occurring protein sequences; and outputting the trained encoder artificial neural network and the trained decoder artificial neural network.

Description

SYSTEM AND METHOD FOR GENERATING PROTEIN SEQUENCES

TECHNICAL FIELD

[0001] The present invention relates to tools for protein design; and more particularly, to a system and method for generating protein sequences.

BACKGROUND

[0002] As an increasing number of diseases have been associated with specific molecular pathogenesis, and gene therapy is gaining more attention as it provides targeted treatment from the source. Efficient and targeted gene delivery can greatly influence treatment effect. Different delivery vehicles have been designed to increase efficiency and specificity. Viral vectors are an attractive class of tools for gene delivery, since viruses are highly evolved to efficiently gain access to the host. Viral vectors have been engineered to exclude the replication and toxicity related coding regions for safety consideration. There are several commonly used viral vectors, including lentivirus, retrovirus, herpes simplex virus 1 , adenovirus (Ad), and adeno-associated virus (AAV).

[0003] Compared to other viral vectors, adenovirus vectors are known to have several advantages for gene therapy applications: 1 ) the delivered DNA exists episomally, avoiding potential oncogenesis caused by integration; 2) adenovirus vectors have a large packaging capacity of 36kb, which are promising for delivery gene circuits or treating diseases involving pathogenic mutation of multiple genes; 3) adenovirus can extremely efficiently transduce most dividing and quiescent cells, including rat CNS cells (20-60% efficiency), human embryonic lung cells (80% efficiency), human squamous carcinoma (75% efficiency), pancreatic tumor cells (65% efficiency); 4) Adenovirus vectors can be scaled in production. However, with their natural coexistence with humans, wild-type virus vectors often trigger complex immunological reactions and hinder the treatment efficiency; meaning this type of pre-existing immunity is a significant limitation for the application of adenovirus vectors. Pre-existing neutralizing antibody against the most used adenovirus strain (Ad5) was found in over 30% of tested sera samples from Europe, Asia, and the United States. It has also been determined that anti-Ad2 neutralizing activity existed in over 40% of tested children’s sera samples, and 74% of children’s sera samples contained neutralizing antibodies for at least one serotype. In addition, transient expression is typically observed for genes delivered episomally, and therefore, multiple rounds of adenovirus-mediated therapy may be required.

[0004] It is therefore an object of the present invention to provide a system and method in which the above disadvantages are obviated or mitigated, and attainment of various desirable attributes is facilitated.

SUMMARY

[0005] In an aspect, there is provided a computer-implemented method for developing machine learning models for generation of protein sequences, the method comprising: receiving input training samples comprising naturally-occurring protein sequences; training an encoder artificial neural network to map amino-acid-level representations into latent variables in a latent space, the amino-acid-level representations generated by a pre-trained protein language model using the received input training samples as input, an encoder comprises the encoder artificial neural network and the protein language model; training a decoder artificial neural network to reconstruct protein sequences using the mapped latent variables as input, a decoder comprises the decoder artificial neural network, the decoder artificial neural network and the encoder artificial neural network trained using determination of a loss between the reconstructed protein sequences and the naturally-occurring protein sequences; and outputting the trained encoder artificial neural network and the trained decoder artificial neural network.

[0006] In a particular case of the method, the encoder further comprises a bottleneck attention unit to refine the output feature vectors of the encoder artificial neural network.

[0007] In another case of the method, the encoder further comprises an encoder linear to change dimensionality of the refined output features.

[0008] In yet another case of the method, the decoder further comprises one or more self-attention units to refine the probabilities of amino acids to capture dependencies between amino acid usage at different positions or to capture interactions between different types of amino-acids. [0009] In yet another case of the method, the decoder further comprises a decoder linear to convert the features outputted by the decoder artificial neural network to logits, and wherein the decoder further comprises a softmax activation function to determine the probabilities of amino acids at positions in the sequence from the logits.

[0010] In yet another case of the method, the decoder artificial neural network comprises one or more blocks comprising one-dimensional convolutional layers and one-dimensional transposed convolutional layers.

[0011] In yet another case of the method, the encoder artificial neural network comprises one or more one-dimensional convolutional layers.

[0012] In yet another case of the method, the encoder artificial neural network and the decoder artificial neural network are jointly trainable in a single stage with the same training dataset.

[0013] In yet another case of the method, the generated protein sequences comprise novel synthetic human adenovirus serotypes.

[0014] In another aspect, there is provided a system for developing machine learning models for generation of protein sequences, the system comprising one or more processors and a memory comprising instructions for the one or more processors to execute: an input module to receive input training samples comprising naturally- occurring protein sequences; an encoder module to train an encoder artificial neural network to map amino-acid-level representations into latent variables in a latent space, the amino-acid-level representations generated by a pre-trained protein language model using the received input training samples as input, an encoder comprises the encoder artificial neural network and the protein language model; a decoder module to train a decoder artificial neural network to reconstruct protein sequences using the mapped latent variables as input, a decoder comprises the decoder artificial neural network, the decoder artificial neural network and the encoder artificial neural network trained using determination of a loss between the reconstructed protein sequences and the naturally- occurring protein sequences; and an output module to output the trained encoder artificial neural network and the trained decoder artificial neural network. [0015] In a particular case of the system, the encoder further comprises a bottleneck attention unit to refine the output feature vectors of the encoder artificial neural network.

[0016] In another case of the system, the encoder further comprises an encoder linear to change dimensionality of the refined output features.

[0017] In yet another case of the system, the decoder further comprises one or more self-attention units to refine the probabilities of amino acids to capture dependencies between amino acid usage at different positions or to capture interactions between different types of amino-acids.

[0018] In yet another case of the system, the decoder further comprises a decoder linear to convert the features outputted by the decoder artificial neural network to logits, and wherein the decoder further comprises a softmax activation function to determine the probabilities of amino acids at positions in the sequence from the logits.

[0019] In yet another case of the system, the decoder artificial neural network comprises one or more blocks each comprising one or more one-dimensional convolutional layers and one or more one-dimensional transposed convolutional layers.

[0020] In yet another case of the system, the encoder artificial neural network comprises one or more one-dimensional convolutional layers.

[0021] In yet another case of the system, the encoder artificial neural network and the decoder artificial neural network are jointly trainable in a single stage with the same training dataset.

[0022] In yet another case of the system, the generated protein sequences comprise novel synthetic human adenovirus serotypes.

[0023] In yet another aspect, there is provided a computer-implemented method for receiving input samples comprising naturally-occurring family of protein sequences; selecting a subset of sequences from the family of protein sequences; mapping, using an encoder, the selected subset of input samples into latent vectors in a latent space, the encoder comprising a protein language model to generate amino-acid-level representations from the selected subset of input samples and a trained encoder artificial neural network to output latent vectors from the amino-acid-level representations; generating, using a decoder, protein sequences in the family of protein sequences using the mapped latent vectors as input to a trained decoder artificial neural network, the generated protein sequences comprising probabilities of amino acids at positions in the respective sequence; and outputting the generated protein sequences.

[0024] In a particular case of the method, selecting the subset of sequences comprises performing clustering on the received family of protein sequences and selecting a representative sequence from each cluster.

[0025] In another case of the method, selecting the subset of sequences comprises sampling from a distribution of the family of protein sequences.

[0026] In yet another case of the method, the method further comprising performing analysis of a mean, or interpolation analysis, on the latent vectors to determine which mapped latent vectors are provided to the decoder.

[0027] In yet another case of the method, the method further comprising, at each position in the generated sequence, selecting the amino acid with highest probability.

[0028] In yet another case of the method, the method further comprising filtering out generated sequences that are more repetitive than naturally-occurring sequences.

[0029] In yet another case of the method, the generated protein sequences comprise novel synthetic human adenovirus serotypes.

[0030] These and other aspects are contemplated and described herein. The foregoing summary sets out representative aspects of systems and methods to assist skilled readers in understanding the following detailed description.

DESCRIPTION OF THE DRAWINGS

[0031] An embodiment of the present invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0032] FIG. 1 is a block diagram showing a system for generating protein sequences, according to an embodiment;

[0033] FIG. 2 is a block diagram illustrating an example architecture implemented by the system of FIG. 1 for generating protein sequences; [0034] FIG. 3 is a flowchart showing a method for training a model for generation of protein sequences, according to an embodiment;

[0035] FIG. 4A is a chart illustrating results of example experiments comparing the results of amino-acid pair association scores for the system of FIG. 1 in comparison to PortGPT2 generated sequences, ESM-IF1 generated sequences and natural sequences;

[0036] FIG. 4B is a chart illustrating shannon-entropy for natural hexons and sequences, in accordance with the example experiments;

[0037] FIG. 4C is a chart illustrating average repetition in sliding windows, in accordance with the example experiments;

[0038] FIG. 4D is a chart illustrating helix and strand ratio in natural and generated hexons, in accordance with the example experiments;

[0039] FIG. 4E is a chart illustrating SASA for all amino acids for generated and natural proteins, in accordance with the example experiments;

[0040] FIG. 5A is a chart illustrating a number of clusters at different identity thresholds, in accordance with the example experiments;

[0041] FIG. 5B is a chart illustrating sequence diversity plotted against secondary structure similarity, in accordance with the example experiments;

[0042] FIG. 5C is a density plot for sequence diversity and secondary structure similarity, in accordance with the example experiments;

[0043] FIG. 5D is a chart illustrating Pareto frontiers, in accordance with the example experiments;

[0044] FIG. 6A is a chart illustrating RMSD for all wild-type cluster representative sequences and stable generated sequences, in accordance with the example experiments;

[0045] FIG. 6B is a chart illustrating RMSF for all wild-type cluster representative sequences and stable generated sequences, in accordance with the example experiments; [0046] FIG. 7A is a chart illustrating latent space clustering of sequences from 10 largest clusters at 90% identity, in accordance with the example experiments;

[0047] FIG. 7B is a chart illustrating average positional likelihood for latent space interpolated sequences and direct data interpolated sequences, in accordance with the example experiments;

[0048] FIGS. 7C and 7D are charts illustrating Hamming distance to ad2 and ad5 hexon, respectively, in accordance with the example experiments;

[0049] FIG. 8 illustrates a flowchart showing a method for generating protein sequences, according to an embodiment;

[0050] FIG. 9 illustrates a diagram of an example architecture for an encoder CNN and a decoder CNN;

[0051] FIG. 10 is a chart illustrating approximated reconstruction F1 scores in a ProtT5 model that are compared between hexon dataset and small PAH dataset extracted with respect to ProT-VAE;

[0052] FIGS. 11 A and 11 B illustrate results of pairwise amino acid divergence for two loop regions in hexons, where darker shading means more difference between the compared sequences;

[0053] FIG. 12 is a chart illustrating a repetition profile for natural protein and sequences generated by and fine-tuned;

[0054] FIGS. 13A and 13B are charts showing Shannon-entropy for natural protein and sequences generated using ProteinVAE in FIG. 13A and ProGen2 in FIG. 13B;

[0055] FIGS. 14A to 14C are charts showing helix and strand ratio in natural and proteins in FIG. 14A, generated by ProteinVAE in FIG. 14B and generated by ProGen2 in FIG. 14C;

[0056] FIG. 15 is a chart showing a number of clusters at different identity thresholds;

[0057] FIG. 16 is a scatter plot for sequence diversity and secondary structure similarity;

[0058] FIG. 17 is a chart showing quantification of virus control and synthetic variants; and [0059] FIG. 18 is a chart showing quantification of adenovirus virus control and synthetic variants.

DETAILED DESCRIPTION

[0060] Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.

[0061] Any module, unit, component, server, computer, computing device, mechanism, terminal or other device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors. [0062] To prevent immunity associated with repeated administration of adenovirus- mediated therapy, different serotypes may be needed for each round. Chemically or biologically modified adenovirus capsid may be used to assist in successful evasion of serotype-specific neutralizing antibodies. As a major capsid protein, hexon may be used as a primary target of neutralizing antibodies. Embodiments of the present disclosure can be used to design protein sequences, such as hexon sequences, that are different from natural sequences to avoid neutralization from pre-existing serotype-specific antibodies.

[0063] Changing an entire solvent exposed surface to the extent that all neutralizing antibody epitopes are removed can involve introducing a large number of mutations (e.g., ~ 452 mutations). Introducing this many mutations randomly, or using rational structure-based design, cannot be done without catastrophic loss of protein function. Therefore, instead of generating new proteins with experimental approaches, the present embodiments can be used to computationally model protein sequences; and ultimately, achieve relatively rapid generation of high quality protein sequences at a relatively low-cost.

[0064] Other approaches to generating computational models to determine novel protein designs generally have a number of substantial problems. For example, such models typically involve a large number of parameters, which requires a large amount of data for training. In an example, models using Generative Adversarial Networks for protein sequence generation can comprise 60 million trainable parameters and require training on 16,706 unique malate dehydrogenase (MDH) sequences. For hexon protein sequences, there are currently only 88 serotypes of human adenovirus identified, so the number of available unique hexon sequences are limited.

[0065] FIG. 1 illustrates a schematic diagram of a system 100 for generating protein sequences and for developing machine learning models for generation of protein sequences, according to various embodiments. The system 100 can be colloquially referred to as ‘ProteinVAE’. As shown, the system 100 has a number of physical and logical components, including a processing unit (“PU”) 160, random access memory (“RAM”) 164, an interface module 168, a network module 176, non-volatile storage 180, and a local bus 184 enabling Pll 160 to communicate with the other components. Pll 160 can include one or more central processing units, one or more graphical processing unit, microprocessors, dedicated hardware, or other integrated processing circuits. RAM 264 provides relatively responsive volatile storage to Pll 260. The interface module 168 enables input to be provided; for example, directly via a user input device, or indirectly, for example, via an external device. The interface module 168 also enables output to be provided; for example, directly via a user display, or indirectly, for example, sent over the network module 176. The network module 176 permits communication with other systems or computing devices; for example, over a local area network or over the Internet. Non-volatile storage 180 can store an operating system and programs, including computer-executable instructions for implementing the methods described herein, as well as any derivative or related data. In some cases, this data can be stored in a database 188. In some cases, during operation of the system 100, the operating system, the programs and the data may be retrieved from the non-volatile storage 280 and placed in RAM 264 to facilitate execution. In other embodiments, any operating system, programs, or instructions can be executed in hardware, specialized microprocessors, logic arrays, or the like. While FIG. 1 illustrates a system implemented on a single computing device, it is understood that the processing, or any of the functions undertaken by the system 100, can be distributed over multiple computing devices; for example, in a cloud or distributed computing environment.

[0066] In an embodiment, the Pll 160 can be configured to execute a number of conceptual modules 101 ; for example, an input module 102, an encoder module 104, a decoder module 106, a preprocessing module 108, a latent processing module 110, a post-processing module 112, and an output module 114. In further cases, functions of the above modules can be combined or executed on other modules. In some cases, functions of the above modules can be executed on remote computing devices, such as centralized servers and cloud computing resources communicating over the network module 176.

[0067] Given the limited amount of training data available for certain protein families, the present inventors developed a model used by the system 100 that can be effectively trained on a small training dataset. FIG. 2 illustrates an example architecture implemented by the system 100 for generating protein sequences. The system 100 can implement a Variational Autoencoder (VAE) that is composed of an encoder and a decoder. The encoder module 104 implements the encoder that, in some cases, can use an attention mechanism with fixed query vector to extract bottleneck representation z from a pretrained language model hidden representation H. If the pretrained language model was trained on a classification task, a [CLS] token embedding can be used as the query vector. If not, convolutional networks can be implemented to extract crude protein-level representation to serve as the query vector. The decoder module 106 implements the decoder that, in some cases, can use deconvolutional networks for upsampling. Attention blocks can be applied on both dimensions.

[0068] As illustrated in the example of FIG. 2, the encoder includes a pre-trained protein language model (PPLM), in this case ProtBert, receives a protein sequence vector; for example, a one-hot encoded protein sequence. The language model produces high- quality amino-acid-level representations in the form of a matrix of size [L, d_pre]; where L is maximum sequence length in a training dataset, d_pre is a hidden dimension of the PPLM, (for ProtBert, d_pre is 1024). Large protein language models that are pre-trained on a large protein database usually have the ability to provide high-quality amino-acid- level representation (which may already capture certain features of the protein). Using the language model, the task of the encoder is effectively simplified such that it only needs to extract protein-level representations (VAE latent vectors) from these high quality amino-acid-level representations. Advantageously, this simplification allows other portions of the model to be built with limited number of parameters that can be trained on a relatively smaller dataset.

[0069] While the example of FIG. 2 describes using ProtBert, any pre-trained protein language model can be used, as long as the language model is pre-trained on sequences longer than the protein family of interest and the output is a matrix by- positional representation (each amino acid in sequence is represented as a column/vector in the matrix). The PPLM does not have to be a transformer-based model, but transformer-based models are generally preferable due to their performance capabilities. [0070] In the example of FIG. 2, the encoder also includes an encoder convolutional neural network (CNN) that takes the matrix of amino-acid-level representations as input. The encoder CNN structure extracts feature maps with reduced dimensionality, which effectively capture local features. The extracted local feature map can then be flattened and passed through a linear layer to obtain a crude global vector. The global vector has a length of d_pre; where d_pre can be the hidden dimension of the PPLM (d_pre in ProtBert is 1024). Generally, most protein language models, such as ProtBert, were not trained on a classification task, and therefore, do not have a trained CLS token needed for a query vector in a bottleneck attention unit. To create a vector with at least some sequencelevel information, CNNs can be used to condense the amino-acid level representation to a single vector.

[0071 ] In the example of FIG. 2, the encoder also includes a bottleneck attention unit to refine the crude CNN-extracted feature vector. The CNN-extracted feature vector is received as the input query and the key and value input is the pre-trained language model amino-acid level representation. The bottleneck attention unit can output a vector of the length of d_pre; where d_pre is the hidden dimension of the PPLM (d_pre in ProtBert is 1024). Using the bottleneck attention unit can generate better global representation that allows for relatively easier downstream sentence-level (or global) tasks (as tested on sentence similarity, classification, and generation tasks).

[0072] In the example of FIG. 2, the encoder also includes an encoder linear to adjust (increase or decrease) the dimensionality of the global vector. The encoder linear outputs a vector of the length of den, (for example, for a hexon dataset, den is 128). den is typically smaller than d_pre, as it is beneficial for interpretability of the latent vector (which has been verified with clustering by sequence identity in the example of FIG. 7A). However, for some datasets, the optimal den may need to be larger for better generation performance. Without this linear layer, the length must be fixed to the d_pre, which is the hidden dimension of the PPLM.

[0073] As illustrated in the example of FIG. 2, the decoder includes a decoder CNN, which takes as input the output of the encoder linear. In a particular case, the decoder CNN includes 1 D convolution layers of the decoder CNN (Convl d) to reduce the input number of channels to growth rate (d_g), which reduces number of parameters in the following ConvTransposel d layer with bigger kernel. Also included are 1 D transposed convolution layers (ConvTransposeld), or deconvolution layers, are used to upsample a feature map. Also included are linear layers in Upblock are used to create identity transforms of the input that can be concatenated to the ConvTransposel d output, for maintaining gradient. The output of the decoder CNN can be a matrix of size [L, dde], where L is maximum sequence length in training dataset, dde is the hidden dimension of the decoder (for the example of a hexon dataset, dde is tuned to be 128).

[0074] In the example of FIG. 2, the decoder also includes a position-wise multihead self-attention unit that receives the output of the decoder CNN and captures interaction of between two amino acids that occur at a long distance. The output is a matrix of size [L, dde], where L is maximum sequence length in training dataset, dde is the hidden dimension of the decoder (for the example of the hexon dataset, dde is tuned to be 128). For certain protein sequences, such as for hexons, which are a long protein, there can be complex patterns of long-distance interactions.

[0075] In the example of FIG. 2, the decoder also includes a decoder linear to transform decoder feature matrix M to crude logits. The decoder linear outputs a matrix of size [L, 21 ], where L is maximum sequence length in training dataset, and 21 is used because it is the number of standard amino acids + 1 (gap). Converting feature matrix to logits is used for determining which amino acid is at each position.

[0076] In the example of FIG. 2, the decoder also includes an amino-acid-wise multihead self-attention unit to refine the logits to capture interactions between different types of amino-acids. The output is a matrix of size [L, 21 ], where L is maximum sequence length in training dataset, and 21 is used because it is the number of standard amino acids + 1 (gap). The self-attention unit can be used to refine logits to incorporate amino-acid usage differences that are observed between different viruses. For a hidden matrix X of size T x d (T is the length, d is the hidden size), self-attention is

Attciitioii(Q. K, I' ) — softmaxf where the queries Q (size T x d), keys K (size T x ) and values V (size T x d) are all linearly transformed X.

[0077] In the example of FIG. 2, the logits outputted by the self-attention unit can be passed through a softmax activation function to obtain a probability of each amino acid at each position; whereby the amino acid with the highest probability can be assigned to each position (such as using an argmax operation). The output of the softmax activation function, after the argmax operation, is a sequence vector that is outputted by the decoder.

[0078] A pre-trained protein language model can be incorporated in the encoder. The encoder module 104 can use a convolutional neural network to extract a feature vector from the hidden amino-acid-level representation produced by the pre-trained protein language model. ‘Amino-acid-level’ of the hidden representation is generally used to indicate that matrix M has the same length as the maximum sequence length L, where each position in the sequence is represented as a column in M. Using the CNN- extracted feature vector as the query in the bottleneck attention unit, global-level information was integrated to obtain a refined protein-level representation. Instead of training from scratch on potentially limited training data, the parameters in the pretrained model can be kept fixed. This permits use of potentially high-quality amino-acid-level embeddings and can allow for more efficient training on a smaller training dataset. In an example, ProteinBERT can be used as the pre-trained protein language model because it performs well on sequences with shorter length ranges (e.g., 1024 amino acids); however, any suitable pre-trained protein language model can be used. Additionally, while the convolutional neural network can be used to extract the feature vector, any suitable artificial neural network can be used.

[0079] In some cases, such as where the sequence dataset is relatively small (for example, for hexon sequences), the pre-trained protein language model can be run on all sequences once, and the generated amino-acid-level representations can be stored. Advantageously, no further use of the pre-trained protein language model is generally required for training and generation of the encoder/decoder model, which allows for relatively fast training and generation. [0080] For the decoder, as implemented by the decoder module 106, non- autoregressive processing can be used. In some cases, deconvolutional layers can be used to upsample the bottleneck representation, i.e. , protein-level representation, to amino-acid-level representation M of size L x dde, where L means maximum sequence length in training dataset and dde means decoder hidden dimension. For example, a convolution layer with kernel size of 1 x 1 separated deconvolutional layers with a bigger kernel; which provides the reduced number of parameters as potentially needed. The decoder module 106 can then use a position-wise multi-head attention mechanism to capture the dependencies between amino acid usage at different positions; which allows effective modeling of long-distance interactions. The decoder module 106 can then use a linear layer to convert the hidden representation M to a logits matrix. Another multi-head attention block can be used to adjust for amino acid preference in different viruses, and therefore, can be applied across different amino-acid channels. In some cases, more helix sequences can be generated than strand; and in such cases, because strand proteins are generally harder to design, a reweighted cross-entropy loss, which assigns higher penalty to the strand positions, can be used.

[0081] Advantageously, the VAE, comprising the components of the architecture exemplified in FIG. 2 from the encoder CNN to the output of the decoder (e.g., at least the encoder artificial neural network and the decoder artificial neural network), can be trained in a single stage with a single training dataset. After receiving amino-acid-level representations generated by the pre-trained protein language model, the VAE can be trained with evidence lower bound (ELBO) loss. In a particular case, the cross-entropy loss in ELBO is reweighted according to a secondary structure in order to penalize wrong predictions at any position that forms strand more than other positions; and a KL- divergence can be dynamically weighted during training with a feedback control to avoid a KL-vanishing problem. The trained VAE encoder can be used to map naturally- occurring protein sequences to the structured latent space that have a similar distribution as the prior distribution.

[0082] Turning to FIG. 3, a flowchart of a method for developing machine learning models for generation of a protein sequence 300, according to an embodiment, is shown. [0083] At block 302, the input module 102 receives input training samples comprising naturally-occurring protein sequences to be mapped to the latent space using the encoder. The received protein sequences, in some cases, can be one-hot encoded; however, any suitable encoding can be used. In some cases, the input data samples can include a particular family of naturally-occurring protein sequences; which can, for example, be retrieved from an annotated database of protein sequences (for example, full-length hexon protein sequences annotated in the UniprotKB database).

[0084] In an example, the naturally-occurring protein sequences can comprise hexon proteins, which are the major capsid protein in adenovirus with a length spanning from 893 to 992 amino acids and with an average length of 938 amino acids. In this example, to increase the chances of generating complete sequences covering all domains, the input module 102 can receive, or filter, only full-length hexon proteins annotated in the UniprotKB database. These sequences can then be filtered for those shorter than 800 amino acids for quality purposes. Additionally, for ease of downstream application, sequences with non-standard amino acids (U, J, Z, O, B, X) can be filtered. In total, in this example, 711 hexon sequences can be collected as training data. In this example, a training/validation/test set splits ratio can be 7/2/1. Training split can be randomly shuffled and further split into smaller batches of training samples at the beginning of each training epoch. A training epoch is generally one complete iteration through the entire received training data during the training process. Validation data split can be randomly shuffled once and divided into smaller batches at the beginning of the training. The same validation batches are used in each training epoch.

[0085] At block 304, the encoder module 104 uses the encoder I ^x a machine learning model parameterized , to map the input training data samples x into latent variables z (in a latent space). In some cases, the mapping can be assumed to follow a Gaussian distribution as its prior.

[0086] In a particular case, the machine learning model of the encoder comprises a transformer-based featurizer that includes a language learning model and dimension reduction model implemented with convolutional neural networks; however, any suitable architecture and/or structure can be used. In an example, the language learning model can be pre-trained using a Uniref O feature set (including 216 million protein sequences), trained with 300,000 steps on sequences of less than 512 aa in length and 100,000 steps on sequences of less than 2,000 aa in length.

[0087] In some cases, when KL-divergence decreases too much, generated samples can suffer from low diversity. To prevent KL-vanishing and to allow effective manipulation of the impact of KL-divergence, a non-linear proportional-integral- derivative (PID) controller can be used to automatically tune the weight of KL- divergence in the VAE objectives throughout training. The KL-divergence weight /3(f) (beta(t)) is calculated through a feedback control defined as: where e(f) is the error between actual and expected value at time t, and K_p, Ki, and Kd are the coefficients for the proportional, integral, and derivative term, respectively.

[0088] The encoder module 104 can use a bottleneck attention:

3(H 5) = MultiHead{q, K V) where keys K (size T x d) and values V (size T x d) are transformed from output hidden representations of the pre-trained language model H (T x d). The parameter t) includes the weights for transformation of the query, keys, and values. During training, the pretrained language model can stay frozen, where only the parameter d is learned.

[0089] The bottleneck attention unit can be used to generate better global representation, which can allow easier downstream sentence-level tasks (for example, the present inventors tested on sentence similarity, classification, and generation tasks). A high-quality global representation is beneficial for faithful reconstruction because decoder reconstruction generally depends on the quality of the information contained in the global representation. The value of the bottleneck attention unit in the present embodiments is further illustrated in the example ablation study described herein. The version of the model without the bottleneck attention unit generated worse sequences, where larger differences were observed in the amino-acid usage, secondary structure profile, and MSA entropy profile (amino-acid diversity across all positions) when compared to natural sequences.

[0090] In some cases, restrictions of the length of the protein sequences limits the pretrained language models that are available. In an example, ProteinBert (420 million parameters) can be used as the pre-trained protein language model because it is trained with a length limit of 2048 amino acids. Additionally, this language model generally achieves better results on downstream tasks than other models trained on long protein sequences. In such example, ProteinBert contains thirty layers and is trained for three-hundred thousand steps on sequences shorter than 512 amino acids, then for an additional one-hundred thousand steps on sequences with a maximum length of two thousand.

[0091] At block 306, the decoder module 106 uses the decoder to reconstruct input protein sequences x from the latent variable z outputted by the encoder of the encoder module 104. The decoder can be another machine learning model parameterized by Q. The present inventors have determined that protein sequences can be substantially better modeled with a non-sequential processing approach (also referred to as a non-autoregressive approach; whereby all positions can be processed simultaneously).

[0092] In a particular case, the decoder can include a linear layer and eight deconvolution blocks to perform inverse convolution operations on reduced channels. The deconvolutional network increases the spatial size of the input, while decreasing the number of hidden dimensions. To maintain gradient, the bottleneck representation and the output of each previous deconvolutional block can be concatenated as the input for the next block. In some cases, deconvolutional networks output M may not be passed to a recurrent neural network (RNN); instead, a multi-head attention block can be used to capture both short and long range relationships. The decoder module 106 can convert the output of the attention block to logits using a linear layer.

[0093] In some cases, another amino-acid wise attention block can be used to capture the amino acid usage preference among different viruses. The trained attention modules described herein allow soft and dynamic rules of importance to be applied on each sample. Thus, it can identify the important regions and increase their weight while assigning lower weights to unimportant regions. A particular uniqueness of the amino- acid-wise self-attention is that it operates on crude logits and not on a latent matrix. In this case, crude logits means that when softmaxed, they can be converted to probabilities of each amino acid at each position. In an example, an attention map can be calculated on 21 vectors (where length is maximal seq length) that represent logits of each amino acid across all positions. Therefore, for each amino acid, the attention mechanism learns which other amino acids are important to determine its usage and adjust its logits accordingly.

[0094] In a particular case, the machine learning model of the decoder includes a dimension decompression component comprising a transposed convolution neural network. To project the decompressed dimensions to generated protein sequences, the transposed convolution neural network can be followed by multi-head attention layers; for example, a position-wise multi-head attention layer, followed by a linear layer, and followed by an amino-acid-wise multi-head attention layer.

[0095] In other cases, autoregressive processing can be used by replacing the deconvolutional networks with a multi layer long short-term memory (LSTM) recurrent neural network (RNN). Both single direction and bidirectional LSTM models can be used. LSTM hidden sizes can be divided by 2 when testing bidirectional LSTM. The hidden dimensions of the bottleneck representation z and the upsampled decoder hidden representation M can be kept the same.

[0096] In some cases, two or more of the convolutional neural network of the encoder, the bottleneck attention layer, the encoder linear layer, the transposed convolutional neural network of the decoder, the position-wise attention layer of the decoder, the decoder linear layer, and the amino-acid wise attention layer of the decoder can be trained on the same training dataset in a single stage. For example, for predicting hexon sequences, a training dataset comprising 711 naturally occurring hexon sequences of length between 893 to 992 aa. The training objective being cross-entropy (secondarystructure reweighted) + KL-divergence; however any suitable training objective can be used. [0097] At block 308, in some cases, the post-processing module 112 can determine a loss and a reconstruction score for the training and validation samples. In a particular case, the loss can be a negative evidence lower bound (ELBO) and a reconstruction F1-score can be determined on each batch of the training and validation samples. Specifically, the ELBO can be calculated as: Pe ( l )] is the expected conditional log-likelihood, p(z) is the prior

Gaussian distribution, and DKL(qq>(z\x)\\p(z)) is the Kullback-Leibler (KL) divergence. In most cases, the VAE can be trained by maximizing ELBO (or minimizing negative ELBO loss). In other cases, other suitable loss and reconstruction approaches can be used.

[0098] At block 310, in some cases, to minimize loss, the post-processing module 112 can use an Adam optimizer to update the encoder and decoder parameters on each batch of training samples. Adam (Adaptive Moment Estimation) is an optimization technique for training deep neural networks. It is a variant of stochastic gradient descent (SGD) that computes adaptive learning rates for each parameter based on estimates of the first and second moments of the gradients. Hyperparameters of the Adam optimizer can be tuned for optimal performance. It is understood that other suitable optimizers can be used.

[0099] In some cases, blocks 304, 306, 308, and 310 can be performed iteratively until an end condition signifies the end of training. At block 312, at the end of training, the output module 114 can output the trained encoder and decoder models to the interface module 168, the network module 176 and/or the database 188. The output can include the encoder and decoder parameters. In some cases, the parameters can be stored as a binary checkpoint file. Different criteria can be used to determine when to stop training. For example, if the loss on the validation data split or the validation reconstruction F1 -score has converged, the training can be terminated and a checkpoint file can be outputted.

[0100] FIG. 8 is a flowchart illustrating a method for protein sequence generation 800, in accordance with another embodiment. Particularly, the method 800 includes sampling or manipulation of latent space to produce new latent vectors to be decoded into new protein sequences.

[0101] At block 802, the input module 102 receives input data samples comprising a naturally-occurring family of protein sequences.

[0102] At block 804, in some cases, the preprocessing module 108 selects a subset of sequences from the family of protein sequences which are to be provided to the trained encoder. In a particular case, the selection by the preprocessing module 108 can include performing clustering on the received family of protein sequences. In an example, the received natural sequences, in this example hexons, are clustered at 90% sequence identity (in this example, sequence identity is determined using MMSeqs2). All sequences for each of the 13 largest clusters (e.g., size >10) were passed to the encoder.

[0103] At block 806, the encoder module 104 uses the trained encoder , as described herein, being the machine learning model parameterized by , to map the selection of samples x into latent variables z (in the latent space).

[0104] In other cases, if generation of sequences similar to a specific subset of natural sequences is not required or desired, selecting of the subset of sequences (at block 804) and encoding them (at block 806) can be skipped; whereby direct sampling from the prior distribution can be performed; for example, sampling from a Gaussian distribution with mean of 0 and standard deviation of 4. Such a large standard deviation can be used to effectively increase sequence diversity. Other suitable mean and standard deviation determinations can be used.

[0105] At block 808, in some cases, the latent processing module 110 performs analysis on the latent vectors. In a particular case, where the preprocessing module 108 performed clustering, the latent processing module 110 can determine a mean latent vector within each cluster. For example, for each of the 13 largest clusters, 50000 latent vectors can be sampled from a Gaussian distribution using the mean from the mean latent vector from the respective cluster, and using a standard deviation of 3. These sample latent vectors can then be passed through the decoder to generate sequences. [0106] In another case, the latent processing module 110 performs interpolation between latent vectors of two natural protein sequences. In an example, 1000 points are linearly sampled between the latent hidden vectors of adenovirus 2 hexon (^1) and adenovirus 5 hexon (^2): z x - 1000 where i is an integer ranging from 0 to 1000. The interpolated vectors (v were passed through decoder.

[0107] At block 810, the decoder module 106 uses the decoder pe(x|z) to generate generated protein sequences x from the latent variable z, as described herein. The decoder can be another machine learning model parameterized by Q.

[0108] In some cases, the generated protein sequences outputted by the decoder comprises predicted probabilities for amino acids at each sequence position. At block 812, in some cases, the post-processing module 112, at each position, selects the amino acid with the highest probability. Where average positional likelihood can be calculated as the mean of the maximal probability across all positions.

[0109] At block 814, in some cases, the post-processing module 112 filters out generated sequences that are more repetitive than the most repetitive natural sequence. In the case of clusters, within each cluster, the post-processing module 112 filters out generated sequences that are more repetitive than the most repetitive natural sequence in that cluster. For the rest of the sequences, each one is aligned against the whole natural hexon dataset to get the percentage identity towards the closest natural protein. In an example, a binwidth of 2% can be used to separate sequences of different novelty. In this example, for the 17 bins with percentage identity from 60%-92%, the sequence with the highest average position probability is selected for structure prediction (such as using AlphaFold2 (AF2)). In this example, structures with an average pLDDT score of higher than 85% can be selected for further molecular dynamic analysis. [0110] In some cases, to maintain high sequence quality, sequences can be ranked according to average positional likelihood, and the, for example, top 2/3 sequences can used as candidates for downstream application.

[0111] Average positional likelihood (APL) can be calculated as: where P j is the predicted probability of amino acid .7 (including a special token representing a gap in sequence) at the zth position. L is the maximum length of training sequences. Pij is the predicted probability of amino acid .7 (including a special token representing a gap in sequence) at the zth position. L is the maximum length of training sequences.

[0112] At block 816, the output module 114 outputs the generated protein sequences, and, in some cases, filtered generated sequences, to the interface module 168, the network module 176, and/or the database 188.

[0113] Sequence clustering, as described herein, can be performed using different identity thresholds; for example, using MMSeqs2. For sequence homology detection, for example, default MMseq2 settings can be used to perform pairwise alignment between all possible pairs of generated and natural sequences in the training data. To assess structural similarity, SPOT-1 D, for example, can be used to predict three states (e.g., helix, strand, coil) of secondary structure for the amino acids. For outlier detection in secondary structure ratio, given the sum of all three ratios is 1 , a bivariate Gaussian distribution can be fitted only on the helix and strand percentage of natural protein sequences. Mahalanobis distance (d) can be determined between all generated samples and the center of bi-variate Gaussian distribution. Since d² follows a Chi- square distribution, a critical value of a = 0.05, for example, can be used to determine the cutoff distance. Samples with the smallest Mahalanobis distance and the smallest maximum sequence identity can be identified to form a Pareto front. For selected sequences, their 3D structure can be determined using any suitable approach; for example, using Alphafold2. [0114] In some cases, in order to compare sequence repetition patterns across different positions throughout a given sequence, a sliding window with a fixed length / can be defined. For a sequence of length L, there can be a total of L - 1 + 1 possible positions for the start of window, where each position can be normalized, norm_pos = pos/L Windows can be separated into bins (e.g., 20 bins) according to the normalized start of their start position. In each window, the number of repeated amino acids can be determined. Mean repetition number and mean start of window position can be determined for all the bins.

[0115] FIG. 9 illustrates example architectures for the encoder CNN and the decoder CNN. In this example, encoder CNNs used 4 dilated 3x3 convolution layers along the sequence length dimension to reduce dimensionality of the pretrained language model amino-acid level embeddings, (maximal sequence length x pretrained language model latent size). ELU activation function is used between all layers. For each convolution layer, the output length is: where Led is the output for the current layer, L_e -1 is the output of the previous layer (or maximal sequence length L for the first convolutional layer), p is padding, d is dilation, k is kernel size, s is stride size. The latent dimension for each layer’s output is gradually decreasing for the same purpose of dimension reduction d_pre > deco > deci > d_eC2 > d_eC3.

Strided convolution is used to achieve fast length reduction, and dilation is used to achieve a larger receptive field for capturing longer local sequence patterns. Flatten is reshaping the matrix to a vector. It is unlikely that the flattened length, L_eC3 X d_eC3, will exactly equal to d_pre, and therefore a linear layer is used to restore the length to d_pre. The flattened matrix is then transformed to the same length as the latent size of the pretrained language model embeddings to be used as the query in bottleneck attention.

[0116] In the example of FIG. 9, decoder CNNs used 8 UpBlocks to upsample the VAE latent vector (z’) length (equals 1 ) to maximal sequence length. In each UpBlock, a 1x1 convolutional layer is used to transform input I (dimension Lin x din) to a lower dimension (d_g), which reduces the number of parameters needed in the following layer with large kernels. The dilated 3x3 deconvolutional layer with stride of 2 is used to upsample the low-dimensional input (Lin x d_g) to upsampled output U (L_out x d_g). To prevent gradient vanishing, the input is also passed through a linear layer to get an identity matrix (T) of the same length as the deconvolutional output (U). The upsampled matrix U and the identity matrix is then concatenated as the input for the following UpBlock. The output (Z) of the final UpBlock is transformed to a matrix M with decoder hidden dimension (dde) with another 1x1 convolutional layer.

[0117] While the present disclosure generally describes a CNN in the encoder and decoder, it is appreciated that any suitable artificial neural network models can be used; for example, recurrent neural networks or graph neural networks.

[0118] In example experiments that exemplify the substantial advantages of the system 100, 711 unique full-length sequences were collected from the UniprotKB database for training. In such experiments, hexon sequences in the training data were 938 amino acids long, on average, which suggests there are likely to be more inter-residue dependency occurring at longer distances. No other approaches have been able to generate sequences of comparable length. The small dataset combined with long sequence length generally requires a small but expressive model that can be efficiently trained. In the example experiments, a protein language model was used for amino acid-level embedding. This approach allowed for the transfer of knowledge learned by the model on a large protein database; and ultimately achieve better results on limited hexon input data.

[0119] To convert the discrete protein sequence space to a continuous space for ease of sampling and other manipulation, the example experiments used a Variational Autoencoder (VAE) framework to obtain an informative and structured latent space. The example experiments utilized a special bottleneck attention unit in the encoder to map the high-quality amino-acid-level embedding, generated by the pretrained model, to the latent space. For sequence reconstruction from the latent variable, a non-sequential deconvolution-based decoder was used. The model used in the example experiments was able to generate new hexon sequences with only 12.4 million parameters; which is significantly less than the 60 million trainable parameters required in comparable other approaches modelling significantly shorter sequences.

[0120] In the example experiments, to collect training data, all hexon proteins in the UniprotKB database that were annotated to be full-length were extracted. Since such proteins were used to train a model for generating full-length hexons with all domains, sequences that were likely to have incomplete domains were removed with a length cutoff of 800 amino acids. To accommodate ease of downstream application, only sequences with standard amino acids were kept in the dataset. The resultant dataset included 711 hexon sequences with length ranging from 893 to 992 amino acids. To illustrate the generative capacity, another large transformer based language model called ProtGPT2 was fine-tuned on the same hexon dataset. A further common approach was to train models for designing sequences with a fixed-backbone as input. A fixed-backbone design generally requires prohibitively large computational resources. Instead, ESM-lnverseFolding (ESM-IF1 ), another model, was included for comparison as a representative model for this approach. ESM-IF1 was not fine-tuned due to the lack of structural data for most sequences in the training dataset.

[0121] In the example experiments, to evaluate sequence generation, 7,000 samples were generated by sampling a Gaussian distribution with mean of 0 and standard deviation of 4 from the latent space, and decoded by the decoder. The top 1 ,000 sequences with highest likelihood were used for analysis. For comparison, 1 ,000 sequences were also generated using ESM-IF1 and fine-tuned ProtGPT2, respectively. For the ESM-IF1 model, 10,000 sequences were generated using 100 Alphafold2 predicted hexon structures as templates. The top 1000 sequences with the highest log likelihood were selected for comparison. Even after fine-tuning, ProtGPT2 was not able to consistently generate protein sequences within the length range of the training data (only approximately 3% of all sampled sequences were within the length range; in total, 42,550 sequences were generated with fine-tuned ProtGPT2 and inference time was about ~13h). To meaningfully compare the generated sequences, ProtGPT2-generated sequences within the length range were ranked according to perplexity, and the top 1 ,000 sequences with lowest perplexity were selected. [0122] FIG. 10 illustrates approximated reconstruction F1 scores that are compared between hexon dataset and small PAH dataset extracted as described in ProT-VAE. Small Gaussian noises with different variance were introduced to the encoder output to simulate inaccuracy injected by the CNNs and VAE component in ProT-VAE. As can be seen, the simulated reconstruction F1 worsened even at very low variance for the hexon dataset, but the reconstruction performance was maintained even at higher variance in the PAH dataset. Thus, illustrating the advantages of the present embodiments over the approach of ProT-VAE.

[0123] To assess the system’s 100 capacity of learning the distribution of natural hexon sequences, several metrics were computed by comparing both local and wholesequence level patterns. As an initial evaluation, local amino acid positional patterns were examined. Amino-acid pair association score (based on minimal proximity function) was calculated for all possible combinations of amino acids in natural sequences and sequences generated by 3 models, as illustrated in FIG. 4A. Similarity of the pair association score between the generated sequences by the system 100 and natural sequences indicated that the system 100 has learned the local amino acid patterns of natural hexons. In comparison, almost all amino acid pairs occur at a closer distance than randomly shuffled sequences in fine-tuned ProtGPT2 generated sequences. Fewer associated pairs are detected in ESM-IF1 generated sequences, as indicated by the overall low absolute value.

[0124] FIG. 4A illustrates amino-acid pair association scores for all PortGPT2 generated, system 100 generated, and natural sequences. Negative values indicate shorter distances compared to random shuffled sequences. FIG. 4B illustrates shannonentropy for natural hexons and sequences generated by all models in MSA columns with above 20% occupancy in each datasets. Higher value reflects higher sequence variability across samples. FIG. 4C illustrates average repetition in sliding windows of length 5. Higher value means more repeated amino acids are present in a window. X- axis shows the normalized position of the left end of the window. FIG. 4D illustrates helix and strand ratio in natural and generated hexons. Shaded portions in each plot shows the area in the bi-variate normal distribution fitted on natural samples (a=0.05). FIG. 4E illustrates SASA for all amino acids in system 100 generated and natural proteins. Star indicates the only 2 amino acids with significantly different (a < 0.05) SASA values between two groups.

[0125] Generally, sequence length was one of the characteristics of hexons that added difficulties to its modeling. Generating long (~1000s tokens) and coherent texts in a specific small domain is challenging even for fine-tuned large language models like GPT2, and generated texts typically suffer from degenerate repetition. To evaluate if the generated sequences can avoid the degenerate repetition artifacts while capturing certain local repetitive patterns observed in natural sequences, the number of repeated amino acids was calculated in a fixed-length window sliding across all possible positions in a sequence (as illustrated in FIG. 4C). Regardless of the window size used, the system 100 outputs closely follow the repetitiveness trend in the natural sequences (Pearson’s r= 0.94), while ESM-IF1 generated sequences showed a slightly different repetition pattern (Pearson’s r= 0.52). The repetition increases as the generation progresses in ProtGPT2 samples, introducing more repetitive artifacts towards the end of the sequence (Pearson’s r = 0.12).

[0126] After validating the system’s 100 capacity in generating sequences resembling natural hexon proteins with sequence similarity analysis, secondary structure composition of the generated sequences was analyzed. First, the Q3 secondary structure was predicted for all natural and generated sequences with SPOT-1 D. Since the ratio of coil is dependent on the ratio of strand and helix (sum of all three ratios is 1 ), only the strand and helix percentage were analyzed. The strand and helix ratio are correlated (Pearson correlation r = -0.86) in natural hexons. Similar trend existed in the ProteinVAE samples (Pearson correlation r = -0.86), but not in the ProtGPT2 samples (Pearson correlation r= -0.60) nor ESM-IF1 samples (Pearson correlation r = -0.73) (as illustrated in FIG. 4D). To further analyze secondary structure profile, a bivariate normal distribution was fitted on the natural set, and out-of-distribution samples were identified in the generated sequences (a=0.05). More samples generated by the system 100 (877 sequences) share similar secondary structure composition with natural hexons, compared to ESM-IF1 samples (164 sequences) and ProtGPT2 samples (439 sequences) (as illustrated in FIG. 4D). [0127] To further analyze the structure profile, solvent accessible surface area (SASA) profiles were computed for all 20 amino acids in 100 randomly selected sequences from samples outputted by the ssytem 100 and natural hexons, respectively (as illustrated in FIG. 4E). SASA was calculated from Alphafold2 predicted structures. SASA differences are insignificant for 18 amino acids. Even the 2 amino acids (E, N) with significantly changed SASA, the surface exposureness is retained. This comparison supports that generated sequences by the system 100 are structurally similar to natural hexons.

[0128] To visualize sequence diversity, an equal number of natural sequences and generated sequences by all models were clustered at different thresholds. Although, sequences generated by the system 100 closely resemble sequence patterns found in natural hexon populations, they consistently had more clusters and higher diversity than natural sequences at all thresholds (as illustrated in FIG. 5A). In ProtGPT2 generated samples, there exists a large number of clusters even at extremely low identity threshold. This is likely the result of ProtGPT2 inserting sequence fragments from the vast Uniref50 database that it was pre-trained on. This artifact is also present in the randomly selected ProtGPT2 samples used in molecular dynamic simulation (FIG. 10B).

[0129] Furthermore, since the goal is to generate functional hexons that are biologically relevant, the ability of the model to diversify the sequences while keeping high structure resemblance towards natural hexon protein is of interest. As shown in FIG. 5D, more samples generated by the system 100 had similar secondary structure composition to natural protein. Additionally, to assess sequence diversity meaningfully, sequences with less than 80% target and query coverage when aligned to its closest natural sequence were removed, as this increases the chances of the generated sequences having most domains required for functionality and decreases the possibility of involving non-hexon domains that could introduce unwanted effects in future gene therapy. All the samples outputted by the system 100 satisfy this sequence-level constraint. In contrast, only 163 samples are left in the ProtGPT2 group, which further supports that ProtGPT2 incorporated non-hexon fragments in generated sequences. While the absolute sequence identity of the outputted samples only covers the higher end of the range seen in ProtGPT2 samples, all 877 analyzed samples have high structural similarity towards natural sequences (as illustrated in FIGS. 5B and 5C). This is likely because only the top 1000 sequences with the highest average positional likelihood were selected from the 1500 sequences generated by the system 100, which ensures high sequence quality at the cost of lower diversity. There were only 30 sequences in ESM- IF1 generated samples that satisfied the screening conditions, and all remaining samples had lower secondary structure similarity towards natural hexons. To directly evaluate sequence diversity against structural similarity, the Pareto frontier of all generated samples were plotted in FIG. 5C. In the comparable range, the system 100 produced samples more diverse without disruption of structural profile.

[0130] FIG. 5A illustrates a number of clusters at different identity thresholds. FIG. 5B illustrates sequence diversity plotted against secondary structure similarity. X-axis is the maximum percentage identity of 3-state secondary structure on all aligned pairs of generated and natural sequences. Y-axis is the maximum sequence identity on all aligned pairs. Sequences closer to the bottom-right corner are ideal, as they are structurally similar to natural protein but more novel in sequence. FIG. 5C illustrates a density plot for sequence diversity and secondary structure similarity. FIG. 5D illustrates Pareto frontiers, where the optimal sequences designed by two models are highlighted along respective frontier.

[0131] To evaluate the stability of the predicted structures, the example experiments subjected the structures to a classical molecular dynamics simulation for 100 nanoseconds. The root-mean-square deviation (RMSD) and root-mean-square fluctuation (RMSF) of the structures were analyzed and used to screen the sequences (as illustrated in FIGS. 6A and 6B). FIG. 6A illustrates RMSD for all wild-type cluster representative sequences and stable generated sequences. FIG. 6B illustrates RMSF for all wild-type cluster representative sequences and stable generated sequences.

[0132] Advantageously, the system 100 provides ease of sampling due to the structured VAE latent space. In the example experiments, to validate that evolutionary relationships and sequence similarities have been captured in the latent vectors, the 10 largest clusters (at 90% sequence identity threshold) were plotted in dimension-reduced hidden space (as illustrated in FIG. 7A). Multiple clusters can be found in the hidden space distinctly separated. The hidden space of the present embodiments also appears around the mean of 0 with no obvious hole.

[0133] The example experiments also performed interpolation between hexons from two adenovirus serotypes, Ad2 and Ad5, which are experimentally-tested to be interchangeable. Ad5 is the most common adenovirus vector, which means creating mutants to avoid its pre-existing neutralizing antibodies would be more impactful than creating mutants for a random natural hexon. Hexon exchange between certain serotypes could result in unsuccessful virus assembly. By interpolating between exchangeable hexons, there are increased chances of forming viable viruses with resulting hexon variants and the rest of the proteins from the Ad5 genome. 1 ,000 vectors were linear interpolated between Ad5 and Ad2 hexon hidden vectors in latent space. The interpolated vectors were decoded to sequences. As a control, a different data interpolation approach was implemented to sample between Ad5 and Ad2 hexon sequences directly. Both the control and latent interpolation achieved monotonic changes in hamming distance (as illustrated in FIGS. 7C and 7D). However, the latent interpolation allows for generating of natural-resembling sequences, as indicated by higher average positional likelihood (as illustrated in FIG. 7B). This example showcases the substantial advantages of using the system 100 to aid rapid sequence design based on known biological constraints.

[0134] FIG. 7A illustrates latent space clustering of sequences from 10 largest clusters at 90% identity. FIG. 7B illustrates average positional likelihood for latent space interpolated sequences and direct interpolated sequences. FIGS. 7C and 7D illustrate Hamming distance to ad2 and ad5 hexon, respectively.

[0135] To test the hypothesis that non-sequential generation is more suitable in the case of protein design, deconvolution-based upsampling was compared with an LSTM approach. After tuning the number of layers, processing direction, and hidden dimension, the best performing LSTM-based model (14.0M parameters) only achieved a reconstruction accuracy of 0.18 on the validation set, while the deconvolution-based (12.4M parameters) model achieved 0.86. As an alternative to using LSTM as an upsampling method, training LSTM with additional per-time-step token information in a 1 teacher forcing manner was experimented, and this led to detrimental overfitting. Thus, generation with the LSTM-based model is not generally suitable given the extremely low reconstruction accuracy.

[0136] To examine each component’s contribution to the generation capacity, an ablation study was conducted by leaving out non-standard elements of the network one at a time. Reconstruction and generation performance was summarized for each version of the ablated model in TABLE I. The full model performs best overall. The encoder CNN module improved the reconstruction accuracy, which is likely due to the addition of local features that reflects detailed differences among sequences. The bottleneck attention unit has more effect on improving generation, which is likely tied to1 its ability to produce a better protein level representation that complies with the sampling process. The amino acid attention improved generation in terms of secondary structure ratio and sequence variability profile. In addition, the secondary structure reweighting on the cross-entropy loss improved amino acid usage and sequence variability profile.

Encoder Bottleneck Amino-acid-wise Secondary-structure-weighted JSD (Amino JSD (Secondary MSA Entropy

CNN attn. attn. CE Loss Test F1 acid usage) Structure) Correlation

ProtGPT2 (fine-tuned) I I I / / 0.040 0.073 0.521

Final + + + + 0.861 0.022 0.015 0.911 w/o encoder CNN + + + 0.836 0.023 0.015 0.878 w/o bottleneck attn. + - + + 0.868 0.039 0.038 0.500 w/o aa attn. + + + 0.847 0.013 0.024 0.669 w/o ss weighted CE + + + - 0.858 0.023 0.012 0.902

Base . + . . 0.790 0.015 0.044 0.436 [0137] TABLE I shows the results of the ablation study, where each of the settings are repeated with 3 different random seeds. Whereby: • Encoder CNN: CNN feature vector extractor in encoder; 1 • Bottleneck attn.: Autobot attention mechanism; • Amino-acid-wise attn.: Attention mechanism across different amino-acid channel; • Secondary-structure-weighted CE Loss: cross entropy reweighted to assigned 1 ,2x penalty to the strand positions;

• Test F1 : reconstruction F1 score on test set;

• JSD (Amino acid usage): Jesen-Shannon distance between amino acid usage frequency;

• JSD (Secondary Structure): Jesen-Shannon distance between secondary structure ratio;

• MSA Entropy Correlation: Pearson correlation between MSA entropies in valid columns of generated and natural sequences.

[0138] The example experiments illustrate that the system 100 can be used to learn the intrinsic relationships of a long protein sequence from a limited number of samples. Computational evaluation between the system 100 and a fine-tuned large language model, ProtGPT2, revealed that the system 100 can more reliably capture population level statistical characteristics of natural hexons. Specifically, the system 100 generated sequences resembled natural sequences on both sequence level and structural level. In addition, sequences generated by the system 100 are more diverse than natural sequences, capable of forming more clusters at the same identity threshold. The system 100 has a structured latent space which can be used for sequence interpolation between biologically validated interchangeable protein sequences. Further, using only 711 training samples, the system 100 was able to generate diverse sequences that are tested to be stable in molecular simulations, with the most novel sequence containing 291 amino acids that are different from its closest natural sequence, and 39.62% hexon viral surface area was changed in such design.

[0139] Considerable efforts have been made towards computationally expanding known protein families with novel sequences. In conventional bioinformatics, hidden Markov models (HMMs) can be used to capture pairwise coupling within the MSA of a protein family, and as an extension, it can be used to generate new sequences. However, it has been shown that HMMs have inconsistent levels of success, mostly due to its inability to learn higher-order relationships that exist in natural protein families. As an alternative, modern deep learning models, including Generative Adversarial Networks (GAN), VAEs, and large generative protein language models have been implemented to learn the complex constraints in biological sequence design. However, previous approaches have mostly focused on shorter protein sequences with a large number of members from the same family.

[0140] The difficulties with the task of generating diverse protein sequences, such as for hexon sequences, were further demonstrated with the unsatisfactory performance of a fixed-backbone design model (ESM-IF1 ), and a large language model (ProtGPT2) finetuned on a hexon dataset. Although some fixed-backbone-design approaches have some promising results on smaller proteins, they require significantly higher GPU memory for generating long sequences, as the inter-residue distance information requires quadruple amount of memory for processing as the length increases. For instance, 24GB GPU memory is needed to generate 1 hexon sequence using ESM-IF1 model. Only 21GB GPU memory is needed to generate 1000 hexon sequences in parallel for system 100. For sequences of even longer length, generation might not be feasible, as the memory requirement for generating a single sequence might exceed typical GPU memory. It is also possible to improve the ProtGPT2 generation, but it would take significantly more computational resources to search for the optimal generation settings. Instead, the system 100 is able to distill knowledge from a protein language model and leverage it to facilitate efficient learning of complex sequence patterns from limited data. Moreover, the model used by the system 100 was able to generate 1000 sequences in less than 1 second, while the generation of 1000 lower- quality sequences took approximately 13 hours for the ProtGPT2 model.

[0141] Other approaches, such as ProT-VAE and ReLSO, use autoencoders and language models, but generally do not have the ability to output results for generating proteins at the length range of the present embodiments; such as at the length range of hexons. Further, such approaches are generally trained for different objectives, such as for exploring fitness landscape and generating functionally improved sequences. Furthermore, such approaches generally require substantially larger labeled datasets (for example, ProT-VAE requires 6447 and 20,000 sequences, and ReLSO requires 10¹⁰, 20⁴, and 51 ,175 sequences). [0142] Generally, as determined by the present inventors, other autoregressive protein language models are unsuitable for generation of long sequences. Typically, in natural language processing, such models have substantially difficulty generating long (~1000s tokens) and coherent texts in a specific small domain, even for fine-tuned large language models like Generative Pre-trained Transformer 2 (GPT-2), and generated texts typically suffer from degenerate repetition.

[0143] In the example of the ProT-VAE approach, a generic CNN network is used for compressing and decompressing pre-trained language model (PLM) hidden states, and a protein-family-specific VAE portion is trained to further reduce the protein-level representation to a single vector and reconstruct the hidden states given the vector. The VAE component is trained with maximum mean discrepancy loss instead of KL- divergence. ProT-VAE is built on a ProtT5 model that was pre-trained on sequences with a maximum length of 512 amino acids, which substantially hinders the performance on generating longer proteins like hexons. Due to the model design, exchanging the PLM would not only require retraining the VAE component on family-specific sequences, but also retraining the generic CNN network on a large protein database. In contrast, accommodating another PLM by the system 100 only requires training the VAE component on sequences from a specific protein family.

[0144] The ProT-VAE approach differs from the present embodiments in a number of substantial ways. For example, the transformer-based featurizer used in ProT-VAE comprises a T5 transformer encoder and the projection to sequence comprised a T5 transformer decoder; each trained with Uniprot sequences of less than 512 aa (46M sequences), and with a masked language modeling objective. Similarly, the convolutional neural networks used in the ProT-VAE approach were trained with Uniprot sequences of less than 512 aa (46M sequences), with a mean squared error reconstruction objective, and with fixed T5 weights. The dimension reduction and dimension decompression included linear layers that required separate training.

[0145] With respect to the ProT-VAE approach, the computational cost of training is substantially greater than the present embodiments. The system 100 in contrast requires no training on large databases; while the CNNs in ProT-VAE are trained on large databases requiring large memory and computational resources to accomplish in a reasonable timeframe. Furthermore, the ProT-VAE CNNs need to be retrained with a high relative cost for each PPLM, and only then can the VAE portion be trained.

[0146] In the example of the ReLSO approach, the model was designed with an autoencoder, instead of a VAE architecture, and fitness information was jointly trained to be encoded in the latent space. The ReLSO was trained on both positive and negative samples with a specially designed interpolation loss. ReLSO is substantially less effective in generating protein sequences, like hexons, due to limited data capabilities; i.e. , capabilities to handle a small number of natural sequences in the protein family of interest. For example, where there are not many full-length natural proteins in the family that can be used to train larger (more parameters) models.

[0147] The capacity of the system 100 to learn the complex protein sequence distribution from limited samples can be applied in a variety of different sequence design situations. Traditionally, exploration of the protein fitness landscape has been done with mutagenesis libraries. This approach is inherently inefficient as only neighboring regions can be explored. In contrast, another machine learning model can be trained to map the latent space of the present embodiments to the protein fitness landscape, and apply the model to conditionally generate sequences with functional improvement. In this way, exploration of distant regions of the fitness landscape, where significant functional enhancement might be, can be determined.

[0148] In the example experiments, structures were predicted with the Alphafold2 (AF2) reduced database. After folding a small number of natural hexons, it was observed that the model predictions ranked the first for all sequences.

[0149] In the example experiments, Clustal Omega was used to calculate multiple sequence alignment (MSA) for the entire natural hexon dataset mixed with an equal number of generated sequences. Columns with more than 80% gaps in either natural or generated dataset were removed. Shannon entropy within each column was calculated as: where p(x_z) is the frequency of amino acid / in each column. Pearson correlation was calculated between valid entropy values of natural and generated sequences.

[0150] For any pair of amino acid a and b, the minimal proximity score and the pair association metrics were determined. Distance between each occurrence of a at position Xi with its nearest occurrence of b at position y, was computed, and then averaged across all occurrences of a:

[0151] To remove dependency on the number of occurrences for different amino acids, the position of a was fixed and b was randomly shuffled. Mean _anc| standard deviation (^<7T^>m(a,Rand(6))) _of the randomly shuffled sequences (where position of b is shuffled, but position of a is fixed) were calculated, and the minimal proximity score was normalized to obtain the association score:

P_m (a, b) — P_m (a, Rand (6))

[0152] For the score shown in FIGS. 4A to 4E, averaged association scores were plotted for each group of sequences. A null value is assigned if a pair of amino acids does not exist in a sequence.

[0153] In the example experiments, to visualize the latent space, Principal Component Analysis (PCA) was used to reduce the number of dimensions to 2. The 10 biggest clusters (457 sequences in total) at 90% sequence identity in natural hexon sequences were used for plotting. For interpolation, 1000 points were linearly sampled between the hidden vector of adenovirus 2 hexon and adenovirus 5 hexon. Hidden codes were passed through decoder to get the predicted probabilities. At each position, the token with the highest logit is chosen, and average positional likelihood is calculated as the mean of the max logit across all positions. As a control, direct interpolation between two sequences was performed by sampling a Bernoulli random variable with a probability to choose amino acid from adenovirus 2 hexon (1- a probability from adenovirus 5 hexon) at each position. 1000 different a were linearly selected from 0 to 1 . 10 sequences were sampled at each a, resulting in 10000 sequences in total. The system 100 was run on the directly interpolated samples, and collected the predicted probabilities. The average positional likelihood was calculated as the mean of the logit for the input sequence.

[0154] In the example experiments, for comparison, the ProtGPT2 model was fine-tuned on the training data. Learning rate from 10’⁶ to 10’⁴ was tested. The final model used for generation was trained with a learning rate of 10’⁵. The fine-tuned model was prompted with “M” at the start of sentence. It was observed that generation performance drastically worsened with inclusion of any token with “X”, and all such tokens were removed. The maximum token allowed to place in a sequence was set to 1000 to accommodate the case of the model using single amino acid tokens only. Inference of 50 sequences was repeated for 851 batches (42550 sequences in total, approximately 13 hours inference time), until 1500 sequences within the length range of hexon were accumulated. Sequences were ranked according to their perplexity, and only kept the top 1000 for comparison. To accommodate downstream analysis, “Z” and “B” found in ProtGPT2-generated sequences were replaced with appropriate standard amino acids.

[0155] In the example experiments, for the comparison with ESM-IF1 Generation, 100 natural hexons were randomly selected from the training data, and their structures were computationally predicted with Alphafold2. For each computationally obtained structure template, 100 sequences were generated with ESM-IF. The 10000 generated sequences were ranked according to their log likelihood, and only the top 1000 sequences were used for comparison.

[0156] In the example experiments, for molecular dynamics analysis, input structures were used to build a protein-membrane system using C HARM M -GUI. The proteinmembrane systems were solvated in an explicit TIP3P water box. Charge neutrality was maintained by addition of counter ions, and physiological condition was mimicked using 0.15 M KCI. The protein-membrane systems were energy minimized using steepest descent before pre-equilibration phase, which was conducted for 500 ps under the constant number of particle, volume, and temperature (NVT) condition. For each protein-membrane system, the production phase was carried out for 100 ns. The particle-mesh Ewald (PME) approach was used with a cut-off radius of 1 .2 nm for long- range electrostatic interactions. Heavy atom-hydrogen atom bonds were constrained using the parallel linear constraint solver (P-LINCS) algorithm. The Nose-Hoover thermostat with a coupling time constant of 1 ps and the Parrinello-Rahman barostat with a coupling time constant of 5 ps were used for the production phase. A reference coupling pressure of 1 bar and a compressibility of 4.5 x w⁵ bar¹ were used. For all molecular dynamics simulations, periodic boundary conditions were applied in all directions. Simulations were carried out using CHARMM36m force field by GROMACS/2021 .3.

[0157] As evidenced in the example experiments, the present embodiments provide a number of substantial advantages over other approaches in its ability to generate viable diverse sequences. Additionally, using the pretrained-featurizer increases data- efficiency. The present embodiments are able to generate relatively long sequences in comparison to other approaches. Further, the non-autoregressive decoder means that there is no accumulation of error, and thus, less repetition towards the end of generated sequences. Additionally, using the attention layers in the decoder allows the system 100 to take into consideration long distance relationships. Additionally, the design of the model means that using a different pre-trained protein language model means that no training on large databases is required, and thus, the present embodiments can easily accommodate other pre-trained protein language models.

[0158] In some cases, the present embodiments can be applied to generating sequences in the Adeno-associated virus (AAV) family. AAV is a potential gene therapy vector because of its better safety profile and broader tropism. However, problems exist in AAV usage which can be advantageously overcome with the present embodiments. Specifically, when re-administering, the cross-reactive neutralizing capsid antibodies prevent the re-administration of AAV. Cross-reactivity can even prevent administering different natural serotypes to the same patient, which creates the need for computationally generating artificial capsid protein that have higher diversity than observed in natural serotypes to identify candidates that have low change of being neutralized by cross-reactive antibodies in previously treated patient.

[0159] AAV capsid is mainly made of three proteins: VP1 , VP2, and VP3. VP2 and VP3 have overlapping coding regions inside the VP1 protein, and, in some cases, the VAE of the present embodiments can be trained on VP1 sequences because these sequences can be easily trimmed to create corresponding VP2 and VP3. AAV belongs to the dependoparvovirus genesus. Dependoparvovirus VP1 protein sequences can be, for example, extracted from UniprotKB database. In an example, obtained sequences can be filtered for length longer than 700 and any sequences with non-standard amino acids can be removed. In this example, 398 sequences can be kept as training data.

[0160] In further example experiments, the VAE model was successfully trained on the VP1 dataset with some hyperparameter tuning. Due to even smaller number of sequences, the data was only split into training and validation data in 7:3 ratio. Reconstruction F1 -score reached an even higher level of 0.91 on the validation data than in the experiments with the hexon dataset. 1000 latent vectors were sampled from a gaussian distribution with mean of 0 and standard deviation of 3. Latent vectors were passed through the decoder to obtain sequences. Sequences were ranked according to average protein likelihood. The top-ranking sequences were of higher quality but lower diversity. The bottom -ranking sequences were of lower quality. The three middleranking proteins were randomly selected to get a larger range of diversity while maintaining sequence quality. Their structure was predicted with Alphafold2.

[0161] In the further example experiments, initial analysis was done by comparing the experimentally obtained natural protein structure (pdb: 7MTW) with the generated structure. Only the VP3 (truncated version of VP1 ) structure was included in the experimentally obtained structure, 7MTW. As can be seen, generated sequences have very similar structures in the VP3 regions when compared to the natural protein. The generated sequences only share low sequence identity towards any known protein in the NCBI nr-protein database. Structure of the first ~137 amino acid in VP1 has never been successfully experimentally obtained. This means that this region does not exist in the training data for Alphafold2, and Alphafold2’s accuracy of the predicted structure in this region is lower than normal. Therefore, more underdefined structures are seen in this region in the prediction of the generated protein.

[0162] When used for generation, latent vectors different from mapped natural sequences can be decoded to generate novel sequences. In this way, advantageously, the system 100 can be used to generate protein sequences comprising novel serotypes. In an example, the system 100 can be used to generate novel synthetic human adenovirus serotypes. In an example, to select potential human adenovirus hexons from generated sequences outputted by the system 100, a logistic regression classifier can be trained from the encoder embeddings of the training data with labels identifying whether or not a sequence is from a human adnovirus. Where area under a receiver operating characteristic curve (ROC) of the trained classifier can be, for example, 0.97. Sequences generated from each cluster can be encoded and classified. Percentage of generated sequences classified as human adenovirus hexon correlated with that of natural sequences in each cluster (for example, a pearson’s r = 0.81). There exist two loop regions in hexons where its amino acid divergence contains serotype determining information. Sequences that are classified as human adenovirus hexon can be aligned with natural human adenovirus hexons with known serotypes. Amino acid divergence can be calculated for each pair of sequences in the 2 loop regions. Example experiments showed that generated sequences diverged more than 4.2% and more than 1 .2% from any known serotypes in loopl and Ioop2, respectively; which suggested the generated hexons are likely to produce novel synthetic human adenovirus serotypes. In these example experiments, FIGS. 11A and 11 B illustrate results of pairwise amino acid divergence for two loop regions in hexons; where darker shading means more difference between the compared sequences.

[0163] FIGS. 12 to 16 illustrate example computational analyses and generated AAV. In these examples, recombinant plasmid for producing wild-type adenovirus was produced using a pAdEasy system.

[0164] Integrating synthetic hexons was performed using Sfil digest sites flanking the hexon region. The vector backbone was digested, separated using agarose gel electrophoresis, extracted using the NEB Monarch™ DNA Gel Extraction Kit, dephosphorylated using NEB Quick CIP enzyme to prevent self-ligation, and stored. The regions immediately flanking the hexon were PCR amplified using Invitrogen Platinum™ SuperFi II Green PCR Master Mix and ligated to hexon sequences using Gibson assembly master mix. The entire ligated fragment (hexon with flanking regions) was then re-integrated into the previously prepared adenovirus backbone using Sfil digest sites. Successful integration was tested using a unique AsiSI digest site only present in the wild-type hexon sequence.

[0165] Plasmids were transfected into HEK293T cells seeded in a T75 for approximately 70-80% confluency the next day. The transfection was done using PEI MAX. 10ug of adenovirus plasmid was digested using Pad enzyme and ethanol precipitated. The DNA was mixed with 300uL Gibco™ OptiPRO™ serum-free media. 30ug of PEI was mixed with 300uL of OptiPRO in a separate tube. The PEI dilution was then added to the DNA in drops while gently flicking the tube. The transfection mix was incubated at room temperature for 15 minutes and then added to the T75 flask dropwise. After 1 day, the media was changed to complete DMEM and left until approximately 50% cytopathic effect was observed.

[0166] Cells were then scraped from the T75 and collected, then spun down for 5 minutes at 3000g along with media. The supernatant was transferred to a new tube except for approximately 1 mL. The cells were resuspended and lysed using 3 freezethaw cycles between liquid nitrogen and a 37C water bath. The lysate was transferred to a 1 ,5mL tube and spun down at 20,000g for 5 minutes. The supernatant was combined with the cell media and syringe-filtered through a 0.22um filter.

[0167] Crude viral preparation was combined with PEG solution (20% PEG8000 in water with 2.5 M NaCI) at a 2:1 ratio (lysate:PEG) and left overnight at 4C. The mixture was centrifuged for 1 hour at 4000g at 4C. The supernatant was discarded, and the pellet was resuspended in PBS with 0.01 % Pluronic™ F-68.

[0168] FIG. 12 shows a repetition profile for natural AAV VP3 protein and sequences generated by ProteinVAE and fine-tuned ProGen2 with window size of 5. FIGS. 13A and 13B shows Shannon-entropy for natural AAV VP3 protein and sequences generated by ProteinVAE (FIG. 13A) and ProGen2 (FIG. 13B) in MSA columns with above 20% occupancy in each dataset. Higher values reflect higher sequence variability across samples. ProteinVAE sequences follows similar variability patterns as the natural protein. Due to insertion of random sequences segments seen in ProGen2 generated sequences, fewer valid columns were preserved. [0169] FIGS. 14A to 14C show helix and strand ratio in natural and AAV VP3 proteins (FIG. 14A) generated by ProteinVAE (FIG. 14B) and ProGen2 (FIG. 14C). The plots show the area in the bi-variate normal distribution fitted on natural samples (a = 0.05). In generated sequence plots, the points in the plots represent outliers.

[0170] FIG. 15 shows a number of clusters at different identity thresholds. ProteinVAE generated sequences exhibited higher number of clusters than natural AAV VP3 proteins consistently, which suggested a higher sequence diversity. ProGen2 generated sequences had high number of clusters at extremely low sequence identity threshold, as random sequence sections were inserted.

[0171] FIG. 16 shows a scatter plot for sequence diversity and secondary structure similarity. Y-axis is the maximum sequence identity on all aligned pairs. X-axis is the maximum percentage identity of 3-state secondary structure on all aligned pairs of generated and natural sequences. Sequences closer to the lower-right corner are ideal, as they are structurally similar to natural protein but more novel in sequence.

[0172] FIG. 17 shows experimental validation of the AAV designs, showing that they assemble into viral particles. FIG. 17 shows ddPCR quantification of AAV2 virus control and synthetic variants. Probes against eGFP shows presence of eGFP payload sequence in all controls and variants. AAV2 positive control was purchased from Neurophotonics at a concentration of 1e+12. A first batch of AAVs was produced with AAV2 control 1 and SynAAV#1 . A second batch of viruses was produced later with AAV2 control 2, SynAAV#2, and SynAAV#3. Virus samples were first digested using DNAse to remove residual plasmid to prevent false positive signal. Then further diluted and denatured at 95C for 10 minutes in a thermocycler before being sent for ddPCR quantification. Each sample was also diluted at 1/10 and also sent.

[0173] FIG. 18 shows experimental validation for Ad designs and controls. FIG. 18 shows ddPCR quantification of pAdEasy adenovirus virus control and synthetic variants. Probes against eGFP shows presence of eGFP payload sequence in all controls and variants. Adenovirus 2 control was made using pAdEasy system. pAdEasy control was made using single-plasmid transfection. pAdHex was made using pAdEasy hexon sequence in trans on second plasmid with hexon removed from original adenovirus plasmid. WT Hex variant was made the same way with NCBI hexon sequence. The three synthetic Ad variants were also made the same way with hexon sequences provided in trans. Virus samples were first digested using DNAse to remove residual plasmid to prevent false positive signal. Then further diluted and denatured at 95C for 10 minutes in a thermocycler before being sent for ddPCR quantification. [0174] Although the invention has been described with reference to certain specific embodiments, various other aspects, advantages and modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.

Claims

1 . A system for developing machine learning models for generation of protein sequences, the system comprising one or more processors and a memory comprising instructions for the one or more processors to execute: an input module to receive input training samples comprising naturally- occurring protein sequences; an encoder module to train an encoder artificial neural network to map amino-acid-level representations into latent variables in a latent space, the amino-acid-level representations generated by a pre-trained protein language model using the received input training samples as input, an encoder comprises the encoder artificial neural network and the protein language model; a decoder module to train a decoder artificial neural network to reconstruct protein sequences using the mapped latent variables as input, a decoder comprises the decoder artificial neural network, the decoder artificial neural network and the encoder artificial neural network trained using determination of a loss between the reconstructed protein sequences and the naturally-occurring protein sequences; and an output module to output the trained encoder artificial neural network and the trained decoder artificial neural network for use in generation of protein sequences.

2. The system of claim 1 , wherein the encoder further comprises a bottleneck attention unit to refine the output feature vectors of the encoder artificial neural network.

3. The system of claim 2, wherein the encoder further comprises an encoder linear to change dimensionality of the refined output features.

4. The system of claim 1 , wherein the decoder further comprises one or more selfattention units to refine the probabilities of amino acids to capture dependencies between amino acid usage at different positions or to capture interactions between different types of amino-acids.

5. The system of claim 1 , wherein the decoder further comprises a decoder linear to convert the features outputted by the decoder artificial neural network to logits, and wherein the decoder further comprises a softmax activation function to determine the probabilities of amino acids at positions in the sequence from the logits.

6. The system of claim 1 , wherein the decoder artificial neural network comprises one or more blocks each comprising one or more one-dimensional convolutional layers and one or more one-dimensional transposed convolutional layers.

7. The system of claim 1 , wherein the encoder artificial neural network comprises one or more one-dimensional convolutional layers.

8. The system of claim 1 , wherein the encoder artificial neural network and the decoder artificial neural network are jointly trainable in a single stage.

9. The system of claim 1 , wherein the generated protein sequences comprise novel synthetic human adenovirus serotypes.

10. A computer-executable method for developing machine learning models for generation of protein sequences using a computer, the method comprising: receiving input training samples comprising naturally-occurring protein sequences; training an encoder artificial neural network to map amino-acid-level representations into latent variables in a latent space, the amino-acid-level representations generated by a pre-trained protein language model using the received input training samples as input, an encoder comprises the encoder artificial neural network and the protein language model; training a decoder artificial neural network to reconstruct protein sequences using the mapped latent variables as input, a decoder comprises the decoder artificial neural network, the decoder artificial neural network and the encoder artificial neural network trained using determination of a loss between the reconstructed protein sequences and the naturally-occurring protein sequences; and outputting the trained encoder artificial neural network and the trained decoder artificial neural network for use in generation of protein sequences.

11 . The method of claim 10, wherein the encoder further comprises a bottleneck attention unit to refine the output feature vectors of the encoder artificial neural network.

12. The method of claim 11 , wherein the encoder further comprises an encoder linear to change dimensionality of the refined output features.

13. The method of claim 10, wherein the decoder further comprises one or more selfattention units to refine the probabilities of amino acids to capture dependencies between amino acid usage at different positions or to capture interactions between different types of amino-acids.

14. The method of claim 10, wherein the decoder further comprises a decoder linear to convert the features outputted by the decoder artificial neural network to logits, and wherein the decoder further comprises a softmax activation function to determine the probabilities of amino acids at positions in the sequence from the logits.

15. The method of claim 10, wherein the decoder artificial neural network comprises one or more blocks comprising one-dimensional convolutional layers and onedimensional transposed convolutional layers.

16. The method of claim 10, wherein the encoder artificial neural network comprises one or more one-dimensional convolutional layers.

17. The method of claim 10, wherein the encoder artificial neural network and the decoder artificial neural network are jointly trainable in a single stage.

18. The method of claim 10, wherein the generated protein sequences comprise novel synthetic human adenovirus serotypes.

19. A computer-implemented method for generating protein sequences, the method comprising: receiving input samples comprising naturally-occurring family of protein sequences; selecting a subset of sequences from the family of protein sequences; mapping, using an encoder, the selected subset of input samples into latent vectors in a latent space, the encoder comprising a protein language model to generate amino-acid-level representations from the selected subset of input samples and a trained encoder artificial neural network to output latent vectors from the amino-acid-level representations; generating, using a decoder, protein sequences in the family of protein sequences using the mapped latent vectors as input to a trained decoder artificial neural network, the generated protein sequences comprising probabilities of amino acids at positions in the respective sequence; and outputting the generated protein sequences.

20. The method of claim 19, wherein selecting the subset of sequences comprises performing clustering on the received family of protein sequences and selecting a representative sequence from each cluster.

21 . The method of claim 19, wherein selecting the subset of sequences comprises sampling from a distribution of the family of protein sequences.

22. The method of claim 19, further comprising performing analysis of a mean, or interpolation analysis, on the latent vectors to determine which mapped latent vectors are provided to the decoder.

23. The method of claim 19, further comprising, at each position in the generated sequence, selecting the amino acid with highest probability.

24. The method of claim 19, further comprising filtering out generated sequences that are more repetitive than naturally-occurring sequences.

25. The method of claim 19, wherein the generated protein sequences comprise novel synthetic human adenovirus serotypes.