[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-128581-en":3,"doc-seo-128581-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},128581,549768064778,"Finn","https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8",8,"Research & Report","Protein Language Representation Learning to predict SARS-CoV-2 Mutational Landscape - Abstract and Thesis Overview","The dissertation addresses the urgent need to identify high-risk SARS-CoV-2 mutations that influence transmission, infection risk, and antibody neutralisation escape. It applies language-model techniques to spike protein sequences treated as amino-acid alphabets. Two novel models are introduced: CoVBERT for point-mutation prediction and MuFormer for iterative mutational sequence design using both sequence and structural context. Reported results show CoVBERT surpassing large protein language models on mutation transmissibility prediction, and MuFormer producing de novo sequences validated against historical variant properties.","Protein Language Representation Learning to predict SARS-CoV-2 Mutational Landscape  \nHunar Batra Department Of Computer Science  \nUniversity of Oxford  \nA Dissertation submitted for the degree of Master of Science in Advanced Computer Science under the supervision of Prof. Peter Minary  \nMichaelmas Term 2022  \nAbstract  \nWith the proliferation of SARS-CoV-2 pandemic globally, numerous variants have been emerging on a daily basis containing distinct transmission and infection rates, risks and impact over evasion of antibody neutralisation. Early discovery of high-risk mutations is critical towards undertaking data-informed therapeutic design decisions and effective pandemic management. This dissertation explores the application of Language Models, commonly used for textual processing, to decipher SARS-CoV-2 spike protein sequences which are an amalgamation of amino acids represented as alphabets. Deep protein language models are revolutionising protein biology, and with the introduction of two novel models: transformer encoder-based sequence only CoVBERT for predicting point mutations, and MuFormer which leverages the sequence and structural space to design mutational protein sequences iteratively. CoVBERT has been able to predict highly transmissible mutations including D614G with amasked marginal log likelihood of 0.95, surpassing state-of-the-art large protein language models. This reflects over large language models ability to encapture in vitro mutagenesis by learning the language of evolution.  \nMuFormer is capable of generating de novo protein sequences using AlphaFold2 for fixed backbone design, and curates evolutionary novel mutational sequences by injecting the representation derived state-of-the-art protein language models. The generated mutational sequences have been validated with historical data which exemplified the ability of MuFormer to capture phylogenetic properties for generating mutations such as Omicron and Delta variant, given Alpha variant as the input. MuFormer conditions not only over the sequence, but also the structure to generate end-to-end protein sequences and structure by optimising using two strategies of fixed backbone design (MuFormer-fixbb) and backbone atom optimisation (MuFormer-bba) . Both these variants of MuFormer outperformed AlphaFold2 over the mutational sequence generation task for several structure and sequence likelihood metrics. These models ascertain over the potential of large language models, termed as foundational models, towards learning the representational language of biology which can assist in controlling pandemics by predicting mutations with higher infectivity in advance.  \nContents  \nList of Figures vi  \nList of Tables viii  \n1 Overview 1  \n1. 1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1  \n1. 2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1  \n1.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2  \n2 Related Work 4  \n3 Background 7  \n3.1 Neural Language Models ............................. 7  \n3.1.1 Feedforward Neural Language Models ................. 8  \n3.1.2 Word Embeddings ............................ 10  \n3.1.3 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 11  \n3.1.4 Gated RNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13  \n3.1.4.1 Long Short-Term Memory . . . . . . . . . . . . . . . . . . 13  \n3.1.4.2 Gated Recurrent Units . . . . . . . . . . . . . . . . . . . . 15  \n3.1.5 Seq2Seq Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16  \n3.1.6 Attention Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . 17  \n3.1.7 Masked Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . . 18  \n3.1.8 Autoregressive Decoders ......................... 18  \n3.2 Protein Language Modelling ........................... 19  \n3.2.1 Proteins and Natural Language ..................... 19  ","cbCaitSmCmhFqz86","https://ap.wps.com/l/cbCaitSmCmhFqz86","pdf",23038875,1,111,"English","en",105,"# Overview\n## Motivation\n## Contribution\n## Thesis Overview\n# Related Work\n# Background\n## Neural Language Models\n## Protein Language Modelling\n# CoVBERT-Mutation Prediction Language Model\n## Introduction\n## Proposed Model Overview\n## Model Architecture\n## Dataset Preprocessing\n## Tokenizer Training\n## Positional Embeddings\n## Masking Objective\n## Self-supervised training\n# CoVBERT-Experimental Results\n## Zero-shot SARS-CoV-2 mutation prediction\n## Baseline Protein Language Models\n## Results\n## Interpreting Self-Attention in CoVBERT\n# MuFormer-Structure and Sequence based Mutation Generation Model","[{\"question\":\"What problem does the dissertation focus on regarding SARS-CoV-2?\",\"answer\":\"It focuses on discovering high-risk mutations that affect transmission, infection risk, and evasion of antibody neutralisation, enabling earlier, data-informed therapeutic and pandemic-management decisions.\"},{\"question\":\"How does CoVBERT predict SARS-CoV-2 point mutations?\",\"answer\":\"CoVBERT is a transformer-encoder model using sequence-only input to learn protein “language” patterns and predict point mutations in a zero-shot setting.\"},{\"question\":\"What is MuFormer and how does it generate mutational protein sequences?\",\"answer\":\"MuFormer generates mutational sequences de novo, leveraging both sequence and structural space and validating generated mutations against historical data using Alpha variant input to recover properties of Omicron and Delta.\"}]","Protein Language Representation Learning to predict SARS-CoV-2 Mutational Landscape - Abstract and Thesis Overview | PDF",1786001919,280,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"protein-language-representation-learning-to-predict-sars-cov-2-mutational-landscape-abstract-and-thesis-overview","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/protein-language-representation-learning-to-predict-sars-cov-2-mutational-landscape-abstract-and-thesis-overview/128581/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-22","2026-08-06",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does the dissertation focus on regarding SARS-CoV-2?","Question",{"text":76,"@type":77},"It focuses on discovering high-risk mutations that affect transmission, infection risk, and evasion of antibody neutralisation, enabling earlier, data-informed therapeutic and pandemic-management decisions.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does CoVBERT predict SARS-CoV-2 point mutations?",{"text":81,"@type":77},"CoVBERT is a transformer-encoder model using sequence-only input to learn protein “language” patterns and predict point mutations in a zero-shot setting.",{"name":83,"@type":74,"acceptedAnswer":84},"What is MuFormer and how does it generate mutational protein sequences?",{"text":85,"@type":77},"MuFormer generates mutational sequences de novo, leveraging both sequence and structural space and validating generated mutations against historical data using Alpha variant input to recover properties of Omicron and Delta.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,121,124,129,132,136],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":46,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":46,"category_name":138,"show_sort_weight":107,"slug":139},19,"General","general"]