ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing

Understand

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models taken from NLP.

• These LMs reach for new prediction frontiers at low inference costs.
• Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids.
• The LMs were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores.