2024

DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings

Zhou, Zhihan, Wu, Weimin, Ho, Harrison et al.

Understand

We introduce DNABERT-S, a tailored genome model that develops species-aware embeddings to naturally cluster and segregate DNA sequences of different species in the embedding space.

Differentiating species from genomic sequences (i.e., DNA and RNA) is vital yet challenging, since many real-world species remain uncharacterized, lacking known genomes for reference.
Embedding-based methods are therefore used to differentiate species in an unsupervised manner.
DNABERT-S builds upon a pre-trained genome foundation model named DNABERT-2.

Built on

Supervised contrastive learning, 2021
Original
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan · 2004
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Metabat, an efficient tool for accurately reconstructing single genomes from complex microbial communities
Dongwan D Kang, Jeff Froula, Rob Egan, and Zhong Wang · 2015
Earlier work this paper cites.
dna2vec: Consistent vector representations of variable-length k-mers
Original
Patrick Ng · 2017
Earlier work this paper cites.
Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Martin Steinegger and Johannes Söding · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Earlier work this paper cites.
Deep contextualized word representations, 2018
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Earlier work this paper cites.
On the power of curriculum learning in training deep networks
Guy Hacohen and Daphna Weinshall · 2019
Earlier work this paper cites.
Metabat 2: an adaptive binning algorithm for robust and efficient genome reconstruction from metagenome assemblies
Dongwan D Kang, Feng Li, Edward Kirton, Ashleigh Thomas, Rob Egan, Hong An, and Zhong Wang · 2019
Earlier work this paper cites.

Similar

Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Cited alongside, same era.
Sentence-bert: Sentence embeddings using siamese bert-networks
Original
Nils Reimers and Iryna Gurevych · 2019
Cited alongside, same era.
Manifold mixup: Better representations by interpolating hidden states
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio · 2019
Cited alongside, same era.
A simple framework for contrastive learning of visual representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton · 2020
Cited alongside, same era.
i-mix: A domain-agnostic strategy for contrastive representation learning
Original
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee · 2020
Cited alongside, same era.
On the sentence embeddings from pre-trained language models, 2020
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li · 2020
Cited alongside, same era.
Cuco: Graph representation with curriculum contrastive learning
Guanyi Chu, Xiao Wang, Chuan Shi, and Xunqiang Jiang · 2021
Cited alongside, same era.
Simcse: Simple contrastive learning of sentence embeddings
Original
Tianyu Gao, Xingcheng Yao, and Danqi Chen · 2021
Cited alongside, same era.
Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri · 2021
Cited alongside, same era.
Improved metagenome binning and assembly using deep variational autoencoders
Jakob Nybo Nissen, Joachim Johansen, Rosa Lundbye Allesøe, Casper Kaae Sønderby, Jose Juan Almagro Armenteros, Christopher Heje Grønbech, Lars Juhl Jensen, Henrik Bjørn Nielsen, Thomas Nordahl Petersen, Ole Winther, et al · 2021
Cited alongside, same era.
Pbsim2: a simulator for long-read sequencers with a novel generative model of quality scores
Yukiteru Ono, Kiyoshi Asai, and Michiaki Hamada · 2021
Cited alongside, same era.

Then

Efficient contrastive learning via novel data augmentation and curriculum learning
Original
Seonghyeon Ye, Jiseon Kim, and Alice Oh · 2021
Later among the works it cites.
Pairwise supervised contrastive learning of sentence representations
Original
Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O Arnold, and Bing Xiang · 2021
Later among the works it cites.
Nucleosome positioning based on dna sequence embedding and deep learning
Guo-Sheng Han, Qi Li, and Ying Li · 2022
Later among the works it cites.
Critical assessment of metagenome interpretation: the second round of challenges
Fernando Meyer, Adrian Fritz, Zhi-Luo Deng, David Koslicki, Till Robin Lesker, Alexey Gurevich, Gary Robertson, Mohammed Alser, Dmitry Antipov, Francesco Beghini, et al · 2022
Later among the works it cites.
kmer2vec: A novel method for comparing dna sequences by word2vec embedding
Ruohan Ren, Changchuan Yin, and Stephen S.-T. Yau · 2022
Later among the works it cites.
Learning dialogue representations from consecutive utterances
Original
Zhihan Zhou, Dejiao Zhang, Wei Xiao, Nicholas Dingwall, Xiaofei Ma, Andrew O Arnold, and Bing Xiang · 2022
Later among the works it cites.
The nucleotide transformer: Building and evaluating robust foundation models for human genomics
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al · 2023
Later among the works it cites.
Metagenomic binning using connectivity-constrained variational autoencoders
Andre Lamurias, Alessandro Tibo, Katja Hose, Mads Albertsen, and Thomas Dyhre Nielsen · 2023
Later among the works it cites.
Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution
Original
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, et al · 2023
Later among the works it cites.
Temporal contrastive learning with curriculum
Shuvendu Roy and Ali Etemad · 2023
Later among the works it cites.
Dnabert-2: Efficient foundation model and benchmark for multi-species genome
Original
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…