Fetching the paper…
Reading the bibliography…
Similar to natural language models, pre-trained genome language models are proposed to capture the underlying intricacies within genomes with unsupervised sequence modeling.
Vector quantization
Gray, R · 1984
Earlier work this paper cites.
Gcta: a tool for genome-wide complex trait analysis
Yang, J., Lee, S. H., Goddard, M. E., and Visscher, P. M · 2011
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Bengio, Y., Léonard, N., and Courville, A · 2013
Earlier work this paper cites.
Epd and epdnew, high-quality promoter resources in the next-generation sequencing era
Dreos, R., Ambrosini, G., Cavin Périer, R., and Bucher, P · 2013
Earlier work this paper cites.
Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks
Kelley, D. R., Snoek, J., and Rinn, J. L · 2015
Earlier work this paper cites.
Genetic studies of body mass index yield new insights for obesity biology
Locke, A. E., Kahali, B., Berndt, S. I., Justice, A. E., Pers, T. H., Day, F. R., Powell, C., Vedantam, S., Buchkovich, M. L., Yang, J., et al · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Sennrich, R., Haddow, B., and Birch, A · 2015
Earlier work this paper cites.
Predicting effects of noncoding variants with deep learning–based sequence model
Zhou, J. and Troyanskaya, O. G · 2015
Earlier work this paper cites.
Ba, J., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
Deepcpg: accurate prediction of single-cell dna methylation states using deep learning
Angermueller, C., Lee, H. J., Reik, W., and Stegle, O · 2017
Earlier work this paper cites.
N-gram language modeling using recurrent neural network estimation
Chelba, C., Norouzi, M., and Bengio, S · 2017
Earlier work this paper cites.
Chromatin accessibility prediction via convolutional long short-term memory networks with k-mer embedding
Min, X., Zeng, W., Chen, N., Chen, T., and Jiang, R · 2017
Earlier work this paper cites.
Grad-cam: Visual explanations from deep networks via gradient-based localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D · 2017
Earlier work this paper cites.
Neural discrete representation learning
Van Den Oord, A., Vinyals, O., and Kavukcuoglu, K · 2017
Earlier work this paper cites.
Dna methylation-based forensic age prediction using artificial neural networks and next generation sequencing
Vidaki, A., Ballard, D., Aliferi, A., Miller, T. H., Barron, L. P., and Court, D. S · 2017
Earlier work this paper cites.
10 years of gwas discovery: biology, function, and translation
Visscher, P. M., Wray, N. R., Zhang, Q., Sklar, P., McCarthy, M. I., Brown, M. A., and Yang, J · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Earlier work this paper cites.
Kudo, T. and Richardson, J · 2018
Earlier work this paper cites.
Umap: Uniform manifold approximation and projection for dimension reduction
McInnes, L., Healy, J., and Melville, J · 2018
Earlier work this paper cites.
Mobilenetv2: Inverted residuals and linear bottlenecks
Sandler, M., Howard, A. G., Zhu, M., Zhmoginov, A., and Chen, L.-C · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
iproep: a computational predictor for predicting promoter
Lai, H.-Y., Zhang, Z.-Y., Su, Z.-D., Su, W., Ding, H., Chen, W., and Lin, H · 2019
Cited alongside, same era.
Decoupled weight decay regularization
Loshchilov, I. and Hutter, F · 2019
Cited alongside, same era.
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., and Vinyals, O · 2019
Cited alongside, same era.
Splicefinder: ab initio prediction of splice sites using convolutional neural network
Wang, R., Wang, Z., Wang, J., and Li, S · 2019
Flashattention: Fast and memory-efficient exact attention with io-awareness
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C · 2022
Later among the works it cites.
Masked autoencoders are scalable vision learners
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R · 2022
Later among the works it cites.
Bert-promoter: An improved sequence-based predictor of dna promoter using bert pre-trained model and shap feature selection
Le, N. Q. K., Ho, Q.-T., Nguyen, V.-N., and Chang, J.-S · 2022
Later among the works it cites.
Autoregressive image generation using residual quantization
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S · 2022
Later among the works it cites.
A convnet for the 2020s
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Determinants of enhancer and promoter activities of regulatory elements
Andersson, R. and Sandelin, A · 2020
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Jukebox: A generative model for music
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I · 2020
Cited alongside, same era.
Expanded encyclopaedias of dna elements in the human and mouse genomes
Moore, J. E., Purcaro, M. J., Pratt, H. E., Epstein, C. B., Shoresh, N., Adrian, J., Kawli, T., Davis, C. A., Dobin, A., et al · 2020
Cited alongside, same era.
Deep high-resolution representation learning for visual recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al · 2020
Cited alongside, same era.
Taming transformers for high-resolution image synthesis
Esser, P., Rombach, R., and Ommer, B · 2021
Cited alongside, same era.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al · 2022
Later among the works it cites.
Pango lineage designation and assignment using sars-cov-2 spike gene nucleotide sequences
O’Toole, Á., Pybus, O. G., Abram, M. E., Kelly, E. J., and Rambaut, A · 2022
Later among the works it cites.
The nucleotide transformer: Building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Carranza, N. L., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al · 2023
Later among the works it cites.
Gena-lm: A family of open-source foundational models for long dna sequences. biorxiv
Fishman, V., Kuratov, Y., Petrov, M., Shmelev, A., Shepelin, D., Chekanov, N., Kardymon, O., and Burtsev, M · 2023
Later among the works it cites.
Language quantized autoencoders: Towards unsupervised text-image alignment
Liu, H., Yan, W., and Abbeel, P · 2023
Later among the works it cites.
Finite scalar quantization: Vq-vae made simple
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M · 2023
Later among the works it cites.
Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution
Nguyen, E. D., Poli, M., Faizi, M., Thomas, A. W., Birch-sykes, C. J., Wornow, M., Patel, A., Rabideau, C. M., Massaroli, S., Bengio, Y., Ermon, S., Baccus, S. A., and Ré, C · 2023
Later among the works it cites.
Deep sampling of grna in the human genome and deep-learning-informed prediction of grna activities
Zhang, H., Yan, J., Lu, Z., Zhou, Y., Zhang, Q., Cui, T., Li, Y., Chen, H., and Ma, L · 2023
Later among the works it cites.
Are genomic language models all you need? exploring genomic language models on protein downstream tasks
Boshar, S., Trop, E., de Almeida, B. P., and PIERROT, T · 2024
Closest in time.
Bend: Benchmarking dna language models on biologically meaningful tasks
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., and Boomsma, W · 2024
Closest in time.
Sequence modeling and design from molecular to genome scale with evo
Nguyen, E., Poli, M., Durrant, M. G., Thomas, A. W., Kang, B., Sullivan, J., Ng, M. Y., Lewis, A., Patel, A., Lou, A., et al · 2024
Closest in time.
Saprot: Protein language modeling with structure-aware vocabulary
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F · 2024
Closest in time.
Dnabert-2: Efficient foundation model and benchmark for multi-species genome
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., and Liu, H · 2024
Closest in time.