2020

Molecule Attention Transformer

Maziarka, Łukasz, Danel, Tomasz, Mucha, Sławomir et al.

Understand

Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry.

To move towards this goal, we propose Molecule Attention Transformer (MAT).
Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure.
Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks.

Built on

Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules
Weininger, D · 1988
Earlier work this paper cites.
The properties of known drugs. 1. molecular frameworks
Bemis, G. W. and Murcko, M. A · 1996
Earlier work this paper cites.
Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings
Lipinski, C. A., Lombardo, F., Dominy, B. W., and Feeney, P. J · 1997
Earlier work this paper cites.
Extended-connectivity fingerprints
Rogers, D. and Hahn, M · 2010
Earlier work this paper cites.
ChEMBL: a large-scale bioactivity database for drug discovery
Gaulton, A., Bellis, L. J., Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y., McGlinchey, S., Michalovich, D., Al-Lazikani, B., and Overington, J. P · 2011
Earlier work this paper cites.
Random search for hyper-parameter optimization
Bergstra, J. and Bengio, Y · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate, 2014
Bahdanau, D., Cho, K., and Bengio, Y · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Convolutional networks on graphs for learning molecular fingerprints
Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., and Adams, R. P · 2015
Earlier work this paper cites.
Atomnet: A deep convolutional neural network for bioactivity prediction in structure-based drug discovery
Original
Wallach, I., Dzamba, M., and Heifets, A · 2015
Earlier work this paper cites.
Deep Learning
Goodfellow, I., Bengio, Y., and Courville, A · 2016
Earlier work this paper cites.
Learning to smile (s)
Original
Jastrzębski, S., Leśniak, D., and Czarnecki, W. M · 2016
Earlier work this paper cites.
Molecular graph convolutions: Moving beyond fingerprints
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., and Riley, P · 2016
Earlier work this paper cites.
Rdkit: Open-source cheminformatics software
Landrum, G · 2016
Earlier work this paper cites.
Convolutional embedding of attributed molecular graphs for physical property prediction
Coley, C. W., Barzilay, R., Green, W. H., Jaakkola, T. S., and Jensen, K. F · 2017
Earlier work this paper cites.
Neural message passing for quantum chemistry
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E · 2017
Earlier work this paper cites.
Chemception: A deep neural network with minimal chemistry knowledge matches the performance of expert-developed qsar/qspr models
Original
Goh, G., Siegel, C., Vishnu, A., Hodas, N., and Baker, N · 2017
Earlier work this paper cites.

Similar

Comparison of deep learning with multiple machine learning methods and metrics using diverse drug discovery data sets
Korotcov, A., Tkachenko, V., Russo, D. P., and Ekins, S · 2017
Cited alongside, same era.
Learning graph-level representation for drug discovery
Original
Li, J., Cai, D., and He, X · 2017
Cited alongside, same era.
Quantum-chemical insights from deep tensor neural networks
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., and Tkatchenko, A · 2017
Cited alongside, same era.
Generating focused molecule libraries for drug discovery with recurrent neural networks
Segler, M., Kogej, T., Tyrchan, C., and Waller, M · 2017
Cited alongside, same era.
Attention is all you need
Original
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Cited alongside, same era.
Graph Attention Networks
Original
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y · 2017
Cited alongside, same era.
Relational inductive biases, deep learning, and graph networks
Original
Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V. F., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gülçehre, Ç., Song, F., Ballard, A. J., Gilmer, J., Dahl, G. E., Vaswani, A., Allen, K., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D., Kohli, P., Botvinick, M., Vinyals, O., Li, Y., and Pascanu, R · 2018
Cited alongside, same era.
Neural natural language inference models enhanced with external knowledge
Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., and Wei, S · 2018
Cited alongside, same era.
Three-dimensionally embedded graph convolutional network (3DGCN) for molecule interpretation
Original
Cho, H. and Choi, I. S · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding, 2018
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
Potentialnet for molecular property prediction
Feinberg, E. N., Sur, D., Wu, Z., Husic, B. E., Mai, H., Li, Y., Sun, S., Yang, J., Ramsundar, B., and Pande, V. S · 2018
Cited alongside, same era.
Automatic chemical design using a data-driven continuous representation of molecules
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., and Aspuru-Guzik, A · 2018
Cited alongside, same era.

Then

Moleculenet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V · 2018
Later among the works it cites.
Attention augmented convolutional networks
Original
Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q. V · 2019
Later among the works it cites.
Alchemy: A quantum chemistry dataset for benchmarking ai models
Original
Chen, G., Chen, P., Hsieh, C.-Y., Lee, C.-K., Liao, B., Liao, R., Liu, W., Qiu, J., Sun, Q., Tang, J., et al · 2019
Later among the works it cites.
What does bert look at? an analysis of bert’s attention
Original
Clark, K., Khandelwal, U., Levy, O., and Manning, C. D · 2019
Later among the works it cites.
Gaussian transformer: a lightweight approach for natural language inference
Guo, M., Zhang, Y., and Liu, T · 2019
Later among the works it cites.
Advances of machine learning in molecular modeling and simulation
Haghighatlari, M. and Hachmann, J · 2019
Later among the works it cites.
Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery, 2019
Honda, S., Shi, S., and Ueda, H. R · 2019
Later among the works it cites.
Do attention heads in bert track syntactic dependencies?, 2019
Htut, P. M., Phang, J., Bordia, S., and Bowman, S. R · 2019
Later among the works it cites.
Pre-training graph neural networks
Original
Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V. S., and Leskovec, J · 2019
Later among the works it cites.
Graph warp module: an auxiliary module for boosting the power of graph neural networks
Original
Ishiguro, K., Maeda, S., and Koyama, M · 2019
Later among the works it cites.
A survey on big data and machine learning for chemistry, 2019
Jr, J. F. R., Florea, L., de Oliveira, M. C. F., Diamond, D., and Jr, O. N. O · 2019
Later among the works it cites.
A transformer model for retrosynthesis
Karpov, P., Godin, G., and Tetko, I. V · 2019
Later among the works it cites.
Deep Learning for the Life Sciences
Ramsundar, B., Eastman, P., Walters, P., Pande, V., Leswing, K., and Wu, Z · 2019
Later among the works it cites.
Self-attention based molecule representation for predicting drug-target interaction
Shin, B., Park, S., Kang, K., and Ho, J. C · 2019
Later among the works it cites.
Smiles-bert: Large scale unsupervised pre-training for molecular property prediction
Wang, S., Guo, Y., Wang, Y., Sun, H., and Huang, J · 2019
Later among the works it cites.
Building attention and edge convolution neural networks for bioactivity and physical-chemical property prediction, Sep 2019
Withnall, M., Lindelöf, E., Engkvist, O., and Chen, H · 2019
Later among the works it cites.
Analyzing learned molecular representations for property prediction
Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al · 2019
Later among the works it cites.
Self-attention with relative position representations
Shaw, P., Uszkoreit, J., and Vaswani, A · 2074
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…