2022

Structured Multi-task Learning for Molecular Property Prediction

Liu, Shengchao, Qu, Meng, Zhang, Zuobai et al.

Understand

Multi-task learning for molecular property prediction is becoming increasingly important in drug discovery.

However, in contrast to other domains, the performance of multi-task learning in drug discovery is still not satisfying as the number of labeled data for each task is too limited, which calls for additional data to complement the data scarcity.
In this paper, we study multi-task learning for molecular property prediction in a novel setting, where a relation graph between tasks is available.
We first construct a dataset (ChEMBL-STRING) including around 400 tasks as well as a task relation graph.

Built on

Pre-training graph neural networks
Original
Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J. (2019a) · 1905
Earlier work this paper cites.
Strategies for pre-training graph neural networks
Original
Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J. (2019b) · 1905
Earlier work this paper cites.
Stochastic relaxation, gibbs distributions, and the bayesian restoration of images
Geman, S. and Geman, D. (1984) · 1984
Earlier work this paper cites.
A view of the em algorithm that justifies incremental, sparse, and other variants
Neal, R. M. and Hinton, G. E. (1998) · 1998
Earlier work this paper cites.
Gradient surgery for multi-task learning
Original
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. (2020) · 2001
Earlier work this paper cites.
Generalized energy based models
Original
Arbel, M., Zhou, L., and Gretton, A. (2020) · 2003
Earlier work this paper cites.
A tutorial on energy-based learning
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. (2006) · 2006
Earlier work this paper cites.
Learning energy-based model with flow-based backbone by neural transport mcmc
Original
Nijkamp, E., Gao, R., Sountsov, P., Vasudevan, S., Pang, B., Zhu, S.-C., and Wu, Y. N. (2020) · 2006
Earlier work this paper cites.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009) · 2009
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. and Hyvärinen, A. (2010) · 2010
Earlier work this paper cites.
Merck molecular activity challenge
Merck (2012) · 2012
Earlier work this paper cites.
A fast and simple algorithm for training neural probabilistic language models
Original
Mnih, A. and Teh, Y. W. (2012) · 2012
Earlier work this paper cites.
Multi-task neural networks for QSAR predictions
Original
Dahl, G. E., Jaitly, N., and Salakhutdinov, R. (2014) · 2014
Earlier work this paper cites.
Generative modeling of convolutional neural networks
Original
Dai, J., Lu, Y., and Wu, Y.-N. (2014) · 2014
Earlier work this paper cites.
Deep learning as an opportunity in virtual screening
Unterthiner, T., Mayr, A., Klambauer, G., Steijaert, M., Wegner, J. K., Ceulemans, H., and Hochreiter, S. (2014) · 2014
Earlier work this paper cites.

Similar

Convolutional networks on graphs for learning molecular fingerprints
Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., and Adams, R. P. (2015) · 2015
Cited alongside, same era.
Massively multitask networks for drug discovery
Original
Ramsundar, B., Kearnes, S., Riley, P., Webster, D., Konerding, D., and Pande, V. (2015) · 2015
Cited alongside, same era.
Structured prediction energy networks
Belanger, D. and McCallum, A. (2016) · 2016
Cited alongside, same era.
Semi-supervised classification with graph convolutional networks
Original
Kipf, T. N. and Welling, M. (2016) · 2016
Cited alongside, same era.
Cross-stitch networks for multi-task learning
Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016) · 2016
Cited alongside, same era.
Deep multi-task learning with low level tasks supervised at lower layers
Søgaard, A. and Goldberg, Y. (2016) · 2016
Cited alongside, same era.
A theory of generative convnet
Xie, J., Lu, Y., Zhu, S.-C., and Wu, Y. (2016) · 2016
Cited alongside, same era.
Neural message passing for quantum chemistry
Original
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017) · 2017
Cited alongside, same era.
Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification
Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., and Feris, R. (2017) · 2017
Cited alongside, same era.
Attention-based lstm with multi-task learning for distant speech recognition
Original
Zhang, Y., Zhang, P., and Yan, Y. (2017) · 2017
Cited alongside, same era.
Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks
Chen, Z., Badrinarayanan, V., Lee, C.-Y., and Rabinovich, A. (2018) · 2018
Cited alongside, same era.
UniProt: a worldwide hub of protein knowledge
Consortium, T. U. (2018) · 2018
Cited alongside, same era.

Then

Practical model selection for prospective virtual screening
Liu, S., Alnammi, M., Ericksen, S. S., Voter, A. F., Ananiev, G. E., Keck, J. L., Hoffmann, F. M., Wildman, S. A., and Gitter, A. (2018) · 2018
Later among the works it cites.
Large-scale comparison of machine learning methods for drug target prediction on chembl
Mayr, A., Klambauer, G., Unterthiner, T., Steijaert, M., Wegner, J. K., Ceulemans, H., Clevert, D.-A., and Hochreiter, S. (2018) · 2018
Later among the works it cites.
ChEMBL: towards direct deposition of bioassay data
Mendez, D., Gaulton, A., Bento, A. P., Chambers, J., De Veij, M., Félix, E., Magariños, M., Mosquera, J., Mutowo, P., Nowotka, M., Gordillo-Marañón, M., Hunter, F., Junco, L., Mugumbate, G., Rodriguez-Lopez, M., Atkinson, F., Bosc, N., Radoux, C., Segura-Cabrera, A., Hersey, A., and Leach, A. (2018) · 2018
Later among the works it cites.
Moleculenet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V. (2018) · 2018
Later among the works it cites.
How powerful are graph neural networks?
Original
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018) · 2018
Later among the works it cites.
In-silico molecular binding prediction for human drug targets using deep neural multi-task learning
Lee, K. and Kim, D. (2019) · 2019
Later among the works it cites.
String v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets
Szklarczyk, D., Gable, A. L., Lyon, D., Junge, A., Wyder, S., Huerta-Cepas, J., Simonovic, M., Doncheva, N. T., Morris, J. H., Bork, P., et al. (2019) · 2019
Later among the works it cites.
Self-supervised graph transformer on large-scale molecular data
Rong, Y., Bian, Y., Xu, T., Xie, W., Wei, Y., Huang, W., and Huang, J. (2020) · 2020
Later among the works it cites.
Evaluating scalable supervised learning for synthesize-on-demand chemical libraries
Alnammi, M., Liu, S., Ericksen, S. S., Ananiev, G. E., Voter, A. F., Guo, S., Keck, J. L., Hoffmann, F. M., Wildman, S. A., and Gitter, A. (2021) · 2021
Later among the works it cites.
An analysis of attentive walk-aggregating graph neural networks
Original
Demirel, M. F., Liu, S., Garg, S., and Liang, Y. (2021) · 2021
Later among the works it cites.
Pre-training molecular graph representation with 3d geometry
Original
Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J. (2021) · 2021
Later among the works it cites.
How to train your energy-based models
Original
Song, Y. and Kingma, D. P. (2021) · 2021
Later among the works it cites.
Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models
Wang, Z., Tsvetkov, Y., Firat, O., and Cao, Y. (2021) · 2021
Later among the works it cites.
Do transformers really perform badly for graph representation?
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. (2021) · 2021
Later among the works it cites.
Is multitask deep learning practical for pharma?
Ramsundar, B., Liu, B., Wu, Z., Verras, A., Tudor, M., Sheridan, R. P., and Pande, V. (2017) · 2076
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…