Molecular representation learning with language models and domain-relevant auxiliary tasks

Understand

We apply a Transformer architecture, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems.

• We study the impact of using different combinations of self-supervised tasks for pre-training, and present our results for the established Virtual Screening and QSAR benchmarks.
• We show that: i) The selection of appropriate self-supervised task(s) for pre-training has a significant impact on performance in subsequent downstream tasks such as Virtual Screening.
• ii) Using auxiliary tasks with more domain relevance for Chemistry, such as learning to predict calculated molecular properties, increases the fidelity of our learnt representations.