Attention-based transformers underly the architecture of many generalizable, state-of-the-art AI models for natural language processing, computer vision and molecular design tasks. AlphaFold2 has demonstrated how this simple concept can be applied to accurately predict folded configurations of proteins and multimers. The model has been shown to learn an energy function for protein folding. Recent progress has led to several deep-learning architectures for predicting protein-ligand binding. They hold the promise to accelerate and greatly simplify virtual screening campaigns for drug discovery. Examples include equivariant transformers that predict spatial configurations of protein-ligand complexes, and models that use graph- or text-based molecular representations as inputs. Here, we explore the opportunity for using transformer models to incorporate sequence information in addition to three-dimensional structures, taking advantage of the abundant availability of unlabeled training data. We leverage an existing architecture for natural language processing that accepts text-based representations of molecules. To achieve generalizability, text embeddings are pre-trained on a large corpus of chemical sequence data, rather than initialized randomly. We show the importance of using pre-trained weights to achieve generalizable predictions. Our model could also be applied in reverse screening situations where a ligand is given, but the receptor is unknown, e.g., to predict off-target effects. Transformers are computationally demanding to train, and therefore could emerge as prime examples of advanced AI and HPC uses in pharmaceutical industry. On the other hand, their fast inference performance could greatly accelerate time-to-solution in the initial stages of drug discovery. The potential of AI models to learn generalizable molecular representations will offer useful insights for molecular design and chemical understanding.

Transformers for protein-ligand binding

Fall 2022 abstracts, posters, and presentations will launch in mid-September.

ACS Fall 2022

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.