An AI foundation model that learns the grammar of molecules

Meet MoLFormer-XL, a pre-trained model that infers the structure of molecules from simple representations, making it faster and easier to screen molecules for new applications or create them from scratch.

Large pre-trained models are fast becoming AI’s Swiss Army knife. Once limited to summarizing text and translating languages, they can now write code, compose music, and answer obscure questions at length.

Now there’s a new skill to add to their repertoire: the ability to infer the shapes and properties of molecules to predict how they might behave and to propose entirely new ones.

Most molecular models need estimates or measurements of a molecule’s 3D shape to accurately predict many of its properties. Chemists can extract this information through simulations or lab experiments, but it’s an imperfect, expensive process that can take months to years. Perhaps unsurprisingly, we have detailed structures for only a few million molecules out of the trillions upon trillions potentially out there.

But now, there could be a way to eliminate this bottleneck in the discovery process with the help of AI. Introducing MoLFormer-XL, the latest addition to the MoLFormer family of foundation models for molecular discovery. MoLFormer-XL has been pre-trained on 1.1 billion molecules represented as machine-readable strings of text. From these simple and accessible chemical representations, it turns out that a transformer can extract enough information to infer a molecule’s form and function.

Instead of training a model on thousands to millions of examples of molecules with detailed structural and property information, we can leverage a dataset that is at least 1,000 times larger. From this mountain of examples, we found that MoLFormer-XL could more easily learn a variety of downstream property prediction tasks. We report our results in the latest issue of Nature Machine Intelligence.

We found that MoLFormer–XL could predict a molecule’s physical properties, like its solubility, its biophysical properties, like its anti-viral activity, and its physiological properties, like its ability to cross the blood-brain barrier. It could even predict quantum properties, like a molecule’s bandgap energies, an indicator of how well it converts sunlight to energy.

MoLFormer-XL outperformed other chemical language models at nearly a dozen molecular property benchmarks. It also did better than graph models trained on molecules with precise 3D geometric labels. MoLFormer-XL’s ability to efficiently learn the structures of such varied molecules could make it a powerful tool for discovery new molecules by their desired properties.

Many molecular models today rely on graph neural network architectures that predict molecular behavior from a molecule’s 2D or 3D structure. But graph models often require extensive simulations or experiments, or use complex mechanisms, to capture atomic interactions within a molecule. Most graph models, as a result, are limited to datasets of about 100,000 molecules, sharply limiting their ability to make broad predictions.

MoLFormer-XL, by contrast, rests on a foundation of more than 1.1 billion molecules, each represented by a compact snippet of text belonging to the SMILES notation system (short for Simplified Molecular Input Line Entry System). Each SMILES string describes how atoms in the types of organic small molecules targeted for drug and material discovery are bonded and arranged in a so-called molecular graph. At scale, this meager information contains a wealth of structural clues.