In February 1988, Dave Weininger published a description of the SMILES language in the Journal of Chemical Information and Computing Science. In the 30 years since then, SMILES has become one of the de facto standards for exchanging chemical information. It has a concise yet expressive form that remains reasonably human-readable, and it is convenient for a broad spectrum of use cases whether copying and pasting a single SMILES string into a webapp, or for storing millions of molecules in a database. Given that so much chemical data are stored and exchanged through SMILES strings, if we wish to have reproducible science, it is of the utmost importance that different tools agree on a meaning of a particular SMILES string. In other words, we must quantify to what extent SMILES is a standard, versus a format that different software interprets differently. While some sources of disagreement may be bugs, others are due to different interpretations of the format. Only once such disagreements are identified and highlighted, can we as a community work to improve the tools we rely on for reproducible research. We describe the results of a benchmark comparison of tools that read SMILES strings. The task is deceptively simple: each tool tested must read each of 50 thousand SMILES strings and report the number of hydrogens on each atom. The benchmark is available online at https://github.com/nextmovesoftware/smilesreading. We describe results for more than a dozen tools, from both the open source and commercial worlds, and highlight areas of agreement and contention. Ongoing discussions with developers in relation to these results have already led to improvements in several toolkits, and we encourage more software developers to participate.

De facto standard or a free-for-all? A benchmark for reading SMILES

256th National Meeting (2018)

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.