OpenAI benchmark reveals AI struggles with drug behavior prediction
A new medicinal chemistry benchmark shows that while AI can reconstruct molecular structures, it still struggles to predict how those compounds behave in the human body.
The short version
- MedChemBench uses 117 published papers and patents to test 99 distinct problems involving 761 molecular structures.
- GPT-6 Astra scored 95.4 percent of the available credit for reconstructing molecular structures at the xhigh reasoning setting.
- The model earned only about half the credit for selecting compounds and predicting their numerical properties.
- The benchmark highlights the difficulty of balancing competing traits like protein inhibition, cell permeability, and metabolic stability.
Why it matters
This research clarifies that current AI models are better at 'drawing' molecules than understanding the complex physics of how they interact with biology. For the pharmaceutical industry, it highlights where human expertise is still essential for navigating chemical trade-offs.
The MedChemBench framework
OpenAI has introduced a new medicinal chemistry benchmark designed to test the reasoning capabilities of advanced computing models. The evaluation, called MedChemBench, specifically looks at whether these models can link molecular designs with laboratory data to interpret research results. It aims to see if AI can move beyond simple pattern matching to perform complex scientific reasoning.
The benchmark is built on a foundation of 117 published papers and patents. These sources provide the data for 99 distinct problems involving 761 different molecular structures. By using real-world research, the benchmark tests how models handle the actual data scientists use to develop new medicines.
Researchers used this framework to evaluate 11 different models across various reasoning settings. The goal is to identify progress and remaining weaknesses in how these systems handle computational reasoning. As interestingengineering.com reported, the benchmark does not claim that these systems can replace human drug discovery research today.
Structural accuracy vs. behavior
One of the most striking findings from the benchmark is the disparity between a model's ability to draw a molecule and its ability to predict its behavior. When using the xhigh reasoning configuration, GPT-6 Astra secured 95.4 percent of the possible credit for recreating molecular structures. This means the model is highly proficient at identifying atom connections, ring structures, and stereochemistry.
However, the model's performance dropped significantly when asked to predict numerical properties or select the best compounds. It earned only about half the credit for these specific tasks. This suggests that while the AI can accurately recreate a chemical drawing, it struggles to understand how specific changes to those structures influence a potential drug’s behavior.
The benchmark highlights that identifying a correct chemical structure is not the same as understanding its functional properties. These specific evaluations demand a synthesis of various laboratory observations alongside molecular modifications, rather than just the ability to mirror a chemical blueprint. This gap represents a significant hurdle for using AI in the practical stages of pharmaceutical development.
Navigating chemical trade-offs
Medicinal chemists routinely have to balance competing properties when developing new drug candidates. A compound might be very effective at blocking a disease-related protein, but it might perform poorly when the body absorbs it or if it breaks down too quickly. Improving one of these characteristics can often weaken another, making it very difficult to find compounds that meet all the necessary requirements at the same time.
The benchmark includes a specific problem regarding inhibitors of PRMT5, a protein studied in cancer research. Models were given molecular structures and early-stage results and had to predict how three related compounds would perform regarding protein inhibition, cell barrier permeability, and liver enzyme degradation. The task was designed to see if models could recognize that high activity against a biological target does not always mean better absorption or metabolic stability.
Later experimental results, which were kept secret from the models, showed distinct trade-offs between the candidates. For instance, one compound inhibited PRMT5 at a lower concentration, but another compound crossed the cell barrier more easily and lasted longer in laboratory tests. The benchmark tests whether AI can navigate these complex chemical trade-offs to identify the best overall candidate.
Precision in molecular assembly
The benchmark also evaluates the ability of models to correctly build molecules from chemical diagrams, including their atom connections and ring structures. In one specific test, models were asked to assemble a compound while maintaining an 18-atom ring. GPT-6 Astra successfully reconstructed the structure correctly in every one of its 10 responses at the highest reasoning level shared by the models.
Other models struggled significantly with this level of precision. For example, GPT-5.6 Sol produced a smaller ring containing only 15 atoms, while GPT-5.6 Luna generated fragments that were not connected. These types of errors are critical because they can change the very identity of the compound being analyzed.
If a model fails to reconstruct the correct structure, it undermines any subsequent predictions about that compound's properties. Accuracy at the structural level is a prerequisite for any meaningful analysis. The benchmark proves that while some models have reached a high level of structural fidelity, others still produce errors that would render their scientific conclusions invalid.
Current limitations and outlook
The researchers evaluated 11 models across different reasoning settings and found that higher reasoning effort generally led to better performance. However, substantial differences remained between the models, and none were able to perfectly master the complex requirements of medicinal chemistry. The team emphasized that these results do not prove that models can currently discover new drugs or improve laboratory decisions.
One reason for these limitations is that the benchmark relies on historical research. Because these models may have been exposed to published sources during their training, it is impossible to rule out prior knowledge. Additionally, the models completed all tasks without the help of external tools, human feedback, or the ability to run new experiments.
Ultimately, MedChemBench serves as a way to track progress and identify specific weaknesses in computational reasoning. It provides a roadmap for what needs to be improved before AI can become a standard part of the drug discovery pipeline. It highlights the ongoing need for human oversight in interpreting the complex interactions between chemistry and biology.
What to watch
- How models will perform when given access to real-time laboratory feedback.
- Whether future models can better navigate the trade-offs between drug absorption and protein inhibition.
Try it
Look up the PRMT5 protein to see how researchers balance drug potency with metabolic stability.
Sources and links
Read the original at interestingengineering.com
First reported by interestingengineering.com. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.