OpenAI launches BioBasicsBench to test AI agents on wet-lab data

Moving beyond polished papers

OpenAI recently introduced BioBasicsBench, a new benchmark designed to test how AI agents handle real-world laboratory data. Unlike many existing evaluations that use charts or images selected specifically for scientific publications, this benchmark focuses on routine wet-lab work. This includes the messy reality of failed experiments, ambiguous images, and conflicting observations that scientists face every day.

The benchmark was developed in partnership with Ligo, an enzyme-design company. It consists of 100 tasks where AI agents must interpret raw evidence to reach a conclusion. Scientists created these tasks from their own experiments and established specific grading rubrics to evaluate the models.

The goal is to see if AI can move past simple pattern recognition to perform actual reasoning. The tasks cover a wide range of fields, including protein methods, imaging, cytometry, quantification, nucleic acid methods, and molecular cloning. By using raw data, the benchmark tests the AI's ability to troubleshoot and recognize when evidence is insufficient to justify a specific claim.

How the benchmark works

The evaluation uses a variety of inputs to simulate a laboratory environment. These include various visual records such as blot and gel images, microscopy stacks, and colony plates, along with sequencing data, plate-reader exports, and videos of lab protocols. Some tasks combine multiple evidence types with supporting reference sheets, such as plasmid maps or ladder charts.

Agents are asked to provide structured conclusions, measurements, or recommendations based on these inputs. The grading rubrics award credit for several factors, including the accuracy of numerical results and the quality of the supporting reasoning. Crucially, models also receive credit for correctly identifying when the provided evidence is not enough to reach a conclusion.

To ensure high-quality testing, each task underwent a rigorous review process. This included an internal check by a Ligo scientist who did not write the task, a review by a scientist with a PhD in wet-lab work, and a final internal review. The total evaluation covered 37 different model and reasoning-effort settings, resulting in 18,500 total results.

Astra vs. Sol performance

The results show a significant gap between different models at the highest reasoning-effort setting, known as xhigh. GPT-6 Astra reached a mean reward of 66.7%, while GPT-5.6 Sol scored 43.8%. While Astra was more successful, it produced fewer output tokens per task, averaging 4,704 compared to 9,099 for Sol.

OpenAI highlighted specific instances where Astra showed superior multimodal handling. In one cell-imaging task, Astra recognized that overlapping detection windows could cause interference from a second dye and suggested specific controls. In another DNA-cloning task, Astra noticed a plate was labeled for the wrong antibiotic and rejected it as a reliable control, whereas Sol accepted it.

Astra also demonstrated an ability to interpret complex visual data, such as recognizing a z-stack—a series of images at different depths—rather than assuming cells were disappearing. These examples suggest that stronger native multimodal capabilities are important for routine wet-lab analysis. However, the models still struggle with consistency, such as failing to separate close neighbors in colony-counting tasks.

The path to lab assistance

Despite the improvements, OpenAI cautions that higher scores do not yet mean the models are ready for reliable laboratory judgment. In some cases, Astra made errors that would have real-world consequences, such as selecting the wrong lanes in a PCR task. Such a mistake would lead to the purification of the incorrect plasmid in a real lab setting.

The benchmark also revealed issues with generalization. For example, Astra noted that contrast varied with focus but still made incorrect generalizations about the darkness of cell compartments. These failures highlight the difficulty of carrying accurate observations through to a final, sound experimental decision without losing logic along the way.

Moving forward, OpenAI intends to use these findings to inform the assessment of biosafety and biosecurity consequences. The practical bar for AI in the lab is high: an agent must be able to check controls and withhold conclusions when evidence is lacking. BioBasicsBench provides a target for these improvements, showing the gap that still exists between observing data and making correct experimental decisions.