OpenAI launches BioBasicsBench with Ligo to test whether AI agents can interpret raw evidence from 100 routine wet-lab tasks. GPT‑6 Astra earned 66.7% of available rubric credit at the shared xhigh reasoning setting, compared with 43.8% for GPT‑5.6 Sol.

 

The score measures partial credit against expert criteria, not the percentage of experiments completed correctly. OpenAI’s own examples also show Astra selecting the wrong colonies, discounting contradictory observations and overcounting colonies by about 12%, placing clear limits on the result.

 

BioBasicsBench adds four elements missing from many science evaluations:

  • Raw gels, plates, microscopy and instrument exports.
  • One hundred tasks drawn from working scientists’ experiments.
  • Rubrics that reward recognizing insufficient evidence.
  • Published successes and concrete model failures.

 

Related Research

 

OpenAI Launches BioBasicsBench With Ligo

OpenAI introduced the benchmark on November 6 with Ligo, a biotechnology company focused on enzyme design. Its 100 tasks cover protein methods, imaging and cytometry, quantification and assays, nucleic-acid methods and molecular cloning.

 

Scientists built tasks from their own experimental data and paired each question with an expert-written grading rubric. Every task received an internal review by a different Ligo scientist, an external review by a PhD-level wet-lab scientist and a final internal review.

 

The agent returns a structured conclusion, measurement or recommendation and supplies reasoning when requested. Scoring awards credit criterion by criterion, allowing a partially correct analysis to receive a partial score and rewarding an agent that correctly says the evidence cannot support a conclusion.

 

That design targets an ordinary scientific bottleneck. Published figures are usually selected and cleaned for communication, while day-to-day laboratory evidence includes failed runs, ambiguous images, inconsistent labels and controls that invalidate an otherwise plausible answer.

 

GPT‑6 Astra Scores 66.7% at Xhigh

OpenAI evaluated seven of its own models across 37 model and reasoning-effort settings. Each setting ran five attempts on every task, producing 18,500 results. The reported metric is mean reward: the average share of rubric credit earned across tasks.

 

At xhigh, the highest reasoning setting reported for every model, GPT‑6 Astra scored 66.7%, with a 95% confidence interval of 60.3% to 73.1%. GPT‑6.1 Sol reached 62.0%, while GPT‑5.6 Sol earned 43.8%.

 

Astra used an average of 4,704 output tokens per task at that setting. GPT‑6.1 Sol used 5,238 and GPT‑5.6 Sol used 9,099, so the reported improvement did not come from generating more text than those two comparison systems.

 

The leaderboard remains a provider-run comparison among OpenAI models. No outside model results, human baseline or independent reproduction accompanied the launch, and the percentage should not be read as a success rate for real experiments or autonomous laboratory operation.

 

Raw Lab Evidence Changes the Test

The inputs include 31 tasks with gels or blots, 27 with protocols or notes, 20 with tables or instrument exports and 18 with photographs of benches, apparatus or reagents. Other tasks use microscopy stacks, sequencing files, instrument screens, videos and agar plates.

 

Those categories overlap because one task may combine several data types. A model might need to inspect a gel, consult a ladder chart, read a handwritten note and decide whether a control was valid before it can answer the biological question.

 

OpenAI highlighted a cloning task where GPT‑5.6 Sol treated a blank negative-control plate as reliable. Astra noticed that the plate label named the wrong antibiotic and rejected it, showing how a small visual detail can change the interpretation of an entire experiment.

 

In another example, cells appeared to vanish across microscope frames. The earlier model searched for a biological explanation, while Astra recognized that the sequence was a z-stack moving through different focal depths. Both examples depend on experimental context as well as image recognition.

 

Astra Still Misreads Controls and Counts

Higher performance did not make the system dependable. In a colony-PCR task, Astra grouped bands visually but attached the wrong experimental meaning to them, selecting colonies 4, 6, 7 and 8 instead of the reference choices 5, 9 and 10.

 

Another answer acknowledged that cell compartments could appear lighter or darker as focus changed, then generalized only from the darker examples. The model inspected contradictory evidence but failed to carry that evidence into its conclusion.

 

A colony-counting task exposed a simpler error. Astra merged close neighbors, missed visible colonies and marked empty agar, producing an overcount of about 12%. In a real workflow, that error would inflate the estimated efficiency of introducing DNA into cells by a similar proportion.

 

The launch page does not link a downloadable task set, evaluation code, complete rubrics or raw model outputs. Those materials, outside model runs and a qualified human baseline would allow researchers to test whether the reported ordering survives independent scrutiny.

 

BioBasicsBench is therefore most useful as a specification for a neglected problem: interpreting messy evidence before a scientific claim is made. Its next measure of value will be reproducibility and transfer to independent laboratories, not another provider-reported score alone.

 

Illustrative feature photograph: quality-control analyst Ashley Doran prepares mitochondrial amplification at the Armed Forces DNA Identification Laboratory, photographed by Staff Sgt. Alexandra Minor on July 19, 2024. The public-domain image does not depict BioBasicsBench; it was cropped and resized.