SIAgentNews.com
← All news
DevelopingOpenAIResearch

A new benchmark tests whether AI can judge an engineering design, not just log it

EJBench uses real battery cooling simulations to test AI engineering judgement. The best of ten frontier models scored 0.71 and sounded equally sure when wrong.

· First reported by arxiv.org

0.71Best score among ten frontier models (GPT-6 Astra)

The short version

  • EJBench tests prediction and decisions from a few earlier results, without running new simulations.
  • Tasks come from a battery cold plate heat-transfer design campaign, in three task families.
  • Answers come from converged simulations, not from an AI judge.
  • Best of ten frontier models, GPT-6 Astra, scored 0.71; stated confidence barely differed between right and wrong answers.
  • A fine-tuned Qwen3.5-9B rose from 0.071 to 0.106 on the held-out test split.

Why it matters

Many teams use AI to organise runs, tabulate results and draft reports. This benchmark tries to measure the harder step of deciding what the results mean for a design. Its finding that models sound equally confident when wrong is a warning for anyone relying on a model's tone as a signal of reliability.

Judgement, not software operation

A paper posted to arxiv.org introduces EJBench, a test of what its authors call engineering judgement. The question is whether a model can look at a few earlier results for a design, predict how that design will behave under new conditions, and then make a decision based on that prediction.

That is different from many existing engineering benchmarks, which check whether an AI agent can operate simulation or CAD (computer-aided design) software. EJBench is software-agnostic. It asks what an engineer would conclude without running a new simulation.

The title, "Bookkeeping Is Not Judgement," sums up the distinction. Logging and tidying results is one skill. Knowing what those results imply for a new situation is another.

Where the tasks come from

The tasks are drawn from a real design campaign on battery cold plates. These are cooling components, and the physics involved is conjugate heat transfer, meaning heat moving between solid parts and a flowing fluid.

The benchmark has three families of tasks:

  • Matching unlabelled test results to the operating points that produced them.
  • Choosing which stored runs to reveal before signing off a design.
  • Ranking new duties by peak cell temperature.

The second task is the most telling. It tests whether a model knows what evidence it needs before it commits, not just whether it can compute an answer.

Every answer is checked against a converged simulation, meaning one that has settled to a stable result. The authors did not use another language model as the grader, which removes one common source of doubt in AI evaluation.

What the scores show

The authors tested ten frontier models. The best, GPT-6 Astra, scored 0.71. That is well short of a perfect result, and the paper does not describe it as a solved problem.

The second finding may matter more in practice. Models stated almost the same confidence on tasks they got wrong as on tasks they got right. In engineering, a wrong answer delivered with calm certainty is harder to catch than one that comes with a warning.

For a reader who is not an engineer, the lesson carries over. A fluent, confident answer from a chatbot is not evidence that it is correct, and this benchmark gives a concrete case where the confidence was not informative.

Can a small model be trained for it?

The authors also tried to improve a small open model, Qwen3.5-9B. They used LoRA, a lightweight way to adjust a model without retraining all of it, together with reinforcement learning, where the model is rewarded for good outcomes. The reward was the share of answer criteria that the simulation results confirmed.

On the held-out test split, data the model never trained on, its score rose from 0.071 to 0.106. That is a gain of about half again on a very low base, and it remains a long way below the 0.71 the best frontier model reached. The source does not say how the two figures compare directly, so treat that gap loosely.

The takeaway is modest: simulation-checked rewards can nudge a model in the right direction, but this one experiment does not show that small models can be taught engineering judgement.

What is released and what to be careful about

The authors say they are releasing the train, validation and test splits, the evaluation harness (the code that runs and scores a model) and a leaderboard. That lets other groups test their own models on the same material.

There are limits. All the tasks come from one design campaign about one type of component, so the scores say little about, say, bridges or circuits. The paper is also a fresh preprint, submitted on 16 Nov 2026, and has not been independently checked here.

Our earlier mention of EJBench had only the title to go on. The abstract now fills in the results and shows the paper is about more than the bookkeeping-versus-judgement slogan.

What to watch

  • Whether other groups publish results on the EJBench leaderboard, and whether the ranking holds up outside battery cooling.
  • Whether models can be made to express confidence that actually tracks how often they are right.

Try it

Pick ten technical questions whose answers you already know. Ask a chatbot to answer each and state its confidence, then check whether its confidence is lower on the ones it got wrong.

Sources and links

First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news