Rogo's new finance test finds no single best AI model, and big gains from picking per task
Rogo's 928-question benchmark grades AI agents on finance work step by step, and finds the top models each win in different areas.
The short version
- 928 questions written by 52 finance practitioners, with rubrics averaging 17 line items each
- Ten models tested: six closed, four open-weight, each run three times per question
- Rubric scores ran about 16 percentage points above final-answer accuracy
- Opus 4.7, GPT-5.5 and Sonnet 4.6 finished within 0.3 points, but each led in different work
- A simple router beat the best single model by 4.5 points on the rubric score
Why it matters
If you buy or use AI for analysis work, the question 'which model is best?' may be the wrong one. The better question is which model suits which task, and what each answer costs. Keep in mind that Rogo sells finance AI and wrote this test itself, so treat the results as one informed view, not a final verdict.
A test built around real desks
As rogo.com reported, the company has released Big Finance Bench (BFB), a benchmark, meaning a standard exam for comparing AI models. It contains 928 questions drawn from the kinds of work its customers do at banks, private equity firms, research shops and asset managers.
Rogo's argument is that earlier finance tests, which mostly check short answers or arithmetic against a single document, miss how the job works. Context matters: the same valuation multiple can be sensible in one industry and useless in another. Role matters too, since a credit underwriter and a merger banker worry about different risks.
The questions fall into four groups: skills specific to a desk (374), metrics (301), earnings and financial-statement analysis (137), and scenario forecasting (116).
Grading the steps, not just the answer
The questions were written by 52 practitioners, who also wrote the grading rubrics. A panel of 12 senior reviewers checked each one. Questions are pinned to specific periods such as 2Q25, so the right answer does not drift over time.
Each rubric averages 17 line items, labelled as retrieval (finding the right document), definition (using the right meaning of a term) or calculation. Items carry weights from 1 to 10. Across the benchmark that adds up to 15,656 criteria and 36,241 weighted points.
The effect is partial credit. A model that finds the right filing but slips on the math keeps the points for the first step and loses them only where it went wrong.
What the ten models showed
Rogo tested ten systems, six closed and four open-weight, from Anthropic, OpenAI, Google and several other labs. Each ran every question three times in a bare-bones agent setup. An agent here is an AI that takes actions, and this one could search SEC EDGAR (the US filings database), browse the public web and run code in a Python sandbox.
First finding: scoring only the final answer hides a lot. Rubric scores came out about 16 percentage points higher than final-answer accuracy. Many models did the right research and reasoning, then lost everything on a small slip at the end. Rankings shifted too. Rogo says Gemini 3.1 Pro tends to sound confident without finishing the process, while Gemini 3 Flash works more carefully and does better on rubric grading.
Second finding: the top three, Opus 4.7, GPT-5.5 and Sonnet 4.6, sit less than 0.3 points apart overall. Yet none of them wins across the board. GPT-5.5 is strongest on capital structure and M&A. Sonnet 4.6 does best at judging earnings quality and reading financial statements. Opus 4.7 stands out on private capital and forecasting.
Routing and the cost question
If each model has a specialty, you can send each question to the right one. Rogo tested this on paper. The best single model scored 58.8% on the rubric. A coarse router that picks by workflow and source type added 4.5 points. An 'oracle' that somehow knows the best of all ten responses for each question added 13.2 points. The oracle is a hypothetical ceiling, not a product anyone can use.
Cost is the other half. Rogo says its customers' token use, the unit AI services bill by, has risen about tenfold as models improved. In its test, one task cost about $1.26 on Claude Opus 4.7 but nearer $0.02 through a cheaper model. Its view is that once a system fetches the right documents and definitions, the strong models do the arithmetic about equally well. So much of the accuracy gap comes from earlier steps, and much of the cost comes from the surrounding setup.
Rogo concludes that the winners will be firms with good routing, not just big contracts with one model vendor, and says it will rerun BFB regularly as new models arrive.
What is public, and what to be careful about
Rogo released the paper, a 50-question sample set on Hugging Face with answers, rubrics and sources, and the agent code on GitHub. It says the sample's rankings closely track the full set. Outsiders can therefore try to reproduce the results.
Caveats are worth holding onto. The company runs and funds the benchmark and sells to this market. The routing gains are measured on its own questions, and the oracle figure is an upper bound.
What to watch
- Will independent researchers reproduce the rankings using the public 50-question set?
- How do newer model releases change which model leads in which kind of finance work?
- Will Rogo publish cost-per-correct-answer results alongside the accuracy scores?
Try it
Next time you evaluate an AI tool for analysis work, ask which model runs underneath and whether it can switch models by task. Then test it on a few of your own real questions.
Sources and links
- Reportingrogo.com: the article we sourced
First reported by rogo.com. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.