SIAgentNews.com
← All news

New benchmark tests whether AI agents can build RAG systems on their own

A research paper's test, AutoRAGBench, finds an AI agent can beat a human-designed baseline at building document-search systems, which could change how such tools get made.

· First reported by arxiv.org

56.1Top agent's average score, versus 44.4 for the human baseline

The short version

  • AutoRAGBench covers nine RAG development tasks, with no starter code or prescribed strategy
  • Each agent gets six hours on a single H800 GPU per task
  • Nine agent configurations were evaluated
  • GPT-6.1 Sol with Codex CLI scored highest, averaging 56.1, versus 44.4 for the Human-SOTA baseline
  • Starting from a human-built system on seven tasks, the top agent gained 16.6 points on average

Why it matters

Many tools that let you chat with your own files rely on RAG, and building one well takes skilled engineering time. If agents can do much of that work, such tools could get cheaper and faster to set up. Treat it as an early research result, since it is a preprint and tests only nine tasks.

What the researchers built

As arxiv.org reported, a new paper introduces AutoRAGBench, a benchmark, meaning a standard test, for AI coding agents. It asks a practical question: can an agent build a retrieval-augmented generation system by itself? RAG is the common setup in which an AI model looks up information in an outside collection of documents before it answers.

The authors call it the first benchmark to judge agents on building RAG systems from scratch in an open code space, meaning the agent can write whatever code it likes.

How the test works

The benchmark has nine development tasks. For each one, the agent gets no starter code and no recommended strategy. It must study the task data, write a RAG pipeline, and keep improving the design based on the results of its own experiments.

Each run is capped at six hours on a single H800 GPU, a high-end chip used for AI work. The team tested nine agent configurations and looked at both the final systems and the path each agent took to get there.

What the scores show

The best performer was GPT-6.1 Sol running with Codex CLI, a command-line coding tool. It averaged 56.1. The comparison point, a composite Human-SOTA baseline, averaged 44.4. SOTA means state of the art, so this baseline stands in for the best human-designed approaches.

The authors also tried a second setup. On seven of the tasks, the same agent began with a system designed by humans and tried to improve it. On average it added 16.6 points.

How to read it

  • This is a preprint, which may not yet have been peer reviewed.
  • Nine tasks is a limited sample of real-world RAG work.
  • The source gives no detail on how the other eight configurations scored.

The authors conclude that agents can automate effective RAG development. That is a fair reading of these results, but it is not proof the approach works on every kind of company data.

What to watch

  • How the other eight agent configurations scored, and how large the gap is
  • Whether the results hold on tasks outside the nine in the benchmark
  • Whether other researchers adopt AutoRAGBench to compare agents

Try it

If a vendor claims its agent can set up document search for you, ask what tasks it was tested on and what it was compared against.

Sources and links

First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news