SI Agent News
DevelopingClaudeOpenAIGeminiOpen models

A free local dashboard kit lets you compare Claude Code, Codex and Gemini CLI

Agent-Otel-Kit, an open-source project, shows cost, tokens and tool use for three coding agents in one place, running on your own machine.

74metric families from Codex 0.169.0 in one short run

The short version

Why it matters

If you pay for several coding assistants, you rarely get a shared view of what each one uses. This kit offers a way to measure them side by side without sending data to a third party. It also shows how uneven the agents' own reporting still is, which affects any cost comparison you try to make.

What the kit does

Agent-Otel-Kit is a bundle of free tools that collects usage data from three command-line coding assistants: Claude Code, Codex and Gemini CLI. According to the project's page on github.com, one docker compose up command starts the whole stack on your own computer. Docker is software that runs prepackaged services in isolated containers.

The data format is OpenTelemetry, often shortened to OTel, an open standard for reporting how software behaves. The agents already know how to export it. The kit supplies the places to receive, store and display it: a collector, Prometheus for numbers over time, Loki for event logs, Tempo for traces, and dashboards in Grafana and in Perses. A trace is a step-by-step record of what happened during one request.

A small command-line tool written in Go does the wiring. Its setup command edits each agent's settings so they send data to the collector, and doctor checks that data is arriving.

What you can see on the dashboards

There are three dashboards. The first covers cost and tokens (the small chunks of text that AI models count and bill by): totals, breakdowns by model, and the most expensive sessions. The second covers tool calls, meaning each time an agent runs a command or edits a file. It shows counts, failures, success rate, how long calls take, and who approved each call. The third covers sessions and traces, including a waterfall view of a single interaction.

The kit renames labels so the three agents line up under one field, agent_name. That is what makes side-by-side comparison possible. Both dashboard versions are generated from one shared panel list, so Grafana and Perses stay identical.

The project lists the versions it saw: Claude Code 2.1.325, Codex 0.169.0 and Gemini CLI 0.68.0. In a short Codex run, it saw 74 metric families, which gives a sense of how much raw data an agent can emit.

Privacy: what is and is not kept

Because coding agents touch private code, the data policy matters most. By default the kit does not keep prompt text, model replies, tool inputs and outputs, file paths, shell commands or error messages. The agents already hide most of these unless you opt in, and the collector deletes them a second time in case you switch one on by mistake. A setting lets you keep them if you want.

Account emails are dropped too, and so is the process command line, which for some Gemini CLI runs can contain the prompt. Some identifiers do stay: session IDs, account and organization IDs, an installation ID, host name and operating system.

All ports are bound to the local machine, and the stack sends nothing elsewhere. The setup tool changes only each agent's own user config file, saves the original first, and can undo its change. It refuses to undo if you edited the file afterward, unless you force it.

Where the comparison gets uneven

The agents do not report the same things, and the project is open about it. Cost appears for Claude Code only. Codex counts a shell command that ran but failed with a non-zero exit as a successful tool call. Gemini CLI has no single trace per prompt, and Codex over its default connection type reports no per-request model duration.

Token totals need care as well. Codex and Gemini CLI count some token types inside others, so the kit adds only the parts that do not overlap to avoid double counting. The detailed breakdowns still show every type.

There is also a technical wrinkle. The agents report counters as increases since the last export, while Prometheus stores running totals. The collector converts between the two, so the agents can keep their default settings.

One support gap is stated plainly: Gemini CLI with a personal Google login was rejected by Google for this client. That path was tested with an AI Studio API key instead.

How early it is

Our newsletter flagged this as a new open-source project whose maturity and coverage are still open questions. The README suggests care went into it: tests run the collector against recorded Codex and Gemini CLI data, and continuous testing checks that the generated dashboards match the committed files. It is released under the Apache-2.0 license.

It also has practical quirks. Loki and Tempo can answer with errors for up to a minute after startup. Telemetry settings are read only when an agent starts, so you need a fresh session after setup. Claude Code traces need a beta switch turned on.

The stack has set memory limits: 512 MiB each for the collector, Prometheus, Loki and Tempo, 384 MiB for Grafana and 256 MiB for Perses. That is modest, but you are still running several services next to your editor.

What to watch

Try it

Before installing anything, run the setup command with the --dry-run flag for one agent. It prints the settings change without writing it, so you can see what would be altered.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by github.com · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news