A free local dashboard kit lets you compare Claude Code, Codex and Gemini CLI
Agent-Otel-Kit, an open-source project, shows cost, tokens and tool use for three coding agents in one place, running on your own machine.
The short version
- One docker compose command starts the local stack; a small command-line tool points each agent at it.
- Dashboards cover cost, tokens, sessions, tool calls and traces, in both Grafana and Perses.
- Cost data comes from Claude Code only; the other two agents do not report it.
- Prompts, responses, file paths and account emails are stripped by default.
- Gemini CLI with a personal Google login was rejected, so it was tested with an API key.
Why it matters
If you pay for several coding assistants, you rarely get a shared view of what each one uses. This kit offers a way to measure them side by side without sending data to a third party. It also shows how uneven the agents' own reporting still is, which affects any cost comparison you try to make.
What the kit does
Agent-Otel-Kit is a bundle of free tools that collects usage data from three command-line coding assistants: Claude Code, Codex and Gemini CLI. According to the project's page on github.com, one docker compose up command starts the whole stack on your own computer. Docker is software that runs prepackaged services in isolated containers.
The data format is OpenTelemetry, often shortened to OTel, an open standard for reporting how software behaves. The agents already know how to export it. The kit supplies the places to receive, store and display it: a collector, Prometheus for numbers over time, Loki for event logs, Tempo for traces, and dashboards in Grafana and in Perses. A trace is a step-by-step record of what happened during one request.
A small command-line tool written in Go does the wiring. Its setup command edits each agent's settings so they send data to the collector, and doctor checks that data is arriving.
What you can see on the dashboards
There are three dashboards. The first covers cost and tokens (the small chunks of text that AI models count and bill by): totals, breakdowns by model, and the most expensive sessions. The second covers tool calls, meaning each time an agent runs a command or edits a file. It shows counts, failures, success rate, how long calls take, and who approved each call. The third covers sessions and traces, including a waterfall view of a single interaction.
The kit renames labels so the three agents line up under one field, agent_name. That is what makes side-by-side comparison possible. Both dashboard versions are generated from one shared panel list, so Grafana and Perses stay identical.
The project lists the versions it saw: Claude Code 2.1.325, Codex 0.169.0 and Gemini CLI 0.68.0. In a short Codex run, it saw 74 metric families, which gives a sense of how much raw data an agent can emit.
Privacy: what is and is not kept
Because coding agents touch private code, the data policy matters most. By default the kit does not keep prompt text, model replies, tool inputs and outputs, file paths, shell commands or error messages. The agents already hide most of these unless you opt in, and the collector deletes them a second time in case you switch one on by mistake. A setting lets you keep them if you want.
Account emails are dropped too, and so is the process command line, which for some Gemini CLI runs can contain the prompt. Some identifiers do stay: session IDs, account and organization IDs, an installation ID, host name and operating system.
All ports are bound to the local machine, and the stack sends nothing elsewhere. The setup tool changes only each agent's own user config file, saves the original first, and can undo its change. It refuses to undo if you edited the file afterward, unless you force it.
Where the comparison gets uneven
The agents do not report the same things, and the project is open about it. Cost appears for Claude Code only. Codex counts a shell command that ran but failed with a non-zero exit as a successful tool call. Gemini CLI has no single trace per prompt, and Codex over its default connection type reports no per-request model duration.
Token totals need care as well. Codex and Gemini CLI count some token types inside others, so the kit adds only the parts that do not overlap to avoid double counting. The detailed breakdowns still show every type.
There is also a technical wrinkle. The agents report counters as increases since the last export, while Prometheus stores running totals. The collector converts between the two, so the agents can keep their default settings.
One support gap is stated plainly: Gemini CLI with a personal Google login was rejected by Google for this client. That path was tested with an AI Studio API key instead.
How early it is
Our newsletter flagged this as a new open-source project whose maturity and coverage are still open questions. The README suggests care went into it: tests run the collector against recorded Codex and Gemini CLI data, and continuous testing checks that the generated dashboards match the committed files. It is released under the Apache-2.0 license.
It also has practical quirks. Loki and Tempo can answer with errors for up to a minute after startup. Telemetry settings are read only when an agent starts, so you need a fresh session after setup. Claude Code traces need a beta switch turned on.
The stack has set memory limits: 512 MiB each for the collector, Prometheus, Loki and Tempo, 384 MiB for Grafana and 256 MiB for Perses. That is modest, but you are still running several services next to your editor.
What to watch
- Whether Codex and Gemini CLI start reporting cost, so the dashboards can compare spending across all three.
- Whether the project keeps pace as the agents change their telemetry between versions.
- How the Gemini CLI personal-login problem is resolved.
Try it
Before installing anything, run the setup command with the --dry-run flag for one agent. It prints the settings change without writing it, so you can see what would be altered.
Sources and links
- Reportinggithub.com: the article we sourced
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.
Listen to the AI Wire headlines
Listen to the AI WireListen to our daily AI shows every morning: on your commute, in the shower, wherever you are
Listen to the daily showsFirst reported by github.com · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
