SI Agent News
DevelopingGemini

PostHog explains where small decision models beat chatty LLMs

PostHog says small models that give a yes/no or multiple-choice answer with a probability can handle many jobs an LLM does slowly and at higher cost.

98.7%of summaries skip the LLM call in PostHog's production check

The short version

Why it matters

Many teams send every task to the biggest model by habit. PostHog's experience shows a cheaper step in front can cut cost and delay, as long as you pick the right jobs and test the error rate. The real skill is knowing which tasks are simple enough for a small model and which are not.

A model that answers, not explains

Ask a large language model (LLM, the kind of AI behind chatbots) a simple question and you often get a few hundred words of hedging. As posthog.com reported in its post, PostHog has been using a different kind of tool for simple jobs: a decision model. You ask a structured question and get back a typed answer, such as yes/no, a choice from a list, or a score, along with a probability.

PostHog's example is a question about hedgehogs. Asked whether they are mammals, the model answers yes at 0.99. Asked about their conservation status, it answers Least Concern at 0.77. The lower number is the model admitting it is less sure.

That openness is the selling point. In a well-calibrated model, answers scored 0.8 are right about 80% of the time, and a 0.5 is a coin flip. An LLM tends to sound equally confident whether it is right or wrong.

Use the cheap check as a gate

The first use PostHog describes is a gate in front of an expensive LLM call. The post cites a source, TypeSafe AI, for the claim that decision models can be up to 444.6x cheaper and 193.6x faster than frontier models, meaning the most advanced and costly ones.

PostHog applies this in Replay Vision, its product that watches and summarizes recordings of user sessions. Each summary gets two yes/no questions: does it name a real person, and does it show contact details or an ID? Splitting the check into two narrow questions cut false flags to about a tenth of what one broad question produced.

If either answer scores 0.4 or higher, Google's Gemini rewrites the summary without the personal details, and the result is checked again. Otherwise the summary passes. In production, 98.7% skip the LLM call entirely.

The threshold is a deliberate choice. A false positive only costs an extra LLM call, but a false negative stores personal data as if it were clean. So PostHog set the bar low. For its Slack bot, which now answers questions without being tagged, the opposite applies. Being annoying is the costlier mistake, so a message must pass two checks at 0.85 or higher: is it asking for information, and could PostHog answer it?

Routing, reranking and picking charts

The same bot uses a decision model as a router. If a user does not request a specific model, one multiple-choice question picks from a list of options, each with a short note on what it is good for. The instruction is to choose the cheapest option likely to finish the job correctly on the first try, since a failed run costs more than a smarter model.

A third use is reranking search results. A decision model cannot rank ten sessions against a query in one go, because that is too complicated. So PostHog first uses embeddings (numeric fingerprints of text that measure similarity) to build a shortlist. The decision model then answers one simple question per result: does this summary match the search? On 20 test queries, ranking quality improved by about 25%.

A fourth use is replacing hand-written rules. PostHog used to pick chart types for SQL query results with fixed rules, such as a date plus a number meaning a line chart. Now a decision model looks at the first three rows of results, and the SQL itself if the organization opts in. For a pageviews-by-browser query, it chose a bar chart split by browser instead of the rules' basic line chart.

Where it goes wrong

PostHog is blunt about the limits. A score of 0.85 still means about 150 wrong answers per 1,000 checks, so you need a plan for both kinds of error.

Thresholds also do not transfer. When PostHog tried merging similar logs in Error Tracking, the right cutoffs changed between v1 and v2 of the same model. The team made thresholds adjustable per project and kept the model in shadow mode, running quietly without acting, while it recalibrated against hand-labeled data. It also built an internal AI gateway so models can be swapped and tested.

Multi-step reasoning was a clear failure. PostHog tried a hourly job that ranked a Highlights feed with two chained questions. Requests ran up to 32 sessions at 12-38k tokens each, which flooded the model server and blocked other products' calls for hours. Another test used Jev to rank findings from an internal code-review agent. The result was an AUC of 0.52, where 0.5 would be no better than chance.

The cause is size. By default Jev takes 64k tokens per request and 32k per question. An opt-in setting raises both to 262k, but its docs warn that it is slower and less accurate.

A rule of thumb for the rest of us

The takeaway is not that LLMs are bad. PostHog's advice is to use a decision model for inputs too simple for a language model yet too fuzzy for a fixed rule such as a regex, a text-matching pattern. Anything open-ended or multi-step stays with an LLM.

PostHog is also working on a middle path. Its in-house AI research team built Jeeves, a hybrid that reasons before deciding when it is unsure. It is now PostHog's default for any decision call that does not name a model, and the post says other developers have similar variants.

For a non-engineer, the lesson is to ask three questions before paying for a big model. Can a plain rule do it? If not, can a narrow yes/no question do it? Only then is it worth a full LLM.

What to watch

Try it

Pick one repeated task you give an AI, rewrite it as a single yes/no question, and check the answers against a small set you have labeled by hand.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by posthog.com · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news