New automated harness tests Claude and GPT-6 writing compliance

Moving beyond gut feelings

Most comparisons between AI models like Claude and ChatGPT rely on a user's subjective preference after a few prompts. While this works for casual use, it fails for professional teams that operate under strict style sheets, word budgets, and editorial oversight.

A style sheet is inherently testable because it contains objective requirements. A draft either meets a specific word count, includes a required date, or avoids a banned character. As shattered.io reported, these rules can be turned into a measurable scoreboard for performance.

The new testing method uses a small Python harness to send the same five briefs to both models. It repeats each brief five times to generate a dataset of 25 drafts per model. This allows teams to see which model consistently follows instructions rather than which one simply sounds the best.

How the rules test works

The harness functions by turning a brief into two distinct components: a plain-language instruction for the AI and a set of programmatic checks for the grader. By using the same rules for both, the test ensures that the model's instructions and the evaluation criteria remain in perfect lockstep.

The Python checker evaluates several specific constraints, such as word ranges, paragraph counts, and the presence of forbidden characters like semicolons or exclamation points. It also checks for banned stock phrases and ensures the reading level stays within a specific grade range. It even analyzes sentence variety to ensure the prose does not become repetitive.

The system produces four specific numbers to help budget owners make decisions. These include the compliant-draft rate, the rule pass rate for partial credit, and length consistency. Most importantly, it calculates the cost per compliant draft, which divides the total token spend by the number of drafts that actually passed every rule.

Who is affected by these results

Content teams and newsrooms are the primary users of this testing method. These groups work under constraints that have nothing to do with creative talent, such as specific client requirements or house style guides. A model that overshoots a word count by 40 words costs an editor time, which becomes a real line item when multiplied by hundreds of drafts.

The harness is designed for anyone with basic technical knowledge who can run a Python environment. It requires an Anthropic API key and an OpenAI API key with billing enabled. It does not require a vendor-specific software development kit, making it accessible to developers who want to build custom audit trails.

Because the test uses fictional products and events, it removes factual accuracy from the equation. This allows teams to isolate the model's ability to follow instructions from its ability to retrieve facts. Once a model passes this compliance filter, the team can then focus on judging the tone and brand voice.

The current model landscape

The test is designed to work with the latest releases from both major providers. On the Anthropic side, it covers the Claude 5.5 family, including the flagship Opus, as well as the Sonnet and Haiku tiers. These models represent the newest generation of Anthropic's offerings.

On the OpenAI side, the test includes the GPT-6 family. This includes the GPT-6.1 Terra model, which was announced in late October 2026. The harness allows users to list the exact model IDs available to their specific accounts to ensure the code runs correctly.

Because vendors do not publish matching tiers, the test allows users to pick their own pairings. The goal is to see how specific models stack up on general tasks. Since vendors ship on a monthly rhythm, the harness is intended to be rerun whenever a new snapshot or model version lands to keep the data current.

What comes next for AI writing

The future of AI writing in professional settings involves treating these models as filters. A model that fails a high percentage of hard constraints needs a better prompt or a retry loop before its creative voice is even considered. This prevents teams from wasting time on outputs that require heavy manual editing.

For teams with specific needs, the next step is to replace the fictional briefs in the harness with real-world tasks. A team that writes product descriptions all day should use their actual house rules to get the most accurate data. This provides a tailored scoreboard for their specific workflow.

Ultimately, the goal is to move toward objective compliance. By building a harness once and rerunning it regularly, teams can let the numbers decide which model is the best fit for their budget and their rules. This data-driven approach removes the noise of subjective taste from the initial selection process.