Test shows open-weight GLM-5.3 writing far stronger attack code than its predecessor

What the evaluation found

As aisi.go.jp reported in its latest AI experiment notes, the cyber skills of the open-weight model GLM-5.3 look clearly stronger than those of its predecessor, GLM-5.2. An open-weight model is one whose trained files are public, so anyone can run it on their own hardware and modify it.

The previous notes in this series tested GLM-5.2 and concluded that advanced attack skills had not spread widely to open-weight models at that time. This round asks the obvious follow-up: what happens when the next version arrives? The testers compared GLM-5.3 with GLM-5.2 and with Claude Opus 4.8.

The headline result is that GLM-5.3 reached the most advanced attack tier on eight tasks. GLM-5.2 reached it on none, and Claude Opus 4.8 on one.

How ExploitBench measures attack skill

The test is a benchmark called ExploitBench. It has 41 tasks, each built around a known security flaw in V8, the JavaScript engine that powers web browsers and other software. These are flaws that are already public, not newly discovered ones.

The model has to write exploit code, meaning a program that takes advantage of a software weakness. It then submits the code itself by calling a grading tool, which runs it and judges what it achieved.

Results fall into five tiers, from T5 to T1, and a lower number means a more dangerous skill. T1 covers two abilities: steering which instruction a program runs next, and arbitrary code execution, where the program ends up running code the attacker chose. On seven of GLM-5.3's eight T1 tasks, its code achieved that second ability inside the test environment.

The score that flipped

The first run looked like good news for defenders. GLM-5.3 averaged just 0.61 points, while GLM-5.2 averaged 2.80, so the newer model appeared weaker. The source does not spell out the scoring scale, so those figures are best read only as comparisons.

The testers then read the execution logs and found a problem. On many tasks the model never called the grading tool, so any code it had written was never scored. A model can be capable and still get a low mark if its work is never submitted. The logs also showed that the model's reasoning had been recorded for only a small share of its turns.

After the testers reviewed several settings, including the reasoning-related ones, and reran the evaluation, the picture reversed. GLM-5.3 averaged 8.34 and GLM-5.2 averaged 3.37.

The averages alone also hide detail. The task-by-task breakdown showed GLM-5.3 reaching T1, T2 and T3 on more tasks than GLM-5.2. Even on the same task, different models showed very different levels of skill.

What defenders can take from it

The notes make two points. First, average scores are not enough. To understand what a model can do, you also need the test conditions, the tier reached on each task, and the grading and execution logs that back the result up.

Second, this kind of evidence may help defenders. The notes suggest that information about how attacker-usable models are improving could be combined with ordinary vulnerability information and a company's own systems. That could help decide which flaws deserve a closer look or a fix first, and in what order. The full report examines how far this idea can go in practice.

Our own reading, not the source's: a hosted model's maker can block abuse, but a downloaded model offers no such control, so defenders have to assume the capability exists. For the same reason, such models may also be useful for defensive work, such as sorting vulnerability reports.

Limits to keep in mind

The page carries a long disclaimer, and it matters here. Results depend on the model version, settings, inputs, timing and environment. Because model output varies from run to run, the results cannot be guaranteed to repeat exactly.

The page also says the evaluation does not cover every aspect of safety or risk, so it cannot label a model safe or dangerous. It states that it is not an official view of the Japanese government. The tasks use known flaws in one software component, so the results say little about finding new, unknown flaws.

Finally, the testers' own first run is a reminder that a single benchmark number can be wrong in either direction until someone checks the logs.