Researchers test whether AI agent skills actually do what they say
A new paper compares what agent add-ons claim to do with what they actually do, and finds a gap in 80.0% of 49,943 skills checked.
The short version
- Skills are third-party add-ons that can touch files, credentials, networks and the shell.
- The method compares declared capabilities with actual ones, using code analysis plus an LLM.
- 80.0% of 49,943 OpenClaw skills had at least one deviation from their description.
- 81.1% of deviations traced to developer oversight; 18.9% to adversarial intent.
- On a 906-skill malicious benchmark, the method scored an F1 of 0.949.
Why it matters
If you install add-on skills for an AI agent, the description you read may not match what the code can do. This work suggests the mismatch is common, and mostly accidental, but it still widens the access you are handing over. Automated auditing of skills before you trust them looks feasible at registry scale.
The gap nobody was checking
Agent skills are add-ons that give an AI agent new powers. According to the paper, as listed on arxiv.org, these can include reading files, using credentials, making network calls and running shell commands.
The authors say most safety work so far looks at malicious prompts or risky actions while an agent is running. The skill itself, the package you install, goes unverified. That is the hole this paper targets.
How the check works
The researchers call the problem behavioral integrity verification. In plain terms, it asks: does the skill do only what it says it does? The method lists the capabilities a skill declares, lists the ones it actually has, and compares the two sets using a shared vocabulary that covers code, instructions and metadata.
To build the lists, it pairs fixed, rule-based code analysis with a language model that helps pull out capabilities. The output is structured evidence that feeds three uses: sorting types of mismatch, working out why they happened, and flagging malicious skills.
What they found in the wild
The team ran the method on 49,943 skills from the OpenClaw registry. 80.0% had at least one deviation, and four categories of combined threat showed up.
Most of the mismatches were not sinister. The root-cause step traced 81.1% to developer oversight and 18.9% to adversarial intent. Still, 5.0% of skills carried predicted multi-stage attack chains, meaning sequences of steps that could add up to an attack.
How well it catches bad skills
On a benchmark of 906 skills built to test malicious-skill detection, the method reached an F1 of 0.949. F1 is a score from 0 to 1 that balances catching bad items against falsely flagging good ones. The authors say this beat the rule-based and single-pass LLM baselines they compared against.
These are the authors' own results from an abstract, not an independent test, so treat them as promising rather than settled.
What to watch
- Will registries or agent platforms adopt automated declared-versus-actual checks before listing skills?
- Do independent teams reproduce the 0.949 F1 result on other skill collections?
Try it
Before installing a skill, compare its stated description with the permissions it requests, and try it first in a throwaway folder or account with no real credentials.
Sources and links
- Reportingarxiv.org: the article we sourced
First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.