AI-agent forum simulation matched Voat's scale but got toxicity in the wrong places
Researchers ran AI agents in a Reddit-like forum and compared them to real Voat data. Activity and topics matched, but the toxicity landed in the wrong places.
The short version
- 30 simulated 30-day runs were compared with 30 matched windows of real Voat v/technology data.
- Overall mean toxicity was 0.143 in the simulation against 0.119 on Voat.
- Simulated opening posts averaged 0.116 in toxicity, against 0.0307 for real ones.
- The simulated network was denser: average degree 2.72 versus 2.31 on Voat.
- A neutral-persona test cut mean toxicity by 46.7%, so the mismatch looks tunable.
Why it matters
Companies and researchers increasingly use AI personas to stand in for crowds, such as when testing moderation or predicting reactions. This study shows a simulation can look right on average while getting the thing you care about wrong in specific places. Check a simulation against real data layer by layer before trusting it.
What the researchers tested
As Bioengineer.org reported, a team published a study in EPJ Data Science that puts AI-run online communities to a quantitative test. Earlier work often judged such simulations by whether they seemed believable, or showed a single run. This team instead ran 30 independent simulations of 30 days each and compared them with 30 matched, non-overlapping windows of real posts from the v/technology section of Voat, a Reddit-like forum.
A note on the headline: the real data came from Voat, not Reddit itself. The simulated platform, called Y Social, is built to mimic a Reddit-style forum. It works as a digital twin, meaning a working replica of the real kind of site.
How the simulated forum worked
The agents ran on Dolphin Mistral 24B, a language model without the usual safety tuning that tends to suppress hostile speech. The researchers chose it on purpose so that arguments and toxic talk could appear, and they say this was a realism choice, not a policy statement. Each agent got a persona with traits such as education level, political leaning and a tendency toward toxicity.
The agents were also stateless. They had no long-term memory, and the server simply fed the relevant thread history and user interests into each prompt. The team set the forum's size and activity from a dataset of real Voat data. Real v/technology turned out to be small and volatile, with about 576 unique users per 30-day window and high churn. The simulation started with 50 agents and added and removed users daily to match.
Where the simulation got it right
On scale and rhythm, the agents did well. Counts of root posts (the opening post of a thread), unique users and daily active users overlapped with the real forum's numbers. In both worlds a few people posted a lot and most posted little, though the simulation clustered more tightly around low activity, while Voat had more extreme heavy posters.
Topics matched closely too. 99.6% of the simulation's topics had at least one match in the Voat material. Themes such as privacy, AI and electric vehicles lined up, with similarity scores between 0.69 and 0.82. Simulated posts also resembled real ones moderately well, while real comments were more varied in style than the simulated ones.
Where it went wrong: toxicity and network shape
Overall toxicity looked fairly close: 0.143 in the simulation against 0.119 on Voat. That average hides a split. Simulated opening posts averaged 0.116, while real ones averaged only 0.0307. Replies went the other way, with the simulation producing less of the harsh tone that real comment sections show. The authors say the persona setup and the prompts for opening posts likely pushed agents toward confrontational starts. A shuffling test indicated the pattern came from the gap between posts and comments, not from arguments escalating within a thread.
The social network differed as well. The simulated average number of connections per user was 2.72, versus 2.31 on Voat, and clustering was 4.6 times higher. Only 5.90% of connections were repeat exchanges, compared with 8.70% on Voat. The authors link this to the lack of memory, which makes it hard for the same people to keep talking and form lasting hubs.
Is the toxicity problem fixable?
The team ran a separate 10-day test. A version with neutral personas cut mean toxicity by 46.7% and cut the share of items above a toxicity score of 0.5 by 51.5%. Removing politics from the personas changed little. The authors read this as a sign that the mismatch can be tuned by adjusting the setup, rather than being a built-in flaw of using language models.
There are limits. The test was a small forum with shallow threads, which the authors call a favorable first case for memoryless agents. They expect bigger, more varied communities to make the mismatches worse, especially around hubs and repeat interaction. They say the results show structural resemblance at the right scale, not a copy of any individual's behavior. Next they plan to add light memory within a thread and test other ways of ranking the feed.
What to watch
- Will adding per-thread memory bring the network shape and repeat interactions closer to real forums?
- Do the toxicity mismatches grow in larger, more varied communities, as the authors expect?
Try it
When you see a claim that AI personas "act like real users," ask whether it was checked against real data separately for posts and replies, not just as one overall average.
Sources and links
First reported by Bioengineer.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.