AI in policing risks magnifying historical bias and creating feedback loops

The problem of dirty data

Police departments are increasingly using big data to predict where crimes might occur rather than just responding to them. However, researchers warn that the data fed into these systems is often skewed toward public places and specific neighborhoods. Because AI treats these records as objective patterns, the resulting predictions can appear impartial even when they are based on biased human history.

New York University researchers describe this issue as 'dirty data.' This refers to police records that have been shaped by corrupt, biased, or unlawful practices, such as rigged crime figures or the arrest of innocent people. The AI cannot distinguish between a legitimate crime and a record created by a flawed policing practice; it simply treats every entry as a pattern to repeat.

The risk is that the technology acts like a map-maker who only has access to 50-year-old maps. If the map-maker is asked to shade in future crime hotspots based on those old maps, they will only copy the errors of the past. This means the AI is not discovering new crime, but rather following the historical path of where police chose to look before.

Creating a feedback loop

One of the primary concerns for sociologists is the creation of a dangerous feedback loop. When an AI shades a specific area as a hotspot, police are sent there more frequently. Because there are more officers in that location, more crimes are recorded there simply because people are looking for them.

Areas outside of these shaded zones may appear safer because fewer people are monitoring them. When the next set of data is collected, the AI sees the high volume of arrests in the original hotspot and shades it even darker. This cycle ensures that the system continues to prioritize the same neighborhoods, regardless of whether crime rates are actually higher there than elsewhere.

Sarah Brayne, a sociologist at the University of Texas at Austin, noted that these models can contribute to the future occurrence of the events they are trying to predict. This means the technology doesn't just observe the world; it actively shapes the distribution of police presence based on its own previous outputs.

Technical limits and nuances

Not all experts agree that AI is destined to simply copy historical bias. Dr. Dimitrios Kollias of Queen Mary University of London argues that modern systems are more complex than simple retrieval tools. He notes that AI can combine learned information in new ways to generate solutions that were not explicitly present in the original training examples.

While this capability allows for more sophisticated analysis, it also introduces new risks regarding autonomy. As systems become more capable of making novel decisions, Kollias emphasizes the need for meaningful human control. The goal is to ensure that as the technology evolves, its behavior remains safe and reliable for the public.

The challenge remains in the flawed information stored by authorities. Even if the AI is capable of complex reasoning, it is still drawing from a foundation of human-made records. If the person describing the city holds a grudge against a specific neighborhood, that bias ends up on the map, and the AI will treat that grudge as a geographical fact.

Solutions and human oversight

To combat these issues, researchers suggest that AI systems must be subject to rigorous human oversight and independent audits. Dr. Carolyn McKay of the University of Sydney Law School argues that the public functions of state policing should be scrutinized by outside parties. This would involve checking the data used by these forces to find and remove bias before the tools are deployed.

One potential solution is to provide the AI with more context, such as data on broken streetlights or missing public services. By showing the AI which areas have been neglected, it might identify environmental factors rather than just policing patterns. This could shift the focus from sending more police cars to sending funding to repair the community and its infrastructure.

The healthcare sector provides a real-world example of how changing an AI's objective can reduce bias. A 2019 study in the journal Science found that an AI favoring patients based on 'cost' unfairly favored white patients. When researchers changed the goal to predicting 'illness' instead of 'cost,' the share of black patients selected for extra help rose from about 18% to 47%.

The role of human judgment

Ultimately, the problem of AI bias is a reflection of human bias. People are the ones who decide what a suspicious person looks like on CCTV or what constitutes a risky loan. These cultural, educational, and political biases are baked into the data that the AI eventually consumes.

While some fear that AI will make discrimination worse, others believe it can be part of the solution. A carefully designed and independently evaluated system could make these patterns more visible and measurable. By making bias visible, humans have a better opportunity to identify and reduce it systematically.

The consensus among researchers is that humans must stay in the loop. If the AI is allowed to operate without oversight, it risks trapping society in a cycle of yesterday's mistakes. By keeping humans involved in the decision-making process, we can ensure that technology serves to improve justice rather than automate past prejudices.