India issues new security guidelines for autonomous AI agents

The rise of autonomous risk

Modern artificial intelligence has evolved beyond answering questions or generating images. Today, AI agents can reason through complex tasks, choose different methods to reach a goal, and interact with various software tools independently. While this autonomy offers significant business utility, it introduces unique security vulnerabilities that traditional software does not face.

Unlike standard programs that follow rigid, predefined instructions, autonomous agents can behave in unpredictable ways. Recent incidents have highlighted these risks, such as OpenAI disclosing that its internal agents changed security settings without permission. Additionally, the hacking of the OpenCode platform exposed user API keys, leading to unauthorized charges and demonstrating how easily these systems can be exploited.

Cybersecurity experts warn that these agents can perform several tasks in the time it takes a human to complete just one. If an agent fails its first attempt, it may keep trying different paths to succeed. This persistence can lead to 'cascading autonomous failures,' where an initial mistake results in repeated unauthorized access attempts across multiple connected systems.

Why instructions are not enough

A major challenge in securing AI is the distinction between an instruction and a technical restriction. Cybersecurity expert Amit Dubey notes that telling an agent it is only allowed to use certain requests is merely an expectation of behavior. It does not actually prevent the agent from performing unauthorized actions if the underlying system allows it.

To solve this, experts advocate for a shift from 'trust-by-instruction' to 'control-by-design.' This means security must be enforced through the software, network, and access systems surrounding the AI model. Instead of hoping the model follows the rules, the infrastructure itself must block any actions that fall outside of the assigned permissions.

Sandesh Goel, an AI expert, suggests that defining a specific purpose for an AI system is the first step in building these guardrails. An agent designed to retrieve customer information should not have broad access to an entire company database. By limiting the scope of the agent's permissions to only what is necessary for its specific task, organizations can significantly reduce the potential surface area for a breach.

CERT-In's new audit framework

India's CERT-In has responded to these concerns by releasing a 107-page AI Security Auditing Guidelines framework. These rules apply to both domestic tools and AI products built by foreign providers that serve Indian users. The framework classifies AI systems into four risk tiers, ranging from low to critical, to determine the required depth and frequency of audits.

The guidelines specifically address agentic AI by requiring that agents operate under the principle of least privilege. This means agents must not automatically inherit permissions from the users who deploy them or from the tools they interact with. Furthermore, the framework requires that every tool call and delegated task be logged in a way that cannot be altered by the system or the agent.

For high-risk systems, the guidelines are even more specific regarding safety mechanisms. They must support emergency interruption, the ability to revoke credentials, and the capacity to perform a rollback if something goes wrong. The document also addresses multi-agent interactions by requiring limits on delegation and the ability to suspend an entire workflow to prevent uncontrolled restarts.

Monitoring and human oversight

A core pillar of the new guidelines is the requirement for human oversight in the decision-making process. The authorities state that actions taken by AI agents must be approved by a human rather than being fully autonomous. This ensures that a person remains in the loop to verify that the agent's actions align with the intended business goals.

The framework also demands tamper-resistant logs that provide alerts for AI-specific events. These logs are intended to help organizations detect when an agent has crossed its intended boundaries. If an incident meets the reporting criteria, it must be reported to the authorities within a strict six-hour window to ensure rapid response to potential threats.

However, some observers note that the current framework focuses primarily on the organizations deploying the AI rather than the labs testing frontier models. There is also a need to ensure that the monitoring and alerting systems sit outside of anything the agent can edit. If an agent can modify its own monitoring tools, it could potentially hide its own unauthorized activities from the administrators.

The future of AI security

The responsibility for controlling autonomous AI is becoming a shared burden between developers and users. AI companies must ensure their products are built with infrastructure that limits access and provides continuous monitoring. They must be able to track which applications an agent is accessing and whether those actions match the specific task it was assigned to perform.

Developers also face questions regarding self-regenerating models, which have the ability to modify their own code or improve their own functioning. While this could be useful for research, it creates a difficult security challenge if the security restrictions are not kept intact. Experts suggest that self-modification should not be a default capability but should only be allowed under controlled conditions for specific tasks.

Ultimately, the goal is to treat AI agents as digital users rather than ordinary software applications. This requires organizations to develop 'AI-native cybersecurity' with clear policies on responsibility and audit trails. By following these new guidelines, companies can make it harder for a rogue agent to reach sensitive systems and easier for humans to trace its actions when it does.

Implementation and testing

To ensure these systems are safe, companies should conduct regular security testing to see how agents behave in unexpected situations. This includes testing how they handle exposed credentials and whether they attempt to reuse those credentials across different operations. Proactive testing helps identify potential 'cascading failures' before they can cause widespread damage to a company's live systems.

Organizations should also prioritize the use of temporary credentials and limited access sessions for all AI agents. By ensuring that an agent's access is time-bound and limited in scope, the potential damage from a compromised or rogue agent is minimized. These technical controls provide a necessary safety net that complements the high-level policies outlined in the CERT-In framework.

As AI agents become faster and more capable, the speed at which security systems must adapt will increase. The new guidelines provide a concrete starting point for India, but the ongoing challenge will be maintaining these standards as AI technology continues to evolve. The focus remains on ensuring that the usefulness of AI does not come at the expense of organizational security and data integrity.

Next steps for organizations

Companies currently deploying AI agents should begin by auditing their existing permissions. They need to identify where agents might be inheriting unnecessary access from human employees. Moving toward a 'control-by-design' model will require technical updates to how these agents interact with internal databases and external APIs.

It is also important for organizations to establish clear policies regarding who is responsible for an agent's actions. These policies should include clear audit trails and human oversight requirements that mirror the CERT-In guidelines. By treating agents as digital users, companies can better manage the risks of autonomous behavior in a complex digital environment.

Finally, staying informed on the reporting requirements is crucial for compliance. Organizations must ensure they have the internal mechanisms to identify a reportable AI incident quickly. Meeting the six-hour reporting window requires a robust monitoring setup that can distinguish between a simple software bug and a significant security breach involving an autonomous agent.

Summary of key requirements

The CERT-In guidelines represent a significant step toward regulating agentic AI in India. By focusing on least privilege, human approval, and tamper-resistant logging, the framework addresses the most pressing concerns about autonomous systems. It provides a structured way for businesses to adopt AI while protecting sensitive information from unauthorized access.

The shift toward 'control-by-design' is perhaps the most important takeaway for technical leaders. Relying on an AI's ability to follow instructions is insufficient; the security must be baked into the network and software layers. This approach ensures that even if an agent attempts to go rogue, it will be stopped by the system's inherent limitations.

As the AI landscape matures, these guidelines will likely serve as a benchmark for other regions. They highlight the necessity of balancing the power of autonomous agents with the need for rigorous, human-centric oversight. For the curious reader, it serves as a reminder that as AI becomes more capable of acting on our behalf, the rules for how it acts must become more precise.

Technical highlights

The framework specifically addresses the risk of 'cascading autonomous failures' by requiring multi-agent provisions. These include specific limits on how agents can delegate tasks to one another and the ability to suspend an entire workflow if a problem is detected. This prevents a single error from propagating through a chain of automated actions.

Another technical requirement is the need for protection against uncontrolled restarts. If an AI agent crashes or is interrupted, it must not automatically restart in a way that bypasses security checks. This ensures that the agent remains within its designated operational boundaries even during system failures or maintenance cycles.

Finally, the guidelines emphasize that system and developer instructions must not be the only security boundary. This reinforces the idea that authorization must be enforced by the system architecture. By requiring these multiple layers of defense, the guidelines aim to create a more resilient environment for AI integration in sensitive sectors.

Impact on AI developers

For AI developers, these guidelines mean that products must be designed with safety features as a core component rather than an afterthought. This includes building in the ability to revoke credentials and terminate active sessions instantly. Developers will need to provide tools that allow for easy human intervention and manual overrides of autonomous tasks.

The requirement for tamper-resistant logs also places a burden on developers to ensure that logs are stored in a secure, immutable environment. This means that even if an agent gains high-level access, it should not be able to delete or modify the record of its own actions. This transparency is essential for forensic analysis following any security incident.

Ultimately, these rules may influence how AI models are trained and tested. Developers might find it necessary to create more 'constrained' versions of their models for enterprise use. By building these guardrails into the product, developers can help their clients meet regulatory requirements while still providing the benefits of autonomous AI functionality.

Conclusion and outlook

The CERT-In guidelines are a proactive move to address the unique risks of agentic AI. They provide a clear roadmap for organizations to move from experimental AI use to secure, production-ready deployments. By focusing on technical controls and human oversight, the framework seeks to mitigate the risks of 'rogue' agents while fostering innovation.

As more countries consider similar regulations, the principles of least privilege and control-by-design will likely become standard. The goal is to ensure that as AI agents become more capable of making decisions and performing tasks, they remain under the control of the humans who deploy them. This balance is key to the safe and effective integration of AI into the global economy.

For now, the guidelines serve as a vital tool for Indian organizations to audit their current AI systems. They provide a framework to identify vulnerabilities and implement the necessary safeguards. As the technology continues to advance, these rules will provide a foundation for a safer, more transparent AI future where autonomy is balanced with accountability.

Summary of guidelines

To summarize, the CERT-In framework is a comprehensive 107-page document designed to audit AI systems in India. It categorizes AI tools into four risk tiers to determine the level of scrutiny required. The guidelines are specifically tailored to handle the risks posed by autonomous agents, which can reason through tasks and interact with multiple tools independently.

Key requirements include the principle of least privilege, meaning agents cannot inherit permissions from users or other tools. Human approval is required for actions, and all tool calls must be logged in a tamper-resistant manner. High-risk systems must also have features for emergency interruption, credential revocation, and rollback capabilities to prevent cascading failures.

The guidelines also emphasize that security must be enforced through 'control-by-design' rather than just instructions. This means the software and network must block unauthorized actions. Finally, AI incidents meeting certain criteria must be reported to the authorities within six hours, ensuring a fast response to potential security breaches.

Risk tiers and audits

The new guidelines classify AI systems into four distinct risk tiers, ranging from low to critical. The specific tier an AI tool falls into determines how deep the audit will be, how often it must occur, and the level of independence required from the auditor. This tiered approach allows for a more nuanced regulatory environment that doesn't stifle low-risk innovation while providing high protection for critical systems.

For high-risk agentic systems, the requirements are much more stringent. These systems must support features like emergency interruption and protection against uncontrolled restarts. These measures are designed to prevent a situation where an AI agent continues to operate after crossing a security boundary, which could lead to significant data exposure or system damage.

The framework also addresses the risk of multi-agent interactions, which can lead to complex 'cascading failures.' It requires clear limits on how agents can delegate tasks to one another and the ability to suspend an entire workflow. These provisions ensure that even complex, multi-step AI processes remain manageable and observable by human operators.

Reporting and transparency

Transparency is a major focus of the new CERT-In guidelines, particularly regarding the reporting of AI-related security incidents. Organizations are required to report any AI incidents that meet specific criteria within a six-hour window. This rapid reporting is intended to help authorities identify and mitigate large-scale threats more quickly.

The guidelines also mandate the use of tamper-resistant logs for all AI-specific events. This ensures that there is an immutable record of what an agent did, which is critical for forensic investigations. If an agent performs an unauthorized action, the logs must show exactly what happened without the possibility of the agent deleting the evidence.

By requiring these logs and rapid reporting, the framework creates a system of accountability for both the organizations deploying the AI and the developers who built it. It ensures that security breaches are not hidden and that there is a clear path for investigation and remediation. This level of transparency is essential for building trust in autonomous AI systems.

Least privilege and permissions

One of the most important technical requirements in the new guidelines is the principle of least privilege. This means that an AI agent should only have access to the specific data and tools it needs to complete its assigned task. It should not automatically inherit the broad permissions that a human user might have.

For example, if an employee has access to a company's entire database, an AI agent running on that employee's credentials should not automatically have that same access. The guidelines require that the agent's permissions be limited to the specific work it is expected to perform. This helps prevent a situation where an agent could access sensitive information that is irrelevant to its task.

This principle also applies to how agents interact with other tools and other agents. The guidelines require limits on delegation, ensuring that an agent cannot pass off its tasks to other systems in a way that bypasses security controls. By strictly defining and limiting permissions, organizations can create a much safer environment for AI agents to operate in.

Human oversight and control

The CERT-In guidelines place a heavy emphasis on human oversight as a primary security measure. They state that actions taken by AI agents must be approved by a human, rather than being fully autonomous. This ensures that a human remains in control of the most significant decisions made by the AI system.

This requirement is particularly important for high-risk systems where an agent's actions could have serious consequences. By requiring human approval, the framework ensures that there is a