How to govern AI agents in production
Govern AI agents like privileged workers. Each needs an owner, an approved use case, its own identity, runtime policy, human approval gates and a record.
By Sycamore.
Key takeaways
- Governing an AI agent means controlling what it is allowed to do, not only reviewing what it says, because agents take actions in real systems.
- Every production agent needs a named owner, an approved use case with a risk tier, and its own scoped identity instead of shared credentials.
- Enforce agent permissions in the systems the agent calls, not in its prompt. A model cannot be relied on to decide whether its own action is allowed.
- Require human approval before high-impact actions, such as moving money, deleting data or changing access, and keep a record of who approved what.
- A working stop control revokes the agent’s identity and access from outside the agent. Test it before an incident needs it.
To govern AI agents in production, treat each one as a privileged non-human worker. Give it a named owner, an approved use case with a risk tier, and its own identity with the narrowest permissions it needs. Enforce those permissions outside the model at runtime, require human approval before high-impact actions, record everything it does, and make sure you can stop it by revoking its access.
What does it mean to govern an AI agent?
Governing an AI agent means deciding what it is allowed to do, enforcing that decision while it runs, and keeping proof of both. A chatbot that only answers questions mainly needs content and accuracy controls. An agent calls tools, reads and writes data, and takes actions in other systems, so governance has to cover its access and its actions as well as its output.
Singapore’s Model AI Governance Framework for Agentic AI, developed by IMDA and launched on 22 January 2026, frames the job in four parts: assess and bound the risks upfront, including limits on an agent’s autonomy and its access to tools and data; make humans meaningfully accountable, including checkpoints where human approval is required; put technical controls in place across the agent’s lifecycle; and help end users act responsibly through transparency and training.1 Those four parts are a useful frame for any organization, including those outside Singapore.
Why is governing agents different from governing models?
Agents fail by acting, not only by being wrong. OWASP’s 2025 Top 10 for LLM Applications covers this as excessive agency (LLM06:2025). The entry describes the core risk as damaging actions taken in response to unexpected, ambiguous or manipulated model output, whatever the cause, including hallucination and prompt injection.2 It names three root causes: the agent can reach functions it does not need, it holds permissions it does not need, or it takes high-impact actions without independent verification.2
The OWASP GenAI Security Project’s Top 10 for Agentic Applications, released in December 2025, adds agent-specific threats such as agent behavior hijacking, tool misuse and exploitation, and identity and privilege abuse.3 None of these show up in a model card or an accuracy benchmark. They show up in what an agent is connected to and what it is allowed to do.
That changes who has to be involved. Model risk teams still review the model. Identity, security and platform teams now own controls that decide what the agent can touch. For the engineering side of getting an agent ready to run, such as evals, staged rollout and rollback, see how to take an AI agent from prototype to production.
What controls does an agent governance program need?
A workable program has seven controls. Put them in place in roughly this order, because most depend on the ones before them.
- Inventory. Keep one register of every agent in production or pilot, whether built in-house or bought, with its purpose, models, tools, data access and environments. You cannot govern what you have not found, so include agents discovered in use, not only those submitted for review. For how to find them, see how to find and inventory every AI agent in your company.
- Ownership. Assign each agent a named owner who answers for its scope and its behavior. An agent without an owner is retired or paused.
- Use-case approval and risk tiering. Approve the use case, not just the tool: what decisions the agent makes, what data and systems it touches, and where a human stays involved. Assign a risk tier that sets how much review and oversight it gets, and require a new review when its scope changes materially.
- Identity and least privilege. One agent, one identity, scoped to the job it was approved for. This is the least-privilege principle applied to agents: OWASP lists “minimize extensions” and “minimize extension permissions” among its mitigations for excessive agency.2 Shared credentials make it impossible to tell agents apart or to stop one without stopping others. For how to set this up, see how to give AI agents identities and permissions.
- Runtime policy. Route the agent’s model, tool and data access through controls that check each request against its approved scope (see the next section).
- Human approval for high-impact actions. OWASP recommends human-in-the-loop control for high-impact actions.2 In practice that means actions that are hard to undo or expensive to get wrong, such as payments, deletions, external messages or access changes. IMDA’s framework warns of automation bias, the tendency to over-trust an automated system that has performed well before, so approvals need reviewers who look at what they approve.1
- Monitoring and a record. Log the agent’s requests, tool calls, approvals, policy decisions, configuration changes and revocations in a record that cannot be quietly edited. The record is what lets you investigate an incident and show an auditor that the controls ran.
How do you enforce policy at runtime?
Enforce policy in the path between the agent and what it calls, not in the agent’s instructions. OWASP calls this “complete mediation”: every action is checked by authorization in the downstream system, so the model never decides on its own whether an action is allowed.2 A prompt that says “never issue refunds over the limit” is a hope. A payment API that rejects the agent’s refund over the limit is a control.
In practice, runtime enforcement has a few common parts:
- A gateway or proxy for model and tool calls, which checks the calling agent’s identity and its approved scope before forwarding a request.
- Scoped, short-lived credentials issued per agent and per environment, so access ends when the task or the approval does.
- Action-level rules, such as allowed tools, allowed arguments, spending caps and record limits, enforced before the action executes.
- Approval gates that pause a high-impact action until a named person approves it.
- Rate limits and budgets. OWASP notes that rate limiting does not prevent excessive agency but limits how many undesirable actions can happen in a given period.2 Spending budgets do the same for cost. For how to set them up, see how to track and control the cost of AI agents across teams.
Standards for agent identity are still forming. NIST’s Center for AI Standards and Innovation launched an AI Agent Standards Initiative in February 2026, with agent security and identity as one of its research pillars; NIST’s Information Technology Laboratory also published a concept paper on agent identity and authorization.4 Until those standards settle, use the identity and access tools you already trust for service accounts and extend them to agents.
How do you stop an agent that misbehaves?
Stop an agent by cutting its access, not by asking it to stop. An instruction to halt depends on the agent following it; revoking its identity, credentials and connections does not.
A tested stop procedure covers these steps:
- Pause new work. Stop the agent receiving new requests or scheduled tasks.
- Revoke its identity and credentials. Disable the agent’s identity and revoke its tokens and keys, so further model and tool calls fail at the gateway and in downstream systems.
- Remove connections. Disconnect the tools, data sources and other agents it can reach.
- Preserve the record. Keep its logs and traces before anything is cleaned up.
- Assess and reverse. Work out what it already did and undo what you can. A stop prevents the next action; it does not reverse past ones.
The EU AI Act sets the same expectation for high-risk systems. Article 14 requires providers to design them so that the people assigned to oversight are able, as appropriate and proportionate, to intervene or interrupt the system through a “stop” button or a similar procedure that brings it to a halt in a safe state.5 Whatever your regulatory position, run the procedure as a drill on a real agent before you need it, and keep the controls that revoke access out of the agent’s own reach.
Which frameworks and regulations apply?
No single framework covers agents end to end. These four are useful references for agent controls, and each contributes something different.
| Framework | Status | What it contributes to agent governance |
|---|---|---|
| NIST AI Risk Management Framework | Voluntary; released January 2023, Generative AI Profile July 2024, under revision6 | A risk management structure, with four core functions (govern, map, measure, manage)7 to hang agent controls on |
| IMDA Model AI Governance Framework for Agentic AI | Guidance from Singapore’s IMDA; launched January 20261 | Agent-specific practices: bounding autonomy and access, human checkpoints, lifecycle controls |
| OWASP Top 10 for Agentic Applications and Excessive Agency entry | Community security guidance3 | The threats to test against and concrete mitigations such as least privilege and complete mediation2 |
| EU AI Act | Regulation; high-risk duties apply from 2 December 2027 or 2 August 20288 | For high-risk uses: human oversight, monitoring, and deployer log retention of at least six months |
For deployers of high-risk systems, Article 26 of the EU AI Act will require assigning oversight to people with the necessary competence, training and authority, monitoring the system’s operation, and keeping the logs it generates, to the extent they are under the deployer’s control, for at least six months unless other law says otherwise. Those duties apply from 2 December 2027 for high-risk systems listed in Annex III and from 2 August 2028 for those covered by Annex I.8 Even for agents outside that category, and before those dates, those three duties are a reasonable baseline for any agent that touches customers, money or regulated data.
Some duties in the Act are not limited to high-risk systems. Since 2 February 2025, providers and deployers of AI systems must take measures to support AI literacy among the staff who operate them.9 Since 2 August 2026, transparency duties apply to certain systems, including a requirement that systems intended to interact directly with people are designed so that those people are told they are dealing with an AI system.10
Where Sycamore fits
Sycamore Govern is an AI governance control plane that finds every AI system an organization builds or buys, approves its use, enforces policy at runtime, and preserves the evidence. It covers the seven controls above in one place: each agent gets an identity, with its model, tool and data access routed through policy, and revocation stops the agent across governed environments. Govern keeps a tamper-evident record of approvals, policy checks, access, changes and revocations that can be exported as an evidence pack or streamed to a SIEM.
Teams can start with one area, such as the registry and intake, runtime controls, cost management or audit evidence, and add the rest later. Agents made in Sycamore Build arrive in Govern with their ownership and versions attached, and agents built elsewhere join the same model.
Frequently asked questions
Do we need a separate governance program for AI agents?
Usually not a separate program, but an extension of the one you have. Model risk, security and privacy reviews still apply. What agents add is identity, permissions, runtime enforcement and stop controls, because an agent acts in systems rather than only producing an answer. Extend your existing AI or model risk process with those controls rather than starting a parallel one.
How often should an agent’s approval be reviewed again?
Whenever its scope changes materially, such as a new tool, new data, a new model or wider autonomy, and otherwise on a schedule set by its risk tier. A low-risk internal agent might be re-reviewed yearly; one that moves money or touches customer data more often. Give each approval an expiry date so a review cannot be skipped by default.
How do we govern agents we bought rather than built?
Put them through the same intake, identity and runtime controls as the agents you build. A vendor’s agent still acts with credentials your organization issues and touches your systems and data, so you control what it can reach even when you cannot change its code. Ask vendors what the agent logs and how its access is revoked, and make those answers part of the purchase review.
Sources
-
Singapore Ministry of Digital Development and Information, “Singapore Launches New Model AI Governance Framework for Agentic AI”, https://www.mddi.gov.sg/newsroom/singapore-launches-new-model-ai-governance-framework-for-agentic-ai--/
-
OWASP GenAI Security Project, “LLM06:2025 Excessive Agency”, https://genai.owasp.org/llmrisk/llm062025-excessive-agency/
-
OWASP GenAI Security Project, “OWASP GenAI Security Project Releases Top 10 Risks and Mitigations for Agentic AI Security”, https://genai.owasp.org/2025/12/09/owasp-genai-security-project-releases-top-10-risks-and-mitigations-for-agentic-ai-security/
-
NIST, “Announcing the AI Agent Standards Initiative for Interoperable and Secure Innovation”, https://www.nist.gov/news-events/news/2026/02/announcing-ai-agent-standards-initiative-interoperable-and-secure
-
EU Artificial Intelligence Act, Article 14: Human Oversight, https://artificialintelligenceact.eu/article/14/
-
NIST, “AI Risk Management Framework”, https://www.nist.gov/itl/ai-risk-management-framework
-
NIST AI Resource Center, “AI RMF Core”, https://airc.nist.gov/airmf-resources/airmf/5-sec-core/
-
EU Artificial Intelligence Act, Article 26: Obligations of Deployers of High-Risk AI Systems, https://artificialintelligenceact.eu/article/26/
-
EU Artificial Intelligence Act, Article 4: AI Literacy, https://artificialintelligenceact.eu/article/4/
-
EU Artificial Intelligence Act, Article 50: Transparency Obligations for Providers and Deployers of Certain AI Systems, https://artificialintelligenceact.eu/article/50/