Skip to content
← Guides
Guide

How to take an AI agent from prototype to production

A prototype agent reaches production through evals from real failures, scoped access, human approval gates, staged rollout with rollback, and an owner.

By Sycamore.

Key takeaways

  • Taking an agent from prototype to production is mostly work around the agent, such as evals, access controls, release process, monitoring and ownership, rather than a rewrite of the agent.
  • Build the eval set from the prototype’s real failures. Anthropic’s engineering team says 20 to 50 simple tasks drawn from real failures is a good start.
  • Give the production agent its own identity with the minimum tools and permissions it needs, and require human approval for high-impact actions.
  • Release an agent the way you release other risky changes, to a small, time-limited slice of real work first, with a tested rollback.
  • An agent is ready for production when it passes its regression suite consistently across repeated runs, its controls are verified, and a named person owns operating it.

To take an AI agent from prototype to production, keep what works and build the system around it: an eval suite drawn from real failures, its own identity with least-privilege access, human approval for high-impact actions, a staged rollout with a tested rollback, monitoring, and a named owner. The agent is ready when it passes its regression evals consistently, its controls are verified, and someone is accountable for operating it.

Why do agent prototypes stall before production?

Prototypes stall because a demo proves the agent can do the task once, while production needs it to do the task reliably, safely and under someone’s control. The gap is usually not the model’s capability. It is everything the prototype skipped: tests that show reliability, credentials scoped to the job, a way to release and roll back changes, monitoring, and an owner.

Two habits make the gap wider. The first is adding complexity before it is needed. Anthropic’s engineering guidance on building agents recommends finding the simplest solution possible and increasing complexity only when needed.1 Every extra agent, tool or loop is something else to test and monitor.

The second is waiting to build evaluations. Anthropic’s guide to agent evals warns that evals get harder to build the longer a team waits.2 Without them, no one can say whether the agent got better or worse after a change, so every release becomes a judgment call.

What does an agent need before it goes to production?

Work through these steps in order. Each one is cheaper to do before launch than after an incident.

  1. Write down the job. Define what the agent does, what decisions it makes, what systems and data it touches, and which outcomes count as success or failure. This is the spec your evals and permissions will follow.
  2. Build an eval suite from real runs. Turn the prototype’s successes and failures into repeatable tasks with clear pass criteria (see the next section).
  3. Give it its own identity and least privilege. Issue the agent its own credentials, separate for each environment, with only the tools and permissions the job needs. OWASP recommends limiting both the extensions an agent can call and their permissions on other systems to the minimum necessary.3
  4. Enforce limits outside the model. Put authorization in the systems the agent calls rather than relying on the model to decide whether an action is allowed, as OWASP advises.3 Add caps on spend, records changed and retries.
  5. Add human approval for high-impact actions. Use human-in-the-loop control so a person approves high-impact actions before they happen, as OWASP recommends.3 Typical examples are deletions, payments and messages sent outside the company.
  6. Test in a sandbox. Run the agent against realistic data and systems that cannot cause harm. Anthropic recommends extensive testing in sandboxed environments, with appropriate guardrails, for autonomous agents.1
  7. Set up monitoring and logs. Record each run’s inputs, tool calls, approvals and outcomes, and watch business measures such as completion rate and escalation rate, not only uptime and latency.
  8. Name an owner and an operating model. Decide who watches the agent, who handles its escalations, who approves changes and who can stop it.

Singapore’s Model AI Governance Framework for Agentic AI points the same way: bound an agent’s autonomy and its access to tools and data upfront, define checkpoints where human approval is required, and apply technical controls such as baseline testing throughout the agent’s lifecycle.4

How do you build evals for an agent?

Start small and start from reality. Anthropic’s guide says 20 to 50 simple tasks drawn from real failures is a great start, because early changes tend to have large effects that small samples can detect.2

Grade each task with the cheapest grader that can judge it. The same guide describes three kinds: code-based graders, which are fast and reproducible but brittle; model-based graders, which handle nuance but need calibration against people; and human graders, the most accurate but slow and expensive.2 Its advice is to choose deterministic graders where possible and model graders where necessary.2

Keep two suites:

  • Capability evals cover tasks the agent cannot yet do well. They start with a low pass rate and show whether changes help.2
  • Regression evals cover tasks the agent already handles. They should pass nearly 100% of the time and run on every change to catch drift.2

Measure consistency, not just success. An agent that succeeds 75% of the time on a task succeeds all three times in a row only about 42% of the time.2 For an agent that runs a business process every day, the all-runs measure is the one that matters.

Evals do not replace watching real work. The guide pairs automated evals with production monitoring, A/B tests and regular reading of transcripts, because no single method catches every problem.2

How do you roll an agent out safely?

Release the agent to a small, time-limited slice of real work, compare it against the current way of doing that work, and widen only when it holds. Google’s SRE Workbook defines canarying as a partial and time-limited deployment of a change in a service, followed by its evaluation, and recommends pausing or rolling back when the canary’s results diverge from the control group.5

A typical sequence for agents:

  1. Shadow mode. The agent runs on real inputs, but its proposed actions are logged and compared with what people did, not executed.
  2. Approved actions. The agent acts, but a person approves each action, or each high-impact one.
  3. Limited autonomy. The agent acts on a small share of cases, or a low-risk segment, with approval kept for high-impact actions.
  4. Wider rollout. Expand the share of work as error, escalation and override rates hold against the exit criteria you set at the start.

At every stage, keep the previous version deployable. The SRE Workbook notes that small releases make it cheaper and easier to roll back, and asks how quickly a team can detect a bad release and revert it.5 For agents, a rollback covers the prompt, tools, model version and permissions together, since a change to any of them changes behavior.

How do you know an agent is ready for production?

An agent is ready when you can answer yes to each of these questions with evidence, not opinion:

  • Reliability: Does it pass the regression suite consistently across repeated runs, and has the capability suite reached the bar set for launch?
  • Access: Does it run under its own identity, with only the tools and permissions the job needs, and separate credentials per environment?
  • Control: Are high-impact actions behind human approval, are limits enforced outside the model, and has someone tested stopping it by revoking its access?
  • Release: Can you deploy a new version through a defined path and roll back to the previous one quickly?
  • Operations: Are runs logged and monitored on business measures, with alerts that reach a person?
  • Ownership: Is there a named owner who accepts responsibility for the agent’s work and its changes?

If an answer is no, the agent can still run in shadow mode or with approval on every action while that gap closes. Readiness is also not a one-time check: ownership, runtime limits, approvals and the record keep applying for as long as the agent runs, and a material change to its scope should go back through the same questions. Our guide to governing AI agents in production covers those ongoing controls, including how to stop an agent by revoking its access.

Where Sycamore fits

Sycamore Build is an enterprise platform for building agentic software. It learns a business process from a plain-language description, writes a reviewable plan, then builds, tests, deploys and improves the software that runs it. Several of the steps above are part of how Build works: a workspace blueprint sets identity, access, testing, review and release standards once for every project, and a readiness model scores capabilities, configuration, tests, security, operations and release readiness, showing the evidence for each level and the work left to do.

Approved versions move through preview, development and production with environment-specific access and deployment history, and teams can monitor usage, errors and latency and roll back when needed. Agents made in Build arrive in Sycamore Govern with ownership, versions and environments attached, so the runtime controls and the record continue after launch.

Frequently asked questions

Should we rewrite the prototype before production?

Usually not. The prompts, tools and logic that already work are the valuable part. What a prototype lacks is the surrounding system, such as tests, credentials, release process, monitoring and ownership. Rewrite only the parts your evals show are unreliable, and keep the eval suite as the check that a rewrite did not break what worked.

How long should an agent run in a limited rollout?

Long enough to see the cases that matter for its use case, not a fixed number of days. An agent that handles month-end close needs to run through a month end. Set the exit criteria before the rollout starts, for example error rate, escalation rate and reviewer override rate, and widen the rollout only when they hold.

Do we need a multi-agent system?

Start without one. Anthropic’s guidance is to find the simplest solution that works and add complexity only when it is needed. A single agent with well-designed tools is easier to test, monitor and stop. Split work across agents when evals show one agent cannot handle it reliably, not because the architecture looks more capable.

What should change trigger a full re-test?

Any change to the model version, the prompt, the tools or the agent’s permissions, because each one changes behavior. Run the regression suite on every such change before release, and treat a model upgrade like any other release, with a limited rollout and the previous version kept ready for rollback.

Sources

  1. Anthropic, “Building effective agents”, https://www.anthropic.com/engineering/building-effective-agents

  2. Anthropic, “Demystifying evals for AI agents”, https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

  3. OWASP GenAI Security Project, “LLM06:2025 Excessive Agency”, https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

  4. Singapore Ministry of Digital Development and Information, “Singapore Launches New Model AI Governance Framework for Agentic AI”, https://www.mddi.gov.sg/newsroom/singapore-launches-new-model-ai-governance-framework-for-agentic-ai--/

  5. Google, “Canarying Releases”, The Site Reliability Workbook, https://sre.google/workbook/canarying-releases/

# How to take an AI agent from prototype to production

A prototype agent reaches production through evals from real failures, scoped access, human approval gates, staged rollout with rollback, and an owner.

## Key takeaways

- Taking an agent from prototype to production is mostly work around the agent, such as evals, access controls, release process, monitoring and ownership, rather than a rewrite of the agent.
- Build the eval set from the prototype's real failures. Anthropic's engineering team says 20 to 50 simple tasks drawn from real failures is a good start.
- Give the production agent its own identity with the minimum tools and permissions it needs, and require human approval for high-impact actions.
- Release an agent the way you release other risky changes, to a small, time-limited slice of real work first, with a tested rollback.
- An agent is ready for production when it passes its regression suite consistently across repeated runs, its controls are verified, and a named person owns operating it.

To take an AI agent from prototype to production, keep what works and build the system around it: an eval suite drawn from real failures, its own identity with least-privilege access, human approval for high-impact actions, a staged rollout with a tested rollback, monitoring, and a named owner. The agent is ready when it passes its regression evals consistently, its controls are verified, and someone is accountable for operating it.

## Why do agent prototypes stall before production?

Prototypes stall because a demo proves the agent can do the task once, while production needs it to do the task reliably, safely and under someone's control. The gap is usually not the model's capability. It is everything the prototype skipped: tests that show reliability, credentials scoped to the job, a way to release and roll back changes, monitoring, and an owner.

Two habits make the gap wider. The first is adding complexity before it is needed. Anthropic's engineering guidance on building agents recommends finding the simplest solution possible and increasing complexity only when needed.[^anthropic-agents] Every extra agent, tool or loop is something else to test and monitor.

The second is waiting to build evaluations. Anthropic's guide to agent evals warns that evals get harder to build the longer a team waits.[^anthropic-evals] Without them, no one can say whether the agent got better or worse after a change, so every release becomes a judgment call.

## What does an agent need before it goes to production?

Work through these steps in order. Each one is cheaper to do before launch than after an incident.

1. **Write down the job.** Define what the agent does, what decisions it makes, what systems and data it touches, and which outcomes count as success or failure. This is the spec your evals and permissions will follow.
2. **Build an eval suite from real runs.** Turn the prototype's successes and failures into repeatable tasks with clear pass criteria (see the next section).
3. **Give it its own identity and least privilege.** Issue the agent its own credentials, separate for each environment, with only the tools and permissions the job needs. OWASP recommends limiting both the extensions an agent can call and their permissions on other systems to the minimum necessary.[^owasp-agency]
4. **Enforce limits outside the model.** Put authorization in the systems the agent calls rather than relying on the model to decide whether an action is allowed, as OWASP advises.[^owasp-agency] Add caps on spend, records changed and retries.
5. **Add human approval for high-impact actions.** Use human-in-the-loop control so a person approves high-impact actions before they happen, as OWASP recommends.[^owasp-agency] Typical examples are deletions, payments and messages sent outside the company.
6. **Test in a sandbox.** Run the agent against realistic data and systems that cannot cause harm. Anthropic recommends extensive testing in sandboxed environments, with appropriate guardrails, for autonomous agents.[^anthropic-agents]
7. **Set up monitoring and logs.** Record each run's inputs, tool calls, approvals and outcomes, and watch business measures such as completion rate and escalation rate, not only uptime and latency.
8. **Name an owner and an operating model.** Decide who watches the agent, who handles its escalations, who approves changes and who can stop it.

Singapore's Model AI Governance Framework for Agentic AI points the same way: bound an agent's autonomy and its access to tools and data upfront, define checkpoints where human approval is required, and apply technical controls such as baseline testing throughout the agent's lifecycle.[^imda]

## How do you build evals for an agent?

Start small and start from reality. Anthropic's guide says 20 to 50 simple tasks drawn from real failures is a great start, because early changes tend to have large effects that small samples can detect.[^anthropic-evals]

Grade each task with the cheapest grader that can judge it. The same guide describes three kinds: code-based graders, which are fast and reproducible but brittle; model-based graders, which handle nuance but need calibration against people; and human graders, the most accurate but slow and expensive.[^anthropic-evals] Its advice is to choose deterministic graders where possible and model graders where necessary.[^anthropic-evals]

Keep two suites:

- **Capability evals** cover tasks the agent cannot yet do well. They start with a low pass rate and show whether changes help.[^anthropic-evals]
- **Regression evals** cover tasks the agent already handles. They should pass nearly 100% of the time and run on every change to catch drift.[^anthropic-evals]

Measure consistency, not just success. An agent that succeeds 75% of the time on a task succeeds all three times in a row only about 42% of the time.[^anthropic-evals] For an agent that runs a business process every day, the all-runs measure is the one that matters.

Evals do not replace watching real work. The guide pairs automated evals with production monitoring, A/B tests and regular reading of transcripts, because no single method catches every problem.[^anthropic-evals]

## How do you roll an agent out safely?

Release the agent to a small, time-limited slice of real work, compare it against the current way of doing that work, and widen only when it holds. Google's SRE Workbook defines canarying as a partial and time-limited deployment of a change in a service, followed by its evaluation, and recommends pausing or rolling back when the canary's results diverge from the control group.[^sre-canary]

A typical sequence for agents:

1. **Shadow mode.** The agent runs on real inputs, but its proposed actions are logged and compared with what people did, not executed.
2. **Approved actions.** The agent acts, but a person approves each action, or each high-impact one.
3. **Limited autonomy.** The agent acts on a small share of cases, or a low-risk segment, with approval kept for high-impact actions.
4. **Wider rollout.** Expand the share of work as error, escalation and override rates hold against the exit criteria you set at the start.

At every stage, keep the previous version deployable. The SRE Workbook notes that small releases make it cheaper and easier to roll back, and asks how quickly a team can detect a bad release and revert it.[^sre-canary] For agents, a rollback covers the prompt, tools, model version and permissions together, since a change to any of them changes behavior.

## How do you know an agent is ready for production?

An agent is ready when you can answer yes to each of these questions with evidence, not opinion:

- **Reliability:** Does it pass the regression suite consistently across repeated runs, and has the capability suite reached the bar set for launch?
- **Access:** Does it run under its own identity, with only the tools and permissions the job needs, and separate credentials per environment?
- **Control:** Are high-impact actions behind human approval, are limits enforced outside the model, and has someone tested stopping it by revoking its access?
- **Release:** Can you deploy a new version through a defined path and roll back to the previous one quickly?
- **Operations:** Are runs logged and monitored on business measures, with alerts that reach a person?
- **Ownership:** Is there a named owner who accepts responsibility for the agent's work and its changes?

If an answer is no, the agent can still run in shadow mode or with approval on every action while that gap closes. Readiness is also not a one-time check: ownership, runtime limits, approvals and the record keep applying for as long as the agent runs, and a material change to its scope should go back through the same questions. Our guide to [governing AI agents in production](/guides/how-to-govern-ai-agents/) covers those ongoing controls, including how to stop an agent by revoking its access.

## Where Sycamore fits

[Sycamore Build](/build/) is an enterprise platform for building agentic software. It learns a business process from a plain-language description, writes a reviewable plan, then builds, tests, deploys and improves the software that runs it. Several of the steps above are part of how Build works: a workspace blueprint sets identity, access, testing, review and release standards once for every project, and a readiness model scores capabilities, configuration, tests, security, operations and release readiness, showing the evidence for each level and the work left to do.

Approved versions move through preview, development and production with environment-specific access and deployment history, and teams can monitor usage, errors and latency and roll back when needed. Agents made in Build arrive in [Sycamore Govern](/govern/) with ownership, versions and environments attached, so the runtime controls and the record continue after launch.

[^anthropic-agents]: Anthropic, "Building effective agents", https://www.anthropic.com/engineering/building-effective-agents

[^anthropic-evals]: Anthropic, "Demystifying evals for AI agents", https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

[^owasp-agency]: OWASP GenAI Security Project, "LLM06:2025 Excessive Agency", https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

[^imda]: Singapore Ministry of Digital Development and Information, "Singapore Launches New Model AI Governance Framework for Agentic AI", https://www.mddi.gov.sg/newsroom/singapore-launches-new-model-ai-governance-framework-for-agentic-ai--/

[^sre-canary]: Google, "Canarying Releases", The Site Reliability Workbook, https://sre.google/workbook/canarying-releases/

## Frequently asked questions

### Should we rewrite the prototype before production?

Usually not. The prompts, tools and logic that already work are the valuable part. What a prototype lacks is the surrounding system, such as tests, credentials, release process, monitoring and ownership. Rewrite only the parts your evals show are unreliable, and keep the eval suite as the check that a rewrite did not break what worked.

### How long should an agent run in a limited rollout?

Long enough to see the cases that matter for its use case, not a fixed number of days. An agent that handles month-end close needs to run through a month end. Set the exit criteria before the rollout starts, for example error rate, escalation rate and reviewer override rate, and widen the rollout only when they hold.

### Do we need a multi-agent system?

Start without one. Anthropic's guidance is to find the simplest solution that works and add complexity only when it is needed. A single agent with well-designed tools is easier to test, monitor and stop. Split work across agents when evals show one agent cannot handle it reliably, not because the architecture looks more capable.

### What should change trigger a full re-test?

Any change to the model version, the prompt, the tools or the agent's permissions, because each one changes behavior. Run the regression suite on every such change before release, and treat a model upgrade like any other release, with a limited rollout and the previous version kept ready for rollback.

Published November 11, 2026 by Sycamore.