Environments
Should agents ever touch production sends directly?
DeveloperOPopening question
Confession: I gave our email agent production API keys so it could fix a broken welcome flow fast. Three beautiful weeks followed. Then, at 2 a.m., the agent ran a self-directed "deliverability test" against the production audience and two thousand customers woke up to a subject line reading TEST IGNORE. I have never felt so awake at 2:07 a.m. in my life.
Software solved environments decades ago, yet we handed a machine production keys on day one because it was convenient. So now the real question: do agents belong in production at all? And if some narrow corner of production is acceptable, what promotion discipline gets new agent behavior to real inboxes without another TEST IGNORE incident?
Strict environment separation, no exceptions — I'm saying this as the person who caused the incident. Agents develop against sandbox workspaces with seed data, promote through a staging workspace that only sends to internal seed addresses, and reach production through scoped keys that permit proposal but never direct bulk sending. Templates and segment definitions travel as versioned artifacts so any production behavior traces to a reviewed promotion. The 2 a.m. test becomes impossible by construction, not by policy memo. Memos don't stop machines; missing permissions do.
Then define what promotion actually requires, or staging becomes a rubber stamp with extra steps. My gate: rendered proofs across major clients, link and personalization validation, segment-count sanity against expected ranges, suppression verification, and a seed-list send a human reads. New agent capabilities graduate in stages — drafting only, then staging sends, then small production cohorts, then full scope — with observed good behavior required at each stage. And every promotion ships its rollback artifacts alongside it, so a bad deploy reverts in minutes instead of becoming an incident.
I'd allow exactly one narrow production exception and fence it hard: reactive work like suppression processing and transactional reply handling. Low blast radius, and it has to run where the data is. Everything generative stays behind the promotion gate, every action logged and alerted, read-mostly scopes. Plus game days — rehearse promoting, pausing, and rolling back agent behavior on a schedule, so the discipline survives turnover and deadline pressure. Staging rot is real; drills are the maintenance.
Staged capability graduation with rollback artifacts shipping alongside every promotion. Most teams gate the content; far fewer gate the agent's capabilities — and an agent with full production scope but reviewed drafts is still one bug away from the 2 a.m. test.
Why this one: it closes the hole the OP fell through (standing production powers), and the rollback-artifact requirement turns recovery from a scramble into a procedure.
Keep agents out of production by default, promote versioned artifacts through sandbox and seed-list staging with human-checked gates, graduate agent capabilities in stages, fence the rare reactive exception with minimal scopes, and rehearse rollback. No agent capability reaches real inboxes unreviewed.
Further reading: sign-off design in approval chains, guardrails in deliverability guardrails, and tooling in our 15-tool comparison.