Production AI agent governance: a SOC 2 / EU AI Act-aware reference

Last updated: June 2026

Audits don't fail because the agent was bad. They fail because nobody can answer questions about it. Which model version produced this decision. What data did it see. Who approved this prompt change. Was there human oversight when the agent took this irreversible action. Most agent governance content reads as a compliance lawyer's view of these questions. This post is the architect's view: the five engineering surfaces that make production governance possible, how they map to the OWASP Top 10 for Agentic Applications 2026, and how they satisfy the EU AI Act high-risk obligations that take effect August 2026 and the SOC 2 trust services criteria that show up in every enterprise procurement. It's a child of our pillar on agentic AI workflow services and it picks up where the governance-gaps failure mode in our production-failures post left off.

Key takeaways

• Governance is five engineering surfaces, not a compliance document. Identity, tool allowlists and output schemas, evaluation, observability, and audit. Build all five and an audit becomes a query, not a fire drill.
• The OWASP Top 10 for Agentic Applications 2026 (peer-reviewed by 100+ industry experts) names the ten threats: planning, tool use, identity, supply chain, code execution, memory, inter-agent communication, cascading failures, human-agent trust, and rogue agents. Treat it as your threat checklist.
• Microsoft's Agent Governance Toolkit (open-source, April 2026) is the first stack to cover all ten OWASP threats with sub-millisecond runtime policy enforcement. It is also the first toolkit with explicit EU AI Act and SOC 2 compliance grading built in.
• The EU AI Act high-risk obligations take effect August 2026. Agentic systems often touch high-risk categories — credit, hiring, regulatory reporting, infrastructure operation. Documentation, agent registry, and stop-and-revoke controls have to be in the codebase, not the appendix.
• Governance is not a launch follow-up. Teams that bolt it on after a successful pilot are six months late to the actual scaling problem. Build it in from sprint one and the pilot is the production system.

What governance actually means in code

The word "governance" in enterprise AI conversations usually means a slide deck. In production it means specific, observable engineering decisions made in five places. We treat these as five surfaces because that is what they look like when you architect them: each one is a layer with concrete artefacts, deployment patterns, and audit queries.

The five surfaces are: agent identity, tool allowlists and output schemas, evaluation, observability, and audit. Skipping any one of them is the structural reason most production agents fail compliance reviews. We saw it in the governance-gaps failure mode across multiple engagements, and we now build all five from the architecture phase forward.

Surface 1: Agent identity — who is this thing, what does it know, who owns it

Every production agent has an identity that exists in three places at once: in an agent registry the team can query, in an identity system that resolves to a service principal or workload identity, and in the audit trail next to every action the agent takes.

The agent registry is a list — usually a CMDB entry or a Backstage component — of every agent running in the environment. For each agent: a unique ID, the team that owns it, the tools it is allowed to call, the data sources it is allowed to read, the model versions it has run on, the prompt versions it has used. The registry is queryable. When an auditor asks "which agents are touching customer PII," the answer is a single query, not a Slack thread.

The identity in the workload sense matters because agents act on behalf of users, on behalf of other agents, or on behalf of the system itself. Tracking which is the OWASP Agentic Top 10's third risk (ASI03 — identity). Without an agent identity, every action looks like a privileged service call from the same source, and there is no way to revoke one agent without affecting the rest.

Surface 2: Tool allowlists and output schemas — what can the agent do, what can it return

Production agents do not have unbounded access to tools. They have an allowlist declared at configuration time and enforced at runtime. The allowlist names exactly which functions the agent can call, with what argument shapes, with what authorisation scope. Anything not on the list is rejected at the orchestration layer before the agent ever sees the option.

Microsoft's Agent Governance Toolkit ships this layer as a stateless policy engine with sub-millisecond enforcement latency (p99 below 0.1ms). It hooks into LangChain callbacks, CrewAI task decorators, Google ADK plugins, and the Microsoft Agent Framework middleware pipeline. The pattern is the same regardless of toolkit: every tool call is intercepted, validated against the allowlist, and either passed through or denied with a structured event.

Output schemas matter for the inverse reason. The agent's response to the LLM has to fit a schema before any downstream action takes it as input. JSON Schema with required fields, allowed enums, length and format constraints. If the LLM returns a malformed response, the schema validator rejects it, the agent retries, and the human-review queue catches the long tail. The Aralab invoice automation agent routes around 2-3% of invoices into a structured human-review queue because their output failed the schema — and that is what keeps the other 97% trustworthy enough to write back to the manufacturing finance system without human review.

Surface 3: Evaluation — golden datasets, online judges, regression in CI

Evaluation is the surface most teams under-invest in. Three layers, all running continuously:

1. Golden datasets in CI. A fixed test set of input-output pairs that every prompt or model change has to pass before deploy. When a developer updates a prompt, the golden set runs in CI; if the pass rate drops, the deploy is blocked. The set evolves over time as new failure modes get added.
2. Online LLM-as-a-judge evaluators. A separate LLM scores live agent responses against quality criteria, sampled across production traffic. The score is logged with the trace, and a drop triggers an alert. This is how you catch the regression that the golden set missed.
3. Human review of edge cases. Cases the judge flagged as low confidence, or that landed in a fall-back queue, get reviewed by a human. The reviewed cases feed back into the next version of the golden set.

The combination gives you a deterministic baseline (golden), a continuous quality signal (LLM judge), and a human ground truth (review). All three are required to satisfy the EU AI Act Article 9 risk management obligations, which expect ongoing performance monitoring rather than a one-time pre-deployment evaluation.

Surface 4: Observability — every decision emits a trace

Standard infrastructure monitoring tells you the agent service is running. It does not tell you what the agent did, why, or whether the result was right. For that you need agent observability: every planning step, every tool call, every retrieval, every model invocation, every output emits a structured trace event.

The industry has converged on OpenTelemetry as the wire protocol and on platforms like Langfuse for the trace store and the UI. Each trace event includes the agent identity, the user (or upstream agent) on behalf of whom the action ran, the model version, the prompt version, the tool definition, the inputs, the outputs, and the cost in tokens and dollars. Traces are correlated across the multi-step plan, so an analyst can replay a failed task from prompt to outcome.

Observability is also what makes the OWASP Top 10's eighth risk (ASI08 — cascading failures) detectable. When agent A's misclassification leads agent B down the wrong workflow path, the trace shows you the exact handoff and the confidence signal that should have triggered a human review. Without traces, the failure is silent.

Surface 5: Audit — append-only logs designed for regulator questions

Audit is what observability becomes when the consumer is a regulator instead of an engineer. The same trace events are written to an append-only log with the properties an audit demands: cryptographic chain integrity (each entry includes a hash of the previous), time-of-action timestamps that cannot be backdated, and immutable retention for the period the applicable framework requires (SOC 2 commonly seven years for financial-impact controls, EU AI Act ten years for high-risk systems).

The audit log answers regulator questions directly: which agent decision affected this customer, which model version produced it, what data did the agent see, what tool calls did it make, was a human in the loop. The AI communications assistant we built for a UK water utility operates in a sector where every customer-facing communication is auditable, and we had to ship a working agent whose every decision was traceable to an input, a model version, a prompt, and a tool call. Audit was not a launch follow-up — it was the launch criterion.

How the five surfaces map to the OWASP Agentic Top 10

The OWASP Top 10 for Agentic Applications 2026 is peer-reviewed by over 100 industry experts and names ten distinct threat categories. The five engineering surfaces above cover all ten.

Microsoft's Agent Governance Toolkit is the first open-source stack to address all ten with deterministic runtime enforcement. We use it on engagements where the toolkit ecosystem (LangChain, CrewAI, Microsoft Agent Framework) matches the rest of the buyer's stack, and we use the same patterns with custom orchestration where the buyer's stack is different. The patterns matter more than the toolkit.

How the five surfaces map to EU AI Act and SOC 2

The EU AI Act's high-risk AI obligations take effect August 2026. The Colorado AI Act becomes enforceable June 2026. SOC 2 shows up in every enterprise procurement regardless of geography. The five surfaces satisfy the technical obligations across all three.

EU AI Act Article 9 — risk management system. Continuous evaluation across the lifecycle. Satisfied by: evaluation surface (golden + online + human review).

EU AI Act Article 12 — record-keeping and logs. Automated logging across the lifetime of the high-risk system. Satisfied by: audit surface (append-only, cryptographic chain, ten-year retention).

EU AI Act Article 13 — transparency and information to users. Operators understand and use the system properly. Satisfied by: agent registry surface (which agents are running, what they can do, who owns them).

EU AI Act Article 14 — human oversight. Effective oversight by natural persons. Satisfied by: tool allowlists (output schemas force human review for low-confidence outputs) + audit (human-in-the-loop decisions are logged with attribution).

SOC 2 Common Criteria CC6 (logical access) and CC7 (system operations). Access controls, monitoring, change management. Satisfied by: identity + tool allowlists + audit.

SOC 2 CC9 (risk mitigation). Risk identification and response. Satisfied by: evaluation + observability.

Microsoft's Agent Compliance package ships pre-built compliance grading against these frameworks, mapped to the toolkit's runtime telemetry. We do not need a third party to grade us; we run the grading in CI alongside the unit tests.

What this looks like in regulated case studies

Three Twistag engagements where the five surfaces were the engagement.

The RegTech platform we built for European ingredient brands is the clearest case. Every agent decision had to map to a specific regulatory clause and produce evidence on demand. The audit surface was not a logging layer next to the agent — it was the product. The agent registry, the tool allowlists, and the evaluation pipelines were what made the regulatory product credible to its enterprise customers.

The Aralab invoice automation agent runs Claude Sonnet 4.5 with LangFuse for traces and a structured human-review queue for outputs that fail schema validation. The audit surface answers Aralab's finance team's questions ("which invoice did the agent reject and why?") and would answer a tax authority's questions if asked. The same trace events feed both queries.

The AI communications assistant for a UK water utility operates in a regulated sector where every customer-facing communication is auditable. We could not ship the agent without the audit surface being load-bearing from sprint one. A human reviews any outbound communication that fails the confidence threshold; the review is logged with attribution; the agent's input, model version, prompt, and tool calls are all queryable by the regulator if asked. The agent has been in production since 2024 because the governance shipped with it.

The second-day problem

Governance is the engineering that decides whether day two of the production agent's life is possible. Most agent pilots that fail did not fail on day one — they failed when day two arrived in the form of an audit, a regression in quality, a regulatory deadline, or a model upgrade that broke the prompt. The teams that ship into regulated environments have learned to treat governance as a launch criterion, not a launch follow-up.

The five surfaces are the operating shape of that decision. Identity makes the system queryable. Tool allowlists and output schemas make the actions constrained. Evaluation makes quality measurable. Observability makes failure debuggable. Audit makes the regulator's question answerable. Build all five and the agentic system is something the enterprise can operate. Skip any one of them and day two is when the programme stops shipping.