Enterprise AI agents in regulated industries

This guide covers what agents are, how they work, where they break, and what has to be true before one goes anywhere near a validated process. Most published material on enterprise AI agents is written for software companies deploying to other software companies. Regulated deployment is a different problem, and this guide is written for that problem.

What changes when the industry is regulated

The mechanics of an agent are the same everywhere. The constraints around it are not, and the constraints are what determine whether a deployment ships or dies in review.

The output has to be defensible, not just correct

In most enterprise contexts, an agent that produces a good answer has done its job. In a regulated one, a good answer with no traceable derivation is worthless. This inverts the usual engineering priority: accuracy is necessary but it is not the hard part. The hard part is provenance — a durable, inspectable record of which inputs produced which output, through which steps, under which model version. Systems designed for provenance first look quite different from systems that add logging later.

The data usually cannot move

Patient records, unblinded trial data, pre-submission regulatory content and government health data are frequently subject to residency requirements, contractual restrictions or outright statutory prohibition on cross-border transfer. A great many agent products assume an API call to a hosted model — that assumption alone disqualifies them. What matters is that the deployment topology is not an implementation detail chosen at the end. It constrains which models are available, how the agent is updated, and what "tool use" can even mean when there is no outbound connectivity.

Autonomy is a liability, not a feature

Consumer agent marketing sells autonomy: give it a goal, walk away. In regulated work, unsupervised autonomy in a consequential step is the thing the buyer is specifically trying to avoid. The useful framing is not how autonomous is it but where does it stop. A well-designed regulated agent runs a long sequence of low-consequence steps without interruption, then halts at a defined point and hands to a qualified person. Choosing those handoff points is a clinical, regulatory and legal judgement, not an engineering one.

How an enterprise agent is actually built

Six components recur across serious implementations. A product missing several of them is a chat interface with ambition.

Planning and decomposition

The agent receives a request and breaks it into ordered steps. In a regulated system this plan should be explicit and inspectable before execution, rather than emergent during it. A plan you can read is a plan you can approve, log, and re-run identically. Constraining planning to a registered set of steps costs some flexibility — in this context that is a good trade, and it is the difference between a system that passes qualification and one that does not.

Retrieval over a governed knowledge layer

Agents need the institution's own evidence, not the model's recollection of the internet. That means retrieval over a curated, permissioned, versioned corpus, with the retrieved passages carried forward so the final output can point back at them. The quality of this layer sets the ceiling on everything above it — an excellent orchestration engine over a poorly harmonised document dump produces confident, well-formatted, untraceable output.

Tool use, tightly scoped

Tools are the agent's hands: query a database, call a statistical routine, write to a document, look up a code. Two rules matter. Tools are registered — the agent cannot invent a capability at runtime. And tools are permissioned per agent and per user, so the agent inherits the caller's access rather than holding standing privileges of its own. The second rule is routinely violated, usually by giving the agent a service account with broad read access because it is simpler — that decision is what turns an agent into a data-exfiltration path.

Verification before the human sees it

The step most agent products skip. Before output reaches a person, an independent check runs against it: are the cited references real and do they support the claim; are the numbers internally consistent; does the content stay within approved language. Doing this with a second model class plus deterministic rule checks catches a meaningful share of errors before they consume expert review time, and produces a record that a check occurred.

Human sign-off as a system event

The handoff is not "a person looks at it." It is a recorded event with an identity, a timestamp, a version of the artefact, and a decision. Approval, rejection and modification are all logged. If sign-off happens in email or a document comment thread, the audit trail has a hole in exactly the place an inspector will look.

The audit record

Everything above emits to an immutable log: the request, the plan, retrieved evidence, tool calls, model versions, verification results, the human decision. The test is simple and demanding — can you reconstruct, eighteen months later, exactly how a specific sentence in a specific submitted document came to exist? Most systems cannot. Designing for that test from the start is substantially cheaper than retrofitting it.

Read next

• What are AI agents? — the definition, the four required capabilities, and how agents differ from the automation you already run.
• AI agents vs chatbots — why the difference is a risk category rather than a feature list.
• How AI agents work — the execution loop: planning, retrieval, tool use, routing and verification.
• Enterprise AI agent security — the threat model, including prompt injection through retrieved content and over-privileged tool access.
• AI agent governance — policies, guardrails, approval workflows and the documentation an agent needs before it touches a regulated process.
• AI agent use cases in healthcare — where agents are genuinely working in pharma, hospitals and health authorities, and where the claims run ahead of the evidence.

Where to start

The question that determines whether an agent deployment works is not which model or which vendor. It is which workflow, and where the agent is required to stop. See how this plays out in a real deployment in the pharmacy inventory case study.

Tell us the workflow you're considering, and we'll tell you where the handoff belongs.

Book a working session