AI guardrails - inline or out of band

Every AI application on our platform talks to models through a single front door, a model gateway. It handles routing, keys, retries and cost controls, so no application holds a raw model credential. The guardrail is the safety layer: it reads a prompt or a model response and decides whether the content is acceptable, things like prompt injection attempts, toxic output, or personal data where it should not be.

The product supports both deployment shapes, so the choice was ours to make. We evaluated the two side by side: deploy the guardrail inline, where all traffic physically passes through it on the way to the model, or deploy it out of band as a separate service the gateway calls. We chose the second. The gateway sends it each request and each response over an API, waits for a verdict, and blocks the traffic if the verdict says so.

So the guardrail is out of band, but it still checks every message. What changed is that the gateway owns the traffic path, and the rest of this post is the reasoning we used to compare the two options.

The term "out of band" is often read as "the guardrail only watches, it cannot block". That is a fair reading, and it is not what we built. There are two questions here that usually get merged into one.

The first question is about deployment: does the guardrail sit inside the traffic path, terminating and forwarding every connection, or beside it, reachable over an API? The second question is about enforcement: when the guardrail flags something, does anything actually stop, or is it only logged for someone to read later?

Those two questions have independent answers. An inline product can run in monitor mode and block nothing. An out of band product can block everything, as long as something in the path is willing to wait for its verdict and act on it. Ours is the second case: the deployment is out of band, the enforcement is fully synchronous, and the gateway is the thing doing the blocking. For readers who use "out of band" in its strict networking sense, a monitoring channel that cannot block, the precise description of our design is this: inspection happens outside the traffic path, enforcement happens inside it, at the gateway.

Once you separate the two questions, the real debate is not "inline or out of band". It is "what is the enforcement point". Our answer is the gateway, always. The guardrail is an advisor the gateway consults, not a checkpoint the traffic marches through.

This is the comparison we worked through. Neither column wins everything.

Dimension	Inline	Out of band	Better
Coverage	Guaranteed. Traffic cannot skip it.	Depends on gateway policy. Bypass is possible if the policy is weak.	Inline
Coupling	The vendor's uptime, releases and capacity limits sit in your data path.	One API integration, owned by the gateway.	Out of band
Skipping checks that add nothing	Not possible. Every call goes through, including embedding responses.	Set per route. Embedding calls skip the check by written policy.	Out of band
Latency	Extra hop on every call.	Extra hop only on checked calls.	Out of band
Replacing the vendor	A data-path migration for every application.	A change to one integration.	Out of band
Using more than one guardrail	Proxies chained in series. Slow and hard to troubleshoot.	The gateway calls each one and blocks if any objects.	Out of band
Failure mode	One behaviour for every workload.	Set per application, signed by the risk owner.	Out of band
Attack surface	The proxy terminates all traffic, so compromising it means sitting in the path.	The gateway holds every control, so its admin plane needs the strongest protection.	Neither
Works without a gateway	Yes.	No. It needs an enforcement point you trust.	Inline

The summary is simple. Inline gives you guaranteed coverage and works even without a gateway. Out of band wins most other rows, but only if you have a gateway you trust and you actively watch for bypass. We had the gateway, so we went out of band.

The moment that settled the decision came during a design walkthrough, looking at embedding traffic. An embedding response is a vector, a long list of numbers. There is nothing in it for a content filter to read. With an inline guardrail, that response still has to pass through a security product that can do nothing useful with it, paying the hop and the coupling for zero control value. Out of band, the gateway simply does not call the guardrail for that route. Chat completions get checked both ways; embedding responses skip the check by policy, and the policy is written down where it can be reviewed.

The deeper worry was coupling. Put a vendor's product inline and you are tied to it. Its availability becomes your availability. Its release cycle sits inside your change windows for the entire platform. Its capacity ceiling becomes your traffic ceiling. And when a better guardrail comes along, and in this market one will, replacing it means changing the data path of every application. That kind of migration rarely gets funded.

Out of band, the guardrail is one API integration owned by the gateway. It scales on its own hardware, which matters because content scanning is compute hungry in a way that request routing is not. Swapping vendors is a change to one integration, not to the traffic path. And it composes: nothing stops the gateway from consulting two guardrails on a sensitive route, a general-purpose one plus a domain-specific one, and blocking if either objects. Doing the same with two proxies chained in series would hurt both latency and troubleshooting.

None of this is free, and an architecture note that pretends otherwise should not be trusted. We wrote the costs into our security review board paper so they were accepted decisions rather than surprises.

The biggest one is bypass. Inline, the guardrail is unavoidable by construction, since traffic physically cannot reach the model without crossing it. Out of band, the guardrail is invoked because the gateway's policy says so. A misconfigured route, a lazy exception, or an operator with too much access can create a path to the model that never consults the guardrail. Our review states this plainly: a caller can reach the gateway without the guardrail being applied unless the gateway enforces it. The mitigation is to make invocation a platform policy that application teams cannot opt out of, to keep the gateway's admin plane tightly held, and to alert on any model call that completes without a matching guardrail verdict in the logs.

The second cost is concentration. The gateway is now the enforcement point for identity, secrets, routing and safety all at once. That makes it the strongest control point in the platform and, by the same token, the most valuable target in it. Anyone who can change gateway policy can quietly weaken every control that hangs off it. So the gateway gets the hardest controls we have: privileged access management on its admin plane, separation of duties so the person who writes a guardrail policy is never the person who approves it, and audit logging that someone actually reads.

The third cost is latency. Out of band does not mean free. The gateway waits for the verdict before forwarding, so every checked call carries an extra network hop and a scan. We accepted that, because the alternative was carrying that toll on every call including the ones with nothing to inspect.

The fourth cost is streaming, and it is the one that hurts in real projects, including ours. Applications stream model responses token by token so the user starts reading immediately. Our guardrail, like many, gives a verdict on a complete text in one shot; it cannot judge a response that is still arriving. So on guarded routes the gateway has to collect the whole response, check it, and only then release it, which takes away the benefit of streaming. This is not just our product's gap. The common gateway pattern today is the same: collect the chunks, assemble the full response, check, then deliver. Some guardrails can check a response chunk by chunk as it streams, but that carries its own trade-off, since a violation split across two chunks can slip past a checker that sees them separately, and researchers still describe streaming moderation as an open problem sitting between checking too late and checking with too little context. Where streaming matters to a workload, this row of the comparison favours neither option; it is a limit of guardrail products themselves, and worth testing before you promise streaming and a blocking guardrail in the same application.

The question every reviewer asked, correctly, was: what happens when the guardrail service is down or slow? There are only two honest answers. Fail open means the gateway forwards the call unchecked, keeping the application alive but running blind. Fail closed means the gateway refuses the call, keeping the safety promise but taking an availability hit on someone else's outage.

Our default for enterprise applications is fail closed. If a workload was risky enough to require a guardrail verdict on every message, it is risky enough that "we could not get a verdict" should mean "no". An application that touches customer data or drives a decision should not run unchecked just because the guardrail service is having an outage.

Fail closed also exposed a problem we did not expect: the guardrail itself is not deterministic. It uses an LLM to judge content, so the same prompt can pass one day and be blocked the next. In practice that means user tickets that cannot be reproduced, a false positive rate that is a range rather than a number, and application teams starting to doubt a policy that blocks them seemingly at random. We are working through it with the vendor by tuning the judging prompts inside the guardrail, and it has changed how we test: a guardrail policy is not verified by running a prompt once, it is verified by running it many times and looking at how consistent the verdicts are. If you are evaluating a guardrail product, ask about verdict consistency before you ask about accuracy.

It is a default, not a rule, because this is a risk appetite decision and pretending it is a technical one just hides it from the people who own the risk. An internal brainstorming assistant with no sensitive data might reasonably fail open with an alert. The point is that out of band deployment turns the failure mode into a per-application setting the business signs, instead of a single behaviour baked into an inline product. That is a feature: the architecture forces the risk conversation to happen per workload, in writing.

There is no slogan here like "never deploy a guardrail inline", because the trade-off table does not support one. If you have no gateway, no single enforcement point you trust, then an inline guardrail may be the only honest control you can deploy, and detection-only out of band monitoring is weaker than either. The argument in this post assumes you have, or are building, a gateway worth trusting.

If you are, here is what our experience says. Make the gateway your enforcement point and the guardrail a service it consults, so the safety verdict and the traffic path stay decoupled. Write the invocation policy per route, so calls with nothing to inspect are skipped on purpose and everything else cannot opt out. Choose the failure mode per application and have the risk owner sign it, defaulting to fail closed for anything that matters. And put the bypass risk in your security review in plain words, with a detective control behind it, because out of band designs are only as strong as the policy that forces the call.

Guardrail products will keep improving, and that is the reason I want ours at the end of an API call rather than fixed inside the traffic path. The safety layer should be the easiest part of the platform to replace, not the hardest.