Sovereign LLM gateway

A bank wants agentic customer service. Its regulator wants every token to stay inside the bank. Those two sentences are the entire project, and everything difficult about it follows from declining to compromise on either.

3. Decisions worth defending

3.1. Redaction before routing. Personal data detection runs in process before backend selection. It costs 14 ms at p99, and it means a misconfigured route cannot leak. A control that depends on the correctness of the next hop is not a control, it is a hope with a runbook.

3.2. Deterministic, stamped fallback. When a backend degrades, the gateway fails over to a smaller local model and marks the response as having come from the understudy. Downstream systems can see it, dashboards can count it, and the tenant can decide what it means. A silent quality drop is worse than an error, because nobody investigates it and the damage shows up in a churn report six weeks later.

3.3. Quotas as a first class object. Per tenant, per model, per minute, sliding window in Redis. This is not a cost control. It exists because one tenant’s retry storm used to be everybody’s outage, which is Principle 4.3 learned the expensive way.

6. What I would do differently

Model the audit schema first. We built four adapters and then discovered that a uniform audit record was the actual product, because it was the thing the tenant’s compliance team read and the only artifact that survived a backend swap.