Custom LLM training and hosting

1. The constraint

Two requirements arrive together and pull against each other.

The first is that the data cannot leave. For a bank or a telecom operator under residency rules, sending customer text to a public API is not a procurement question, it is a prohibited act, and the model therefore has to run where the data already is.

The second is that the work still has to be good. A client accepting a weaker system because it runs locally is a client who will stop using it, and the internal comparison people actually make is against whatever they can reach on their phone.

Fine tuning an open weight model is the lever that closes most of that gap. The consequence, which is easy to underestimate, is that you have now bought the entire serving stack. A hosted API hides capacity planning, utilisation, batching, quantisation, model versioning and rollback behind a price per token. Running it yourself means those become your problems on a Tuesday, and the one that decides whether the approach survives contact with a finance review is utilisation, because reserved GPUs cost the same whether requests arrive or not.

There is a third thing worth saying early. Fine tuning is the cheap part. Assembling an evaluation you trust costs more, takes longer, and is what determines whether anything ships.

2. The decisions, and where each one is enforced

2.1. Adapters rather than forks. LoRA and PEFT produce a small artifact against a known base, so one base model can serve several specialisations, a version is a file rather than a deployment, and reverting is instant. Full fine tuning gives up all three of those properties for a gain that rarely justifies them at this scale.

2.2. Every candidate is scored against a prompted baseline. This runs before anything else, on the same evaluation set, and it exists because a fine tune that loses to a good prompt is common and invisible without the comparison. The rule is enforced at the registry: a candidate without a baseline result attached cannot be promoted, so skipping the comparison blocks the release rather than quietly passing it.

2.3. Promotion is gated, and the gate has four parts. Golden datasets, regression suites, hallucination and safety probes, and human review on anything touching a regulated flow. Failing one is failing the gate. An override would be used within a month of existing, so none exists.

2.4. A model change is a code change. Versioned in the registry, promoted through the same approvals, rolled back by the same mechanism. Model artifacts get treated as configuration in most organisations, which is how a system arrives at nobody being able to say which weights answered a question a customer is now complaining about.

2.5. Routing per task, on cost and latency and accuracy together. Work that carries no residency requirement can go to a commercial API, which keeps the reserved hardware occupied by the work that has nowhere else to go. Paper 5.17 makes this argument at length. It is a property of the design rather than a description of this estate, which runs open weights only, and the router is what makes that a configuration rather than an architecture.

3. Why a client pays for this

The residency requirement is what creates the market, and the quality bar is what keeps it.

A client under residency rules has two options that both fail. They can use nothing, and watch their competitors automate. Or they can accept a visibly worse local system, which their own staff will route around within a quarter by pasting text into whatever they can reach personally, which recreates the exact leak the rule existed to prevent. A locally hosted model that is genuinely good enough removes the incentive to circumvent it, and that is the actual security outcome.

Cost per model is the second half of the argument, and it is the half that decides renewal. A client can see the GPU bill. Showing that a task moved to a smaller model at a fraction of the cost with no measured loss in quality is the conversation that keeps the platform funded, and it depends on having an evaluation credible enough that “no measured loss” means something.

4. Figures

This note reports none. The three the stub named remain the three that matter: GPU utilisation, cost per model, and the size of the gain over a prompted baseline. All are measured in production and none has been supplied for publication.

The third of those deserves a note. A gain over baseline is only as meaningful as the evaluation it was measured on, and failure mode 5.2 says that evaluation is built from failures already seen. A published improvement figure carries the coverage of its test set inside it, and quoting the number without the coverage is how a system acquires more confidence than it earned.

5. What the evaluation could not catch

The stub for this note said the honest version of the page would have to answer this, so it goes in its own section rather than a caveat at the end.

The gate described in section 2 is built out of known problems. Golden datasets come from production incidents and reviewed transcripts, regression suites come from bugs that were fixed, and safety probes come from categories somebody thought to enumerate. Each one is a record of something that already went wrong. A failure with no precedent in that record passes every check in the gate, arrives in production, and becomes a golden dataset entry afterwards, which protects the next client rather than the one who found it.

Sampling live traffic back into the evaluation narrows the window. It does not close it, because the sample is drawn after the fact and only from behaviour somebody flagged. Human review on regulated flows narrows it further and reviews a fraction. Both of those are worth their cost and neither converts the gate into a proof.

The honest description of what promotion gating buys is a floor rather than a guarantee: no release is worse than the last one on the things we have learned to measure. Papers 5.19 and 5.20 are the argument for why that distinction matters, and this note is a case of it rather than a rebuttal.

6. What I would do differently

Build the golden dataset before training anything. It is the artifact that decides what ships, it takes the longest to assemble because it needs real examples with agreed answers, and it routinely gets started after the first model is already waiting. Every week the evaluation lags the model is a week of decisions made on a demonstration.

Evaluate quantisation per capability rather than on an aggregate score. Failure 5.4 is open and it is the one most likely to be missed here, because an English-weighted benchmark can stay flat while Bangla performance falls away underneath it. The check costs one extra evaluation run per quantisation choice.

Budget base model migration as scheduled recurring work. Failure 5.5 is accepted, and what makes it painful is treating each upgrade as a surprise. Adapters are trained against a base version, so a base upgrade means retraining all of them and re-running the whole matrix. Planning that on a cadence keeps the adapter count honest, because a specialisation nobody will pay to retrain is a specialisation that should not exist.