# Mosthofa Imran, full text mirror Every paper and implementation note as plain text, front matter preserved, so confidence and state travel with the prose. Generated at build. Canonical HTML at https://mosthofaimran.com/. If you quote a claim, carry its confidence value with it. ================================================================ SECTION 3: IMPLEMENTATION NOTES ================================================================ --- 3.1 Mevrik: agentic customer experience platform --- url: https://mosthofaimran.com/impl/mevrik-cx/ state: production stack: Go, Python, TypeScript / Next.js, Flutter, gRPC + protobuf, PostgreSQL + pgvector, ClickHouse, Redis, NATS JetStream, MinIO / S3, Docker, Kubernetes, Helm, OpenTelemetry result: 3M+ conversations/month, 99.9% against contracted SLAs falls over at: Tool surface breadth rather than request volume. Grounding quality degrades as the number of individually registered tools grows, which is why promotion to an agent's tool surface is a reviewed step rather than a decorator.
What this is. A conversational platform where an AI agent handles customer contact end to end across chat, messaging, email and voice, and a human supervises what the agent escalates. It runs multi-tenant in public cloud for most customers and inside the customer's own estate, including fully air-gapped, for banks and regulated operators who cannot let a conversation leave their network. The same build serves both.
## 1. The constraint that shaped everything One sentence decided most of the architecture: **the same product has to run in a shared cloud and in a bank's air-gapped data centre, and neither can be a port of the other.** Paper 5.5 argues that sovereignty is cheap when it is adopted at design time and ruinous when it is retrofitted. This is the system that argument came from. Every decision below is downstream of refusing to maintain two builds. ## 2. Tenancy is a schema property, not a query habit The failure everyone fears in multi-tenant software is one tenant seeing another's data, and the usual defence is discipline in the application layer. Discipline is not a control. The contract here is structural and has four parts: - Every business table carries `tenant_id`, not null. - Every business table has a row-level security policy keyed to the tenant in the connection context, as defence in depth beneath the application scoping. - Every composite index leads with `tenant_id`, so the fast path is the scoped path. - No query joins across tenants. There is no legitimate reason to and no code path that does. The part that makes it hold is not the rule, it is the enforcement. **A new table without `tenant_id` fails the build.** Every migration pull request spins an ephemeral database, applies the full history from zero and runs a row-level security smoke test against it. A policy that is only in a document decays; a policy that fails continuous integration does not. ## 3. Two services, and the discipline was not splitting into more The backend is two services: a modular monolith carrying the gateway, identity, tenancy, conversations and the agent runtime, and a second service carrying AI compute. The interesting decision is the one not taken. Splitting the first service into gateway, identity, conversations and runtime would have looked more modern and bought nothing: they share a database, a tenant context and an auth context, so separating them adds deployment and failure surface without adding scaling headroom. The seams are kept clean so extraction is available later, and extraction happens on a measured signal rather than on taste. AI compute is separate from the first day for a reason that survives scrutiny: it has a genuinely different scaling profile. Bursty GPU work, vector index memory, model caches and high-cost-per-call operations scale on a different axis from a chat gateway holding websockets. Scaling those independently is a real saving rather than an architectural preference. Upgrade triggers are written down in advance, with the signal that fires each one, so growing the system is a decision taken calmly rather than a reaction. Paper 5.12 is about architectures that follow the org chart; this is the attempt not to. ## 4. Own the interface, rent the engine The agent runtime is built on six primitive interfaces defined in-house: tool calling, skills, a guardrail hook pipeline, memory, context curation and model routing. Those interfaces are the contract and they were frozen early. Underneath them, the runtime today wraps a vendor agent framework through an adapter. That is deliberate. Writing the loop from scratch in the first month would have been a way of spending a quarter on something a vendor had already shipped. Wrapping it without owning the interface would have been worse: every skill, tool and integration would have been written against somebody else's abstractions, and replacing the runtime later would mean rewriting all of them. So the loop is rented and the surface is owned. When the framework stops fitting, the implementation behind the interfaces changes and the skills, tools, hooks and evaluation suites written against them do not. ## 5. Guardrails are a pipeline, not a policy Every conversational turn runs through the same ordered hooks, and the ordering is the design.
EVERY TURN, IN THIS ORDER inbound message pre-flight PII redacted here, before the model is ever called agent runtime reason, call tools, observe, decide post-flight grounding, tone, sanitise, meter customer tool gateway injects tenant id at dispatch the model cannot construct a cross-tenant call THE AUDIT LANE the harness writes the trace at every stage, not the agent append only · hash chained · replicated to object-locked storage No prompt, tool input or model output leaves a stage without a record written by something other than the thing being recorded.
Figure 1. Three decisions in one picture. Redaction happens before the model, not after. The gateway injects tenancy rather than trusting the model to scope its own calls. The audit is written by the harness, so the record does not come from the process it describes.
Three of those are worth stating on their own. **Redaction runs before the model, not after.** A post-hoc scrub of a response is a cleanup; redacting on the way in means personal data was never in the prompt. The second is a control, the first is a hope. **The gateway injects tenancy at dispatch.** Tool calls do not carry a tenant chosen by the model. The gateway attaches it when the call is dispatched, so no output the model can emit constructs a cross-tenant request. This is the single control I would keep if I could only keep one. **The audit is written by the harness.** Paper 5.21 argues that a record authored by the process being observed is testimony rather than evidence, and that the fix is a boundary. This is that boundary, built before I had written the argument down: the trace is emitted by the runtime around the agent, append-only and hash-chained, and the agent has no write path to it. ## 6. Actions are planned, approved, then executed Any action that changes the world runs through a fixed loop. The agent emits a structured plan naming intent, target, side effects and whether the action is reversible. The surface renders it in plain language. A person approves at a chosen scope. Only then does the call dispatch, with audit written before and after. The part I am most pleased with is the smallest. **Every tool must declare whether it is reversible, and a pre-commit hook rejects any new tool that does not.** Irreversible actions require a second confirmation in which the operator types the resource name. The rule is enforced by the thing that will not let you commit rather than by a paragraph in a wiki. Where an agent assists a human rather than acting, suggestion-only is structural too: its tool allow-list cannot contain a write tool, enforced at registration, and a write tool tagged for that agent fails the build. ## 7. One image, four ways to run it Cloud multi-tenant, isolated multi-tenant, single-tenant in the customer's cloud, and air-gapped on-premise. Same container images, same deployment charts, same database schema. Configuration decides which mode a deployment is in. No build-time forks and no schema forks per edition. The reason is maintenance arithmetic rather than elegance: one security patch has to reach every edition, and reproducible builds across editions are an audit requirement rather than a nicety. A fork per deployment mode is a promise to do every fix four times, and that promise is always broken quietly. ## 8. What I would do differently **I would put the tool-surface governance in from the first week.** It arrived after the tool registry did, which meant a period where adding an endpoint to an agent's reach was a one-line change nobody reviewed. Grouping several hundred endpoints into a handful of domains turned out to be the single biggest lever on answer quality, and I found that out later than I should have. It is failure 6.1 above and it is the one I would tell someone else to front-load. **I would have built the evaluation gate before the guardrails.** Guardrails stop bad output reaching a customer. An evaluation suite tells you whether a change made the system better, and without one the guardrail pass rate becomes the quality metric by default, which is paper 5.19's argument arriving from the inside. **The numbers in this note are of two kinds and I have marked which is which.** Conversation volume, availability and recovery time are measured. Latency, kill-switch activation and throughput are targets the build is held to. Publishing a target as though it were a measurement is the specific dishonesty this site exists to avoid, and the temptation is real because targets are always rounder. --- 3.2 Sovereign LLM gateway --- url: https://mosthofaimran.com/impl/llm-gateway/ state: production stack: Python (FastAPI), Redis, OpenSearch, PostgreSQL, Ollama, Qwen, open-weight models only, 2 GPUs, self-hosted inference result: falls over at: The quota path. It is the one component every call passes through and the one that holds state, so it saturates before the model adapters do. The rate at which that happens is a property of a specific deployment and is not published here.
How to read this note. What follows is the reference design for a sovereign LLM control plane: the constraints this class of system operates under, the decisions that follow from them, and the failure modes it faces, with the industry-standard answers to each. It is a solution path for a system like the one I built rather than a disclosure of that system's internals. Deployment configuration, thresholds, tenant identities and incident history are deliberately not published.

What is specific and confirmed: the problem, the self-hosted backends running on two GPUs, and the model choices. The figures this page carried until 2026-09-03 were the handoff prototype's and were removed under erratum 7.17. Its failure modes were the prototype's too, and are replaced here by the ones the category actually has (erratum 7.18).
## 1. The constraint A bank wants agentic customer service. Its regulator wants every token to stay inside the bank. Those two sentences are the entire project, and everything difficult about it follows from declining to compromise on either. Agentic behaviour wants the strongest available model. Sovereignty wants a model you can install. Those pull in opposite directions, and the usual resolutions are to give up the capability or to give up the residency guarantee and describe it in language vague enough to survive a procurement questionnaire. The third option is a control plane: put every model behind one boundary, make the choice of backend a per-tenant policy decision rather than an architectural one, and enforce the data rules at the boundary instead of trusting each backend to behave. That is why the local backends run on two GPUs we own rather than on rented inference. A model you can install is a model whose weights, prompts and logs never leave a room you control, and it is the only version of this a regulator can be shown rather than told about. The models chosen for those slots have to self-host and still support tool calling, which is a smaller set than it sounds. The architecture has a slot for a hosted model, reachable only by a tenant whose data classification permits external inference, and this deployment does not use it: every model behind the gateway is open weight and runs on hardware in the room. The slot matters anyway, because the policy that would gate it is the same policy that proves the local tenants cannot reach one. ## 2. The decisions, and where each one is enforced
tenant policy quota redaction route per tenant per model before routing policy picks in order, and the request does not leave until all four have run self-hosted, 2 GPUs weights never leave the room hosted model, unused here the slot policy would gate the boundary: rules enforced here, not delegated to the backend EVERY RESPONSE, WITHOUT EXCEPTION provenance stamp which model answered, and why that one audit record append-only, uniform across backends caller A tenant whose classification forbids the hosted model cannot reach it by misconfiguration, because the policy lookup runs before the router and the router has no other input.
Figure 1. Four steps run before a request leaves the boundary, and two run on every response coming back. Nothing about the arrangement is clever. Its value is that there is exactly one path.
**2.1. Redaction before routing.** Personal data detection runs in process before backend selection, which means a misconfigured route cannot leak. A control that depends on the correctness of the next hop is not a control, it is a hope with a runbook. The ordering is **enforced at the boundary**: the router takes the redacted payload as its only input, so there is no code path in which an unredacted request reaches a backend. **2.2. Backend choice is per-tenant policy, not architecture.** Which models a tenant may reach is data, looked up per request, and a tenant whose classification forbids external inference cannot reach a hosted model by any configuration mistake. Building this as a deployment variable instead is the common shortcut, and it means every new tenant is a new deployment and every mistake is a residency incident. **2.3. Deterministic, stamped fallback.** When a backend degrades, the gateway fails over to a smaller local model and marks the response as having come from the understudy. Downstream systems can see it, dashboards can count it, and the tenant can decide what it means. A silent quality drop is worse than an error, because nobody investigates it and the damage arrives in a churn report six weeks later. **2.4. Quotas as a first class object.** Per tenant, per model, per minute, sliding window. Cost control is the secondary benefit here. Quotas exist because one tenant's retry storm is otherwise everybody's outage, which is Principle 4.3 stated as infrastructure. **2.5. One audit record, uniform across backends.** Same schema whether the answer came from a local model or a hosted one. This is the artifact a compliance team actually reads, and it is the only thing in the system that survives swapping a backend. ## 3. What the boundary is worth commercially The gateway is what makes a regulated customer a configuration rather than a project. Without it, every bank and every telecom operator with a different data classification is a separate deployment, a separate security review and a separate set of promises that somebody has to keep track of. With it, the answer to "can our data leave the country" is a policy row, and the answer to "prove it" is an audit record with the same shape for every tenant. That is the return, and it is a sales return before it is an engineering one. The security review that decides the contract asks which model saw the data and how you know. A stamped response and a uniform audit record answer both questions in a form the reviewer can keep. ## 4. Figures **This note reports none.** The measurements that would matter are gateway overhead at p50 and p99 with redaction included, redaction cost in isolation, the proportion of calls that failed over to the understudy, and the rate at which the quota path saturates. They are real measurements of a real deployment and they are not published, for the same reason the tenant identities are not. The figures this page carried until 2026-09-03 were the prototype's, and removing them was the right call. Publishing a plausible replacement would have been the same defect wearing better clothes. ## 5. Known failure modes The five in the front matter are the ones this category has. They are not incident reports from a particular deployment, and they are not softened: 5.2, 5.4 and 5.5 are open problems in the industry, not oversights waiting to be tidied up. Two are worth restating because they are the ones teams discover late. **Hosted models change underneath you** (5.4), which is the strongest practical argument for keeping a self-hosted side in a mixed estate. And **prompt injection reaches tool calls** (5.5), which redaction does nothing for, because the attack is not about what leaves on the way out. ## 6. What I would do differently **Model the audit schema first.** Adapters get built first because they are the visible work, and then the uniform audit record turns out to have been the actual product: the thing the tenant's compliance team read, and the only artifact that survived a backend swap. It should have been designed before the first adapter, not derived after the fourth. **Measure redaction per locale from the start.** Failure 5.2 is open and it is the one that degrades quietly. A system tested in English and deployed against customers writing in another script has a quality problem that no aggregate metric will surface, because the aggregate is dominated by the cases that work. **Treat tool permissions as the security boundary, not the model.** Failure 5.5 does not get solved at the prompt layer, and time spent hardening prompts is time not spent on the thing that actually contains the blast radius, which is what the tools are allowed to do. --- 3.3 Webhook ingestion with delivery guarantees --- url: https://mosthofaimran.com/impl/ingest-rs/ state: production stack: Rust (tokio), RabbitMQ, PostgreSQL, MinIO result: 100% webhook receipt rate falls over at: Not yet established by measurement. The architectural limit is the durable write on the accept path, since the receiver cannot acknowledge faster than it can persist, and that number has not been published here.
What this is. One service that receives callbacks from other people's systems, used by several platforms rather than owned by one. Social media webhooks, payment and API callbacks, and the mission-critical ones where the sender will not send twice if you mishandle the first attempt. It exists because every platform was solving the same problem badly and separately.
## 1. The constraint The sender does not care about you. That is the whole design. A webhook provider has its own retry policy, its own timeout, its own opinion about what your response code means, and no interest in your deploy schedule. Some retry aggressively and turn one event into six. Some retry once and give up. Facebook's will keep trying and then stop, and an event you dropped is simply gone. So the receiver cannot be a normal HTTP service that does some work and returns. **It has to be a service whose only job is to not lose things**, with the actual work happening somewhere it cannot affect the response. ## 2. The decision everything else follows from Accept and process are different jobs and they are separated by a durable write.
THE ACCEPT PATH, WHICH IS THE ONLY PATH THE PROVIDER SEES provider retries on you verify signature idempotency key durable write reject unsigned seen before? stop before the 200 200 only now provider done EVERYTHING ELSE, WHICH THE PROVIDER NEVER WAITS FOR queue per source handlers per platform dead letter a person drains it raw payload archive, kept replay, any time, without the provider A handler that fails never reaches the provider, so a bad deploy is our problem and not a lost event.
Figure 1. The provider only ever sees the top row. Everything that can fail lives in the bottom two, where failing is survivable and retrying is our decision rather than theirs.
**Verify, deduplicate, persist, then acknowledge.** In that order, with no step moved for speed. The acknowledgement is a promise that the event is durable, and returning 200 before the write turns a routine restart into permanent loss that nobody detects, because the provider believes it succeeded and will never send it again. **The raw payload is kept, not just the parsed one.** Parsers have bugs and schemas change underneath you. Keeping the original bytes means a parsing mistake discovered three weeks later is a replay rather than an apology, and the archive has repaid that storage cost more than once. **Failures go to a dead-letter path a person can drain**, never back to the provider. A handler crash is our problem. Bouncing it upstream converts an internal bug into lost data and into a provider quietly reducing its opinion of your endpoint. Two of these are **enforced at the accept path rather than written down as guidance**. An unsigned payload is rejected by the receiver, so "only verified events enter" is a property of the code that runs and not a rule someone remembers. A replayed event is recognised by its idempotency key before any handler sees it, so a duplicate cannot become a second record even if every handler downstream is careless. The ordering itself is the weaker part and it is worth being plain about. The guarantee holds because the write happens before the acknowledgement, and what protects that sequence is review and the fact that the people who work on it know why it matters. That is thinner than it should be for an invariant this load-bearing. ## 3. Why this is one service and not one per platform Every platform that takes callbacks needs the same six things: signature verification, replay protection, durable receipt, ordered-enough delivery, a dead-letter path and an audit of what arrived. None of those are product features. All of them are hard to get right, and all of them are silently wrong until the day they matter. Written per platform, each team gets its own subtly different bug. Written once, the delivery guarantee is a property of the infrastructure and every platform inherits it, including the ones that had not thought about retries at all. The cost is a shared component with several consumers, which means a change has a blast radius and the ordering compromise in failure 6.3 is a decision made on everybody's behalf. That is a real trade and it is the right one for a guarantee this specific. ## 4. What the guarantee buys the product A hundred percent receipt rate is an infrastructure number that becomes a product one. The person at the far end of a dropped webhook is a customer who sent a message and got no reply. They do not know a queue exists. They know they wrote to a company on Facebook and nobody answered, and the operator on the other side never saw it arrive, so both ends of the conversation believe the other is ignoring them. Every lost callback is one of those. Social and messaging platforms also treat endpoint reliability as a signal about the integration itself. Miss enough callbacks and delivery degrades, the integration gets flagged, and in the worst case a platform review arrives that costs more time than the engineering ever did. The teams building on top stop writing compensating logic, which is the quiet saving. No reconciliation job sweeping for messages that never landed. No "refresh to check if we missed anything" button, which is a confession rendered as a control. No support burden from conversations that arrived half-formed, and no support agent apologising for something the infrastructure did. That is what the accept path buys. It looks like plumbing and it is a product commitment about whether a customer's message can vanish. ## 5. What I would do differently **Build the dead-letter drain interface at the same time as the dead-letter queue.** The queue is a morning of work and the tooling to inspect and re-drive it is a week, so the queue ships first and then poison messages sit in it for longer than anybody would admit, because looking at them is awkward. A dead-letter path nobody can comfortably drain is a landfill. **Make the accept-path ordering a test, not a habit.** A contract test that drives the receiver, kills it between the write and the response, and asserts the event survives would turn the invariant in section 2 into something a build can check. It is an afternoon of work guarding the single decision the entire guarantee rests on, and it does not exist, which is the gap I would close first. **Treat silence as a signal from day one.** Failure 6.4 is still open and it is the one I would front-load in a rewrite. Every alert here fires on something going wrong, and the failure that actually costs you is an upstream that stops sending, which produces no errors at all. A per-source expected-rate check would have cost an afternoon. **The figures on this page are thin and that is deliberate.** The receipt rate is measured. The throughput and latency numbers this note used to carry were the prototype's and are gone rather than replaced with estimates. Where a system's real numbers have not been published, saying so is better than reaching for a plausible one, which is the whole argument of erratum 7.11. --- 3.4 Analytics migration to ClickHouse --- url: https://mosthofaimran.com/impl/olap-migration/ state: complete stack: ClickHouse, Kafka Connect, Airflow result: falls over at: The sort key. A columnar store answers a query that matches its ordering in a fraction of the time and answers one that does not by reading more than the row store would have. The limit is not row count, it is the distance between the ordering chosen at migration time and the queries that arrive two years later.
How to read this note. This is the reference design for moving an analytical workload from a row store to a columnar one under live traffic: the constraint, the decisions that follow, and the failure modes this migration has. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Table names, volumes, schedules and the specific queries are deliberately absent, and the pipeline components in the stack describe the shape of the thing rather than a certified inventory.

The figures this page carried until 2026-09-03 (fourteen billion rows, a p95 falling from 9.4 seconds to 380 milliseconds, and a rollback executed at 02:40) were the handoff prototype's and are removed under erratum 7.19. No replacements are invented.
## 1. The constraint Analytical queries on a transactional database work until they do not, and the transition is not gradual. A row store fetches whole rows. An analytical query wants three columns out of forty across a year of history, so the engine reads roughly thirteen times the data it needs, and the cost grows with total table size rather than with the size of the answer. For a long time this is fine and the fix is an index. Then a dashboard that took two seconds takes forty, and the honest description is that the storage model stopped matching the question. The migration itself is not the hard part. **The hard part is that the analytics were already load-bearing.** Reporting surfaces sit in front of customers, operators run their day from them, and regulated clients have reporting obligations that do not pause. So there is no window, no maintenance weekend, and no version of this where the numbers are allowed to be wrong for an afternoon while a cutover settles. ## 2. The decisions, and where each one is enforced
writes one source row store source of truth until cutover columnar store written, not yet read reconciliation counts, checksums, per partition divergence blocks the read move READS MOVE ONE QUERY AT A TIME, AND MOVE BACK THE SAME WAY query one dashboard per-query flag default is the row store clean for longer than the reporting window, then move one flag flip returns it, with no deploy Backfill runs behind the live stream and is allowed to overlap it, because the write is idempotent on a natural key. A pipeline safe to run twice is a pipeline safe to resume.
Figure 1. Nothing here is a cutover. It is a period during which both stores are correct, and a sequence of small reversible decisions about which one answers.
**2.1. Dual write, and the old store stays authoritative.** Both stores take every write from the beginning. The new one is not read from until reconciliation has been clean for longer than the longest reporting window, because a discrepancy that only appears in a monthly close is invisible for a month. Authority is **enforced at the read path** rather than agreed in a plan: the query layer's default target is the row store, and moving a query is an explicit act. **2.2. Reads move one query at a time, behind a flag.** Not one table, and never all at once. A dashboard is moved, watched against the old answer, and either kept or returned by flipping a flag with no deploy. This is slower than a cutover and it is the only version where "the numbers look wrong" has a same-minute answer. **2.3. The rollback is built first and exercised on a schedule.** Before any read moves, the path back is written and run. Then it keeps being run, against production, on a timer, because a rollback that has not executed this month is a plan and not a capability. Failure 5.5 is what happens when this lapses, and it lapses by default. **2.4. The schema is designed for the query, not translated from the source.** A columnar store rewards denormalisation, a sort key matching the dominant access pattern, and partitioning on the dimension that queries filter on first, usually time. Porting the normalised OLTP schema across is the fastest way to build something that is slower than what you left, and it is the most common way this migration fails. **2.5. Every load is idempotent on a natural key.** This one property makes backfill resumable, makes stream and backfill safe to overlap, and makes a duplicated batch harmless. It costs a key design decision at the start and it removes an entire category of incident. ## 3. Why this is a product decision and not a storage one Analytics that answer in under a second and analytics that answer in forty are different products, not the same product at different speeds. For the customers this served, reporting is not a convenience feature. A telecom operator sizes staffing from it, a bank reconciles against it, and a regulated client has obligations that assume it returns. When a dashboard takes forty seconds, people stop opening it and start asking a person, which converts an infrastructure cost into a support cost and hides it. The migration is also what makes the next thing possible. Query patterns that nobody proposes because they are known to be too expensive (cohorting across a full history, per-tenant breakdowns over a year) become ordinary requests once the storage model fits them. The measured benefit is the dashboards that got faster. The unmeasured one is the analysis that starts getting asked for. ## 4. Figures **This note reports none.** The four that would matter are rows migrated, query latency at p95 before and against after on the same query shapes, the divergence rate observed during dual writes, and the elapsed time from first dual write to full cutover. They exist for the deployment and are not published here. The figures this page did carry were the prototype's, and erratum 7.19 removes them. Publishing a plausible substitute would be the same defect in better clothes, which is the argument erratum 7.13 made and this note inherits. ## 5. What I would do differently **Build the differential test harness before moving the first query.** Failure 5.3 is the one that damages trust rather than uptime, and it is caught by running the same query against both stores and diffing the result, not by watching for errors. That harness is a day of work and it wants to exist before the first dashboard moves, not after the first argument about a number. **Keep a scheduled read from the old store for the whole dual-write period.** Failure 5.5 is open here for the honest reason: the discipline is easy to describe and it is the first thing that lapses once the migration looks finished. A cron job that reads production from the old store, weekly, is what keeps the rollback real, and it costs nothing. **Decide the sort key with a query log, not a design meeting.** The dominant access pattern is an empirical fact sitting in the existing database's logs. It is routinely guessed at instead, and failure 5.4 is the bill for guessing. --- 3.5 Air-gapped delivery pipeline --- url: https://mosthofaimran.com/impl/airgap-delivery/ state: production stack: OCI bundles, cosign, Helm, offline registry result: falls over at: Version skew. A site updated by hand drifts, and every migration has to tolerate arriving several versions late. The practical limit is how many versions back the upgrade path is still tested, not anything about the bundle or the install itself.
How to read this note. This is the reference design for delivering software into environments with no network path back to the vendor: the constraint, the decisions that follow, and the failure modes this kind of delivery has, with the standard answers to each. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Site identities, counts, deployment topology and install history are deliberately absent.

The figures this page carried until 2026-09-03 (six sites, zero failed installs since 2024-09) were the handoff prototype's, as was its summary, and both are removed under erratum 7.20. No replacements are invented.
## 1. The constraint There is no network path, and there is never going to be one. Not a firewall with an exception process. Not a proxy someone can whitelist a host on. A disconnected estate, in a building you will not enter, running on hardware you will not see, installed by an operator who has never met you and may not share a language with you. You cannot ship a hotfix, read a log, attach a debugger, or ask what the screen says. Everything difficult about this follows from one property: **the feedback loop is weeks long and runs through a person who is not an engineer.** In a connected system a bad release is found in minutes and fixed in an hour. Here it is found when somebody writes an email, and the email omits the part you need. So the artifact has to be complete, it has to be verifiable by someone with no way to ask a question, and it has to fail in ways that are legible to a person reading an error message for the first time. ## 2. The decisions, and where each one is enforced
CONNECTED SIDE, WHERE EVERYTHING MUST ALREADY BE DECIDED build with egress denied every dependency, chart, migration, and the registry itself, in one artifact sign reproducible bytes THE GAP physical media DISCONNECTED verify, then preflight key arrived out of band install refuses if either fails install, idempotent safe to run twice support bundle the only diagnostic there is carried back by hand, weeks later, paraphrased Anything resolved at install time is a network call, and there is no network. Everything else follows.
Figure 1. The arrow crossing the gap goes one way. Everything on the left has to be right before it does, because nothing on the right can be corrected afterwards.
**2.1. One artifact, genuinely complete.** Every dependency, every base layer, every chart and migration, and the registry to serve them from, inside a single bundle. Anything resolved at install time is a network call in disguise. This is **enforced at build time** by denying egress in the pipeline, so a package that would have been fetched on the far side fails here instead, where somebody can fix it in ten minutes. **2.2. The same image ships everywhere, and configuration alone decides.** No separate air-gap build. A special build for the hardest environment is the build that gets the least testing, and it is the one that has to work with no way to intervene. Connected estates run the same bytes, which means the air-gapped path is exercised continuously by everybody else. **2.3. Verification is refusal, not a warning.** The installer checks the signature and the hash against a key distributed out of band, and stops if either fails. Not a prompt, not a flag to continue anyway. An override exists in every system that has one, gets used under deadline pressure, and the property being protected is the only reason a regulated site accepted a vendor artifact at all. **2.4. Install is idempotent and resumable.** The operator will run it twice. Something will time out halfway, the media will be re-inserted, a step will be repeated. An installer that is only correct on a clean first run is an installer that will corrupt a site, and no one will be watching when it does. **2.5. Diagnostics are produced locally and designed to be quoted.** Preflight checks run before anything is written and fail with messages meant to be read aloud or pasted into an email, not interpreted. A support bundle the operator can generate on demand is the entire remote debugging story, so it is a feature with a spec rather than a log directory somebody tars up. ## 3. Why this decides which customers exist An air-gapped delivery path decides which customers can buy at all, which is a larger claim than hardening usually gets to make. Central banks, defence and government estates, and telecom operators under sovereignty rules do not have a procurement route for software that requires a connection home. The requirement arrives as a precondition, and a vendor without an answer is filtered out before the technical evaluation starts. The second effect is on everyone else. A product that installs with no network dependency installs cleanly in a restricted enterprise environment too, where the customer's security team has opinions about egress but not a physical gap. The work done for the strictest customer lowers the integration cost for every customer below them, which is unusual: hardening usually taxes the common case, and this is one of the times it subsidises it. ## 4. Figures **This note reports none.** The four that would matter are the number of sites under management, install success rate on first attempt, the version spread across the estate at a given moment, and elapsed time from release to a site being updated. They exist and are not published here, because site counts and identities are the customer's information rather than mine. The figures this page did carry (six sites, zero failed installs since a given month) were the handoff prototype's, and erratum 7.20 removes them. ## 5. What I would do differently **Deny egress in the build pipeline on day one, not after the first failure across the gap.** Failure 5.1 is entirely preventable and it is normally discovered the expensive way, because a connected pipeline is quietly forgiving and the gap is not. This is one line of CI configuration and it should predate the first bundle. **Version the bundle format itself, separately from the software.** The installer on a site is whatever version arrived last, and it has to read a bundle produced by a much newer build. Treating the bundle as a versioned interface with its own compatibility rules avoids the failure where a site cannot be upgraded because it cannot read the thing that would upgrade it. **Write the operator's runbook before the installer.** The error messages, the preflight output and the support bundle are the product for the person actually doing the work, and they get treated as documentation to be written afterwards. Failure 5.5 is open, and the part of it that is addressable is entirely a writing problem that engineers schedule last. --- 3.6 Audit evidence programme --- url: https://mosthofaimran.com/impl/compliance-evidence/ state: production stack: ISO 27001, GDPR, BNM RMiT, SOC 2 Trust Services Criteria, DevSecOps, CI release gating, vendor risk result: falls over at: Not established by measurement. The structural limit is coverage: evidence exists for controls wired into the pipeline, and any control a human performs by hand has no evidence except a human saying so.
What this is and what it is not. This note describes evidence machinery built for platforms operated under ISO 27001, GDPR and Bank Negara Malaysia RMiT, and carried through the security reviews banking and telecom clients run before they sign. It maps onto the SOC 2 Trust Services Criteria because the criteria overlap heavily, and it has been used that way alongside partner organisations holding signed Type II reports. I have not owned a SOC 2 programme end to end. Paper 5.15 section 8 says exactly what that distinction is worth, and this note does not claim more than it does.
## 1. The constraint An audit does not inspect your systems. It inspects a paper trail about your systems, covering a window that has already closed. That sentence decides the architecture. By the time an auditor or a client's security team asks, the period they care about is in the past, and no amount of engineering effort in the present can produce evidence for a Tuesday in March that nobody recorded. Evidence is a by-product you either captured at the time or did not, and nothing written afterwards substitutes for it. The failure mode this produces is well known to anyone who has watched it: a team is told an audit is in eight weeks, and eight weeks of engineering goes into reconstructing a story about the previous six months. The story is usually true. It is also unverifiable, and an auditor who cannot verify it has to record an exception, which is the thing everyone was trying to avoid. ## 2. The decision everything follows from **Evidence is emitted by the pipeline that does the work.** Not collected by a person, not screenshotted, not assembled in a spreadsheet during audit season.
CONTROLS THAT RUN INSIDE THE PIPELINE dependency and image scan access grant and review release gate, environment isolation each one emits as it runs evidence record timestamp, build id, actor, outcome reproducible append-only store retained, not edited THREE READERS, ONE STORE, DIFFERENT QUESTIONS internal review is it still running? client security review before they sign external auditor samples a closed window manual controls: no evidence but an assertion This is the coverage gap, and it is where exceptions come from.
Figure 1. The same record serves an internal reviewer, a client's procurement team and an auditor. They ask different questions and none of them should require a person to go and produce something.
**A control and the evidence that it ran are two different deliverables.** Teams build the first and assume the second. Scanning every image is a control. Being able to show, fourteen months later, that image `sha256:...` was scanned on a given date, by a named pipeline version, with a recorded outcome, is evidence, and it does not exist unless somebody decided it should. **Enforced at the release gate rather than documented as policy.** A build that skips the scan does not produce a warning, it fails. This matters more for evidence than for security: a control with an exception path has to have every use of that path explained to an auditor, and "we can override it in an emergency" turns one control into a sampling exercise across every emergency you ever had. **Retained append-only, because a mutable evidence store is not evidence.** If a record can be edited after the fact, its value in an audit is the value of somebody's word, which is what the record existed to replace. ## 3. Why this is delivery infrastructure, not a compliance function The reason to build it this way is the client's security team, who are the real gate and who arrive during procurement rather than after. For banking and telecom buyers, a security review sits between a signed intent and a contract. It is run by people whose job is to find the reason to say no, on a schedule set by the buyer, and it asks for artefacts rather than assurances. A team that can answer it from a store that already exists answers in days. A team that cannot spends three weeks producing documents, during which the deal does not move and the engineering roadmap does not either. That is the actual return. The compliance framework names the controls, and the evidence machinery decides whether answering for them costs a fortnight per deal or an afternoon. Every regulated customer asks the same questions in a different order, which is exactly the shape of problem that rewards building once. ## 4. Where SOC 2 fits The Trust Services Criteria overlap heavily with what ISO 27001 already requires, and the evidence a well-built programme emits serves both with different labelling. Access reviews, change management, monitoring and vendor risk are the same underlying records. The part that does not transfer is the observation window. SOC 2 Type II is not an assessment of your controls today, it is an assessment of whether they operated continuously across a period, and that period cannot be compressed by working harder. Paper 5.15 is the argument for treating that window as the schedule and everything else as procurement. Paper 5.16 is the argument that the only real compression is owning less infrastructure to evidence. Both grew out of this work, and both are published as arguments with confidence values and retirement conditions rather than as claims of a certification. ## 5. No figures, and why the page says so Section 3 admits a system with the numbers it produced. **This note has none**, and none are invented to fill the gap. The numbers that would matter are the ones this programme was never instrumented to produce: time from a client security questionnaire arriving to it being answered, the proportion of controls with pipeline-generated evidence against those attested by a person, and the count of exceptions raised across review cycles. Those are the right measurements. Not collecting them is a real gap in the programme rather than a gap in this write-up, and stating it is more useful than a plausible figure would be. ## 6. What I would do differently **Instrument the programme itself.** Everything above measures the systems under it and nothing measures the machinery. A programme whose whole argument is that evidence must be generated rather than asserted, and which then asserts its own effectiveness, has a hole in the middle of it. The three measurements in section 5 are cheap and I did not build them. **Write the exceptions register before the first review, not after the first exception.** An exception that is recorded, explained and has a remediation date is a normal artefact of a mature programme. The same exception, discovered by an auditor, is a finding. The difference is entirely who wrote it down first, and it costs nothing to be the one who did. **Test controls against known-bad input from the start.** Failure 6.1 is open and it is the one that keeps its shape no matter how good the evidence trail gets. A scanner that runs on every build and is silently misconfigured produces flawless evidence of nothing. Feeding a deliberately vulnerable dependency through the pipeline once a quarter would catch it, and it is an afternoon of work against a class of failure that an audit is structurally incapable of detecting. --- 3.7 Voice AI for customer service --- url: https://mosthofaimran.com/impl/voice-ai/ state: production stack: streaming STT, TTS, telephony, retrieval grounding, Bangla / Banglish result: falls over at: The latency budget, not the concurrency. Telephony scales by adding capacity. The loop from end of speech to first audio out does not, because every stage that would make the answer better spends the budget that keeps the conversation alive.
How to read this note. This is the reference design for a production voice agent in customer service: the constraint, the decisions that follow, and the failure modes this kind of system has, with the known answers to each. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Prompts, model choices, tuning thresholds, carrier arrangements and per-tenant configuration are deliberately absent.

What is specific and confirmed: the platform, code-switched Bangla and Banglish handled in production, live handoff to a person under SLA routing, and telephony at carrier-scale concurrency with recording retention rules and redaction on stored audio. This note reports no figures, and section 4 names the four that would matter.
## 1. The constraint Voice is a latency contract that nobody signs and everybody enforces. In text, a two second wait is normal. In a phone call, one second of silence is long enough for the caller to say "hello?", and two is long enough for them to conclude the line has dropped. The entire loop has to fit inside that: detect the end of speech, transcribe it, work out what was meant, retrieve whatever grounds the answer, generate the answer, synthesise it, and get audio moving. Six stages, one budget, set by human patience rather than by anything technical. **Every improvement available spends that budget.** A better model, an extra retrieval hop, a safety pass, a more natural voice: each is worth having and each costs time the conversation does not have. That tension does not resolve. It gets managed, or it gets discovered late. There is a second constraint here that is not general. **The callers code-switch.** Bangla and English are mixed inside single sentences, and the models available off the shelf are trained on corpora where that does not happen. They do not fail evenly. They fail on names, amounts and account terminology, which is to say they fail on the words the call is actually about, while the aggregate error rate stays respectable enough to look fine on a dashboard. ## 2. The decisions, and where each one is enforced
ONE BUDGET, SIX STAGES, SPENT IN ORDER caller audio in endpointing per locale streaming STT partials, not waiting intent + retrieval starts on partials generate TTS audio out, and the clock stops here barge-in: the caller speaking stops playback at once TWO PATHS THAT ARE NOT ERROR PATHS handoff to a person transcript and context carried, routed on SLA redaction, then retention enforced on write, not on read Handoff is built as a normal outcome. A system where reaching a person is the failure case will be tuned to prevent it, and the metric that rewards that is failure mode 5.4. Retrieval starting on partial transcripts is the single largest saving available, because it overlaps a stage with the caller still talking rather than making one stage faster.
Figure 1. The only structural way to buy latency is to overlap stages, which is why partial hypotheses drive retrieval before the caller has stopped speaking.
**2.1. Nothing waits for a complete utterance.** Speech to text emits partial hypotheses and retrieval starts on them. This is the one change that buys real time, because it overlaps work with the caller still talking instead of making a stage faster. It costs occasional wasted retrieval when the hypothesis changes, which is cheap, and it is the difference between a system that answers and one that pauses. **2.2. Barge-in is enforced at the audio layer, not requested politely.** Caller speech stops playback immediately, before any component higher up is consulted. A system that finishes its sentence while being interrupted is not perceived as slow, it is perceived as not listening, and that judgement is made once and not revisited. **2.3. Endpointing is tuned per locale rather than set as a constant.** How long a pause means "I have finished" is a property of a language and a speaking style, not a number. A fixed threshold clips one group of speakers and makes the system feel sluggish to another. **2.4. Handoff to a person is a first-class path.** The transcript and the context move with the call, and routing runs against an SLA. Building it as the error path is the common shape, and it produces a system that treats reaching a human as a defect, which is the direct road to failure mode 5.4. **2.5. Redaction and retention are enforced on write.** Stored audio and transcripts are redacted as they are stored, not filtered when they are read. A rule applied at read time is a rule that fails open the first time somebody queries the store a new way, and recorded voice in a regulated context is the least forgiving place for that. ## 3. What voice changes commercially Voice is where automation either becomes real or stays a demonstration, because it is the channel customers use when something matters. The commercial argument rests on availability rather than deflection. A caller reaching an answer in ninety seconds at three in the morning is a different product from a caller waiting for an office to open, and the operators running the service get their queue back for the calls that genuinely need a person. That is also why containment as a standalone target is corrosive: the value is in handling the calls that should be handled and passing on the ones that should not, and a metric that only counts the first will be optimised by damaging the second. The language work is what makes it usable rather than impressive. A voice agent that handles English cleanly and degrades on code-switched speech is a product for a subset of callers, and in this market that subset is the minority. Handling Bangla and Banglish in production is not a feature on a list, it is the difference between the system being usable by the customer base and being usable by a demo. ## 4. Figures **This note reports none.** The four tracked per call are known by name: end to end response latency, word error rate on Bangla and code-switched audio, containment rate and escalation rate. Their values have not been supplied for publication. Two of them should never be published alone even when they are. Containment means nothing without an outcome measure beside it, for the reason failure mode 5.4 gives, and word error rate on aggregate audio hides the code-switching problem that failure mode 5.2 describes. If these numbers arrive, they arrive in pairs or they mislead. ## 5. What I would do differently **Agree the latency budget per stage before building any stage.** Failure 5.1 is accepted rather than fixed because it is structural, but the version of it that hurts is the one discovered at integration, when six teams have each spent a reasonable amount of time and the total is unacceptable. A budget with a named owner per stage turns that from an argument into arithmetic. **Build the code-switched evaluation set before choosing a model.** It is the artifact that tells you whether anything is improving, and it takes weeks to assemble because it needs real audio. Choosing a model on published benchmarks and then discovering how it handles Banglish is the expensive order, and it is the usual one. **Publish the scorer's agreement with human review, from the first day the scorer exists.** Failure 5.5 is open and it decays quietly. A quality score that nobody has checked against a person in six months is a number the organisation trusts more the longer it goes unvalidated, which is precisely backwards. --- 3.8 Custom LLM training and hosting --- url: https://mosthofaimran.com/impl/llm-hosting/ state: production stack: LoRA / PEFT, self-managed GPU, quantisation, model registry, private cloud / on-prem result: falls over at: GPU utilisation, which is an economic limit rather than a technical one. Reserved capacity is paid for whether requests arrive or not, so the approach stops making sense below a utilisation floor that depends on the traffic shape and the hardware, and that floor is a property of a specific deployment.
How to read this note. This is the reference design for training and serving open weight models on infrastructure you own: the constraints, the decisions that follow, and the failure modes this work has, with the known answers to each. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Corpora, model choices, hardware, tuning parameters and per-client configuration are deliberately absent.

What is specific and confirmed came from the owner: open weight models fine tuned with LoRA and PEFT over domain corpora for terminology, tone and Bangla performance, served on self managed GPU infrastructure in private cloud and on-premise installs, with promotion gated on evaluation and model changes moving through a registry under the same change control as code. This note publishes no measurements, and section 4 names the three that would matter.
## 1. The constraint Two requirements arrive together and pull against each other. The first is that the data cannot leave. For a bank or a telecom operator under residency rules, sending customer text to a public API is not a procurement question, it is a prohibited act, and the model therefore has to run where the data already is. The second is that the work still has to be good. A client accepting a weaker system because it runs locally is a client who will stop using it, and the internal comparison people actually make is against whatever they can reach on their phone. Fine tuning an open weight model is the lever that closes most of that gap. The consequence, which is easy to underestimate, is that **you have now bought the entire serving stack**. A hosted API hides capacity planning, utilisation, batching, quantisation, model versioning and rollback behind a price per token. Running it yourself means those become your problems on a Tuesday, and the one that decides whether the approach survives contact with a finance review is utilisation, because reserved GPUs cost the same whether requests arrive or not. There is a third thing worth saying early. Fine tuning is the cheap part. Assembling an evaluation you trust costs more, takes longer, and is what determines whether anything ships. ## 2. The decisions, and where each one is enforced
TRAINING PRODUCES A CANDIDATE, NOT A RELEASE domain corpus terminology, tone LoRA / PEFT adapter, not a fork candidate the gate, and nothing goes round it beats a prompted baseline on the same set golden dataset and regression suite hallucination and safety probes human review on regulated flows fail any one and it stops here THE REGISTRY IS THE ONLY WAY IN, AND THE WAY BACK model registry, versioned same change control as a code release serving batching, quantisation, autoscale on GPU load rollback router, per task adapter, base model, or commercial API Routing is a cost decision on every request. Work with no residency requirement can leave, which is what keeps the reserved hardware busy with the work that cannot.
Figure 1. Training is the short arrow on the left. Everything expensive happens between the candidate and the registry, and everything commercial happens after it.
**2.1. Adapters rather than forks.** LoRA and PEFT produce a small artifact against a known base, so one base model can serve several specialisations, a version is a file rather than a deployment, and reverting is instant. Full fine tuning gives up all three of those properties for a gain that rarely justifies them at this scale. **2.2. Every candidate is scored against a prompted baseline.** This runs before anything else, on the same evaluation set, and it exists because a fine tune that loses to a good prompt is common and invisible without the comparison. The rule is **enforced at the registry**: a candidate without a baseline result attached cannot be promoted, so skipping the comparison blocks the release rather than quietly passing it. **2.3. Promotion is gated, and the gate has four parts.** Golden datasets, regression suites, hallucination and safety probes, and human review on anything touching a regulated flow. Failing one is failing the gate. An override would be used within a month of existing, so none exists. **2.4. A model change is a code change.** Versioned in the registry, promoted through the same approvals, rolled back by the same mechanism. Model artifacts get treated as configuration in most organisations, which is how a system arrives at nobody being able to say which weights answered a question a customer is now complaining about. **2.5. Routing per task, on cost and latency and accuracy together.** Work that carries no residency requirement can go to a commercial API, which keeps the reserved hardware occupied by the work that has nowhere else to go. Paper 5.17 makes this argument at length. It is a property of the design rather than a description of this estate, which runs open weights only, and the router is what makes that a configuration rather than an architecture. ## 3. Why a client pays for this The residency requirement is what creates the market, and the quality bar is what keeps it. A client under residency rules has two options that both fail. They can use nothing, and watch their competitors automate. Or they can accept a visibly worse local system, which their own staff will route around within a quarter by pasting text into whatever they can reach personally, which recreates the exact leak the rule existed to prevent. A locally hosted model that is genuinely good enough removes the incentive to circumvent it, and that is the actual security outcome. Cost per model is the second half of the argument, and it is the half that decides renewal. A client can see the GPU bill. Showing that a task moved to a smaller model at a fraction of the cost with no measured loss in quality is the conversation that keeps the platform funded, and it depends on having an evaluation credible enough that "no measured loss" means something. ## 4. Figures **This note reports none.** The three the stub named remain the three that matter: GPU utilisation, cost per model, and the size of the gain over a prompted baseline. All are measured in production and none has been supplied for publication. The third of those deserves a note. A gain over baseline is only as meaningful as the evaluation it was measured on, and failure mode 5.2 says that evaluation is built from failures already seen. A published improvement figure carries the coverage of its test set inside it, and quoting the number without the coverage is how a system acquires more confidence than it earned. ## 5. What the evaluation could not catch The stub for this note said the honest version of the page would have to answer this, so it goes in its own section rather than a caveat at the end. The gate described in section 2 is built out of known problems. Golden datasets come from production incidents and reviewed transcripts, regression suites come from bugs that were fixed, and safety probes come from categories somebody thought to enumerate. Each one is a record of something that already went wrong. A failure with no precedent in that record passes every check in the gate, arrives in production, and becomes a golden dataset entry afterwards, which protects the next client rather than the one who found it. Sampling live traffic back into the evaluation narrows the window. It does not close it, because the sample is drawn after the fact and only from behaviour somebody flagged. Human review on regulated flows narrows it further and reviews a fraction. Both of those are worth their cost and neither converts the gate into a proof. The honest description of what promotion gating buys is a floor rather than a guarantee: no release is worse than the last one **on the things we have learned to measure**. Papers 5.19 and 5.20 are the argument for why that distinction matters, and this note is a case of it rather than a rebuttal. ## 6. What I would do differently **Build the golden dataset before training anything.** It is the artifact that decides what ships, it takes the longest to assemble because it needs real examples with agreed answers, and it routinely gets started after the first model is already waiting. Every week the evaluation lags the model is a week of decisions made on a demonstration. **Evaluate quantisation per capability rather than on an aggregate score.** Failure 5.4 is open and it is the one most likely to be missed here, because an English-weighted benchmark can stay flat while Bangla performance falls away underneath it. The check costs one extra evaluation run per quantisation choice. **Budget base model migration as scheduled recurring work.** Failure 5.5 is accepted, and what makes it painful is treating each upgrade as a surprise. Adapters are trained against a base version, so a base upgrade means retraining all of them and re-running the whole matrix. Planning that on a cadence keeps the adapter count honest, because a specialisation nobody will pay to retrain is a specialisation that should not exist. --- 3.9 A constant tool surface over an unbounded API surface --- url: https://mosthofaimran.com/impl/openapi-studio/ state: production stack: TypeScript, Model Context Protocol, Astro SSR, Cloudflare D1, Cloudflare KV / R2, JSONPath, Server-Sent Events result: falls over at: Discovery, not throughput. The agent finds an endpoint by searching descriptions, so the ceiling is the quality of the spec it was given. A well-written spec of nine hundred endpoints works; a poorly written one of forty does not, and no amount of engineering on this side fixes the other side's prose.
How to read this note. This is the reference design for putting an agent in front of an API it has never seen: the constraint that shapes it, the decisions that follow, and the failure modes the shape has. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Tool names, schemas, deployment configuration, credential handling specifics and customer data are deliberately absent.

This note publishes no measurements, and section 4 names the ones that would matter.
## 1. The constraint An agent cannot use an API it has to be told about in advance. The obvious way to give a model an API is to generate one tool per endpoint. It works on the first demonstration and stops working on the second real system. A payments API has several hundred operations. A telecom platform has more. Every one of those tools carries a name, a description and a full parameter schema, and all of it has to sit in the model's context before it has read the user's question. The surface is spent before the work begins, and it grows every time the customer ships. There is a second problem underneath the first. **The tool list is fixed at connection time and the API is not.** An endpoint added on Tuesday is invisible to a tool list generated on Monday, so the integration is stale by definition and the fix is a redeploy per customer per change. So the requirement is a tool surface whose size does not depend on the API's size, and whose contents do not go stale when the API moves. ## 2. The decision everything follows from **A small fixed set of capabilities, and the API is discovered at run time.**
agent any client FIXED CAPABILITIES, THE SAME FOR EVERY API find an operation read its schema run it run a saved recipe the description loaded per request changed a minute ago is what the agent sees execution, on the server credentials attached here, recorded here, never sent back to the model the real API real effects A RECIPE IS SEVERAL REQUESTS BEHIND ONE CALL step one response extract by path feeds the next step two the model sees only the result Nothing above scales with the number of endpoints. That is the entire idea.
Figure 1. Four capabilities and a description loaded per request. An API with nine hundred operations presents exactly the same surface to the model as one with nine.
**2.1. The capability set does not grow with the API.** Find an operation, read its schema, run it, run a saved flow. That list is the same for every customer and every specification, so the context cost is a constant and the integration does not need regenerating when an endpoint is added. **2.2. The description is read at request time, not at connection time.** A specification edited a minute ago is the one the agent searches. This is what makes staleness structurally impossible rather than merely unlikely, and it is the reason the tool list can afford to be so small: the detail lives in the document, which is always current, instead of in a tool schema, which is current only until someone deploys. **2.3. Write capability is enforced at the tool list rather than in the handler.** A read-only credential does not receive the tools that mutate or execute: they are absent from the set the server returns, not present and guarded. The difference matters because a model that can see a tool will eventually call it and then explain to the user why it was refused, whereas a model that never receives it does not form the intention. Authorisation that shapes the menu is worth more than authorisation that rejects the order. **2.4. Credentials are attached on the server and never returned.** This is enforced at the execution boundary: the agent asks for a call to be made with the workspace's credentials, and there is no field in any response that carries one back. It does not receive them, cannot echo them into a transcript, and cannot leak them by summarising its own context. **2.5. Multi-step flows collapse into one call.** A recipe is a stored sequence where each step takes values out of the previous response by path expression. The agent calls it once. The intermediate responses stay on the server, which removes both the token cost of relaying them and the opportunity to mangle one on the way through. ## 3. Why this is the product rather than a feature of it The number of endpoints an organisation has is not something anyone chose. It is the residue of every integration, acquisition and migration since the company started, and it only goes up. That number is exactly what the naive design is priced against. Generating a tool per endpoint puts the cost of an agent integration in direct proportion to how much software a customer has already written, which means it works for the smallest prospect and fails for the largest one. The customers with hundreds of endpoints are the ones with the budget. The second effect is on who has to do the work. Under this design a customer with an existing specification is connected without an engineer writing an adapter, because the specification is the adapter. That moves the integration cost from a project to a configuration step, and a platform that takes a fortnight per customer is a consultancy rather than a product. Recipes are where that turns commercial. The valuable operations are never one call: they are authenticate, look up, create, poll until ready. Making that a single named thing the agent can invoke means the buyer sees an assistant that completes the task, rather than a model narrating four API calls and getting the third one wrong. ## 4. Figures **This note reports none.** The four that would settle whether the design works are the proportion of operations an agent locates on the first search against real customer specifications, the number of exchanges needed to complete a task compared with a per-endpoint tool surface, the failure rate of stored recipes over a period in which their underlying specifications changed, and the share of runs where the agent's stated intent matched what the request actually did. The last is the one that matters and the hardest to measure, because it requires a human to read both. ## 5. What I would do differently **Coerce every structured argument at the boundary from the first version.** Failure 5.1 is fixed and it should never have been possible. Models and clients send JSON as strings often enough that it is a property of the medium rather than a bug in any particular client, and accepting one into storage corrupts a document quietly. The rule is that nothing is written until it is the shape the document expects, and it belongs in the first commit rather than the one after the first corrupted spec. **Report what discovery could not resolve, loudly.** Failure 5.3 is open because it lives in the customer's prose, but the system currently fails as an absence: the agent searches, finds nothing, and moves on. Surfacing the operations that were searched for and not found turns an invisible failure into a report the customer can act on, and it is the only lever this side of the boundary has. **Validate stored recipes against the specification on every change.** Failure 5.4 is detectable in advance and is not currently detected. A recipe references paths, operations and response fields, and all three are checkable against the document the moment it is saved. Doing it at save time rather than at run time turns a call that fails in front of a customer into a warning in front of the person who edited the spec. --- 3.10 Attendance as a ledger, and the line it eventually bills --- url: https://mosthofaimran.com/impl/workforce-activity/ state: production stack: append-only ledger, projections, mobile capture, geofencing, payroll export, revenue attribution result: falls over at: The roster. Exceptions like late and absent are differences from an expectation, so a deployment without scheduling generates false exceptions from the first hour and the feature is rejected before anyone sees it work. The limit is organisational readiness rather than load.
How to read this note. This is the reference design for turning field attendance into payroll and into billable lines: the constraints it operates under, the standard decisions, and the failure modes this shape has. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Schema, table and field names, identifiers, client versions, thresholds, customer configuration and the internal document numbering are deliberately absent.

This note publishes no measurements, and section 4 names the ones that would matter.
## 1. The constraint Three consumers want to know where somebody was, and they do not want the same answer. **Payroll** wants a defensible record. It is the input to money paid to a person, it is retained for years, it will be produced in a dispute, and it must never change after the fact without the change itself being recorded. **Dispatch** wants a working estimate, right now. It is deciding who to send, and an answer that is approximately right immediately beats an answer that is exactly right in an hour. **The screen** wants to know whether a person is currently connected, which is neither of those. It is a property of the last few minutes and it is worthless tomorrow. One field cannot hold all three. The instinct is a single status enum with values like present, en route, on break and offline, and it fails on contact: a person can be rostered and absent, or absent and connected, or working and unreachable. Every combination is real and an enum admits one axis. The second constraint arrives at the other end of the pipeline. **A billed line has to be attributable to a person**, because that is what makes commission, utilisation and bonus schemes possible, and the attribution must be correct when several people did the work and honest when nobody knows. ## 2. The decisions, and where each is enforced
the phone writes events one ledger append only, never edited THREE PROJECTIONS, THREE QUESTIONS, NOT ONE ENUM attendance: payroll truth activity: dispatch estimate liveness: computed on read no sweep to fail locked day, payroll THE SAME VISIT, ON ITS WAY TO A BILL visit completed by one or several whose line is it? first match wins explicit, then who reported it done, then assigned credit splits, and sums to one a two-person job is worth one job Unmatched lines stay visible as unclassified. A line that cannot be attributed is a reporting gap somebody can close, and a line silently dropped is a number that lies. The console does not ship before the thing that produces its data, or it is a demo with a refresh button.
Figure 1. One writer, one ledger, three readers who disagree about what they are asking. The bottom half is the same day arriving at an invoice.
**2.1. One append-only ledger, not a table per event type.** Check-in, break, shift end and location are one kind of fact: something happened to somebody at a time. Split across separate tables they are a join every time anybody asks a real question, and the retention rule, the audit rule and the correction rule have to be implemented four times and will diverge. One ledger means the record is **enforced at write**: nothing is edited, corrections are new entries, and the audit trail is the storage rather than a feature added to it. **2.2. Three axes as separate fields.** Attendance, activity and liveness answer different questions over different time horizons and are allowed to disagree. Collapsing them into one status is the decision that looks tidy on day one and produces a report nobody can explain in month six. **2.3. Liveness is computed at read time.** Derived from the last heartbeat when somebody asks, rather than written by a job that marks people offline. A periodic sweep becomes a single point of failure for a fact that is not worth one, and its failure mode is silent and total. Nothing computed on read can be stale in a way the reader cannot see. **2.4. UTC in storage, local days in logic.** Every timestamp is stored in one zone and every business rule runs on the calendar day a human would name. A shift belongs to the day it opened on, which is what somebody means when they say what day they worked, and which stops an overnight shift becoming two days of partial hours. **2.5. Attribution is an ordered rule with fractional credit.** Explicit assignment first, then whoever reported the work finished, then whoever it was assigned to. First match wins, the rule that fires is recorded so the answer can be explained, and when several people attended the credit divides and sums to one. A line nothing can attribute is surfaced as unclassified rather than dropped, because a visible gap is a task and a silent one is a wrong total. **2.6. Integrity is a flag, never a gate.** Location that looks wrong marks the record for review and lets the person start their day. Anything that can refuse a check-in will eventually refuse a correct one, and the cost of that is a worker standing outside a building unable to work. ## 3. Why this is a product decision rather than plumbing The reason to build the ledger properly is that three departments are going to argue about it, and the argument will be settled by whichever number is easiest to produce. Payroll disputes are the sharp end. Somebody says they worked a day the system does not show, and the answer has to be a record with a history rather than a current value, because a current value invites the question of what it used to be. An append-only ledger answers that by construction, which is worth more than any feature built on top of it. The attribution half decides whether the operation can pay people for outcomes. Commission, utilisation and bonus schemes all need to know whose work produced a number, and every one of them is corrosive if the attribution is wrong: a scheme that rewards crew size, or that credits an administrator for field work, changes behaviour immediately and in the wrong direction. **A measurement that determines pay is not a report. It is an incentive**, and it is worth being slower and more explicit about than anything else in the pipeline. There is a sequencing rule underneath all of it that generalises past this system. **The console does not ship before the thing that produces its data.** A live operations view built ahead of the mobile client that writes the events is a demonstration with a refresh button, it will be evaluated as the product, and it will be judged as broken by the first person who compares it with the yard. ## 4. Figures **This note reports none.** The four that would say whether it works are the proportion of shifts closed by a person rather than by the rostered-end rule, the rate of integrity-flagged check-ins and how many survive review, the share of billed lines that reach an attribution rule other than the fallback, and the count of payroll disputes resolved by pointing at the ledger. The third is the one I would want first. If most lines are attributed by the last rule in the chain, the ordered rule set is decoration and the attribution is a guess with a procedure in front of it. ## 5. What I would do differently **Define the vocabulary once, in one place, before the second report is written.** Failure 5.6 is open and it is the one that erodes trust rather than uptime: two views that disagree about what "late" means are both correct and the operation stops believing either. A shared definition object that every report must consult is cheap at the start and a migration later. **Require the roster before enabling attendance, in the product rather than in the documentation.** Exceptions are differences from an expectation, so a deployment without scheduling produces false lateness from the first hour, and the feature is rejected before anybody sees it work. This is stated as a prerequisite and would be better as a gate. **Never let an attribution fallback stay on by default.** Failure 5.7 is disabled rather than fixed. The deeper lesson is that the last rule in a chain answers a question nobody asked, and a rule that has to be switched off in production is evidence that the chain should have ended one step earlier and returned nothing. ================================================================ SECTION 5: POSITION PAPERS ================================================================ --- 5.1 Competence Theatre --- url: https://mosthofaimran.com/papers/competence-porn/ state: holding confidence: 0.8 revised: 2026-08-31 retires: - A longitudinal study showing heavy consumers of technical content outperform matched peers on blind, time-boxed debugging tasks. - Evidence the effect is generational rather than structural, appearing at equal rate in cohorts who entered the field before ranked feeds existed. - A large publisher of technical content disclosing what fraction of the architectures it demonstrated reached production and survived twelve months, where that fraction is high. - A demonstration that the two mechanisms dissociate: a population that consumes heavily but assembles little, or the reverse, showing the production-survival gap in one and not the other. That would make this two unrelated papers sharing a title rather than one argument with two doors into it.
Abstract. The feedback loop that once rewarded building has been rerouted to reward the performance of building. The performance is cheaper to produce, faster to distribute, and structurally unfalsifiable. There are two doors into it, and they lead to the same room: watching a competent person work, and assembling a working system from parts nobody read. Both deliver the sensation of competence. Both supply what a thing does and neither supplies how it fails, which is the only question an incident asks. This paper states the mechanism, gives its strongest counter-argument the floor, and lists the evidence that would retire it. Confidence 0.80. The gap from 0.95 is Section 6.
## 1. The claim **A pilot logs hours. A surgeon logs procedures. We log tabs.** Somewhere in the last decade this industry discovered that watching a competent person work is more pleasurable than being one, and, more importantly, that from the inside the two feel nearly identical. A twelve minute video of someone untangling a difficult module ends in the same warm, settled feeling as having untangled it yourself. That is the business model of the medium rather than a defect in you, and it works because it is pointed at something real: the pleasure of watching craft is one of the oldest pleasures there is. Notice what a demo is engineered to remove. There is no data migration. There is no colleague who left in 2019 carrying the only working mental model of the billing service. There is no compliance officer, no partial failure, no clock skew, no forty page procurement questionnaire asking whether the vector store is FIPS validated. A demo is a jet engine bench tested at sea level and sold as a mountain crossing. Everyone involved knows this. Nobody is lying. The removal is what makes it watchable. For its first year this paper was about watching, because watching was the door I had walked through. There is a second one, and it works from the opposite posture: you are not sitting still consuming anything, you are working hard, assembling. It produces the same signal. That is Section 8, added later, because it took me a year to notice the two were one argument. ## 2. Why the numbers look fine We have more available knowledge per practitioner than at any point in this industry's history, and a persistent, widely reported sense among practitioners that they are behind. Those two facts look like a tension and are the same fact. The supply of things that resemble learning has outgrown the hours in which learning can actually occur, and the surplus has to go somewhere. It goes into the feeling. ## 3. Who absorbs the cost Not, mostly, the person watching. This is the part of the argument I had to rewrite after the correction recorded in the June 2026 revision, and it is where the paper stopped being about individuals. The cost lands on whoever inherits the decision. A demonstrated architecture arrives in a design review carrying the authority of having been seen working, and the evidence for it is a recording in which the hard parts were removed by construction. The person who adopts it pays the removed costs later, one at a time, and pays them in an environment where the demonstration is no longer available to argue with. 
  what the demo removed        who pays it back

  data migration        ->  the team, in month four
  partial failure       ->  whoever is on call
  compliance review     ->  a person you never met
  the departed colleague->  everybody, forever
  procurement           ->  the deal, six weeks late
Figure 1. The demonstration is honest about what it shows. The liability is in what it removed, and the removal is invisible precisely because it is what made the demonstration watchable.
The reader correction that produced the June revision was this: the effect is strongest where tooling is locked down, not where it is abundant. An engineer in a bank with a restricted toolchain watches more demonstrations, not fewer, because watching is the only form of access available. That inverts the naive version of the mechanism, in which abundance causes substitution. Constraint causes it too, and by a different route. ## 4. What survives contact with production The useful test is whether the demonstration carried the information you would need to operate it, rather than whether the technique is good. Almost none do, and the gap is systematic rather than accidental.
What a demonstration showsWhat operating it requires
The happy path, end to endThe failure paths, which are the majority of the code and none of the runtime
A fresh, empty systemBehaviour at year three, with accumulated data and two migrations behind it
One operator who wrote itA rotation of people who did not, at 03:00, holding a runbook
A bounded, chosen problemAn unbounded, inherited one with a compliance constraint attached
SuccessA named failure point, which Principle 4.7 asks for and demonstrations never give
This is the same structure as 5.2. Assembly produces the knowledge of what a thing does and not of how it fails; watching produces even less, and produces it with more confidence, because watching has no compile step to disagree with you. ## 5. The remedy, which is not "log off" Abstinence advice is both unrealistic and wrong. Watching skilled people work is a legitimate and efficient way to learn, and the apprenticeship objection in Section 6 is strong enough that I will not argue otherwise. The remedy is to attach a consequence to the watching, because the consequence is the only thing the medium removed that you can put back yourself. - **Convert one thing per week into a claim with a cost.** Not notes. A change to something you own, small enough to ship, that can fail in front of somebody. - **Predict the failure mode before you look it up.** Write one sentence on how the demonstrated system breaks. Then find a postmortem. The gap between your sentence and the postmortem is the measurement, and it is the only calibration signal available. - **Prefer artefacts with the hard parts left in.** Postmortems, migration write-ups, capacity notes with a stated breaking point. They are less pleasurable, which is the point: the pleasure was being produced by the removals. - **Count what you shipped, not what you consumed.** A pilot logs hours because hours flown is the quantity that predicts competence. The industry has no equivalent, so the substitute measure is whatever is easiest to count, and what is easiest to count is consumption. None of this is a cure and I am not claiming it as one. It is an attempt to reintroduce the feedback loop with consequences attached, which is the distinction Section 6 says I cannot yet state cleanly. ## 6. The strongest objection, unanswered **Apprenticeship.** All pedagogy involves watching. The apprentice watches the master, and this has worked for several thousand years. I do not yet have a clean line between apprenticeship and spectatorship, and until I do, this paper is weaker than its prose sounds. The distinction I am reaching for involves the presence of a feedback loop with consequences attached, but I cannot yet state it in a way that survives a determined counterexample. This objection is the entire reason confidence sits at 0.80 rather than 0.90. ## 7. What this paper does not claim Technical content is not worthless. The sharpest version of the objection above is that it is the primary way most people learn, and I have no answer to that. The producers are not dishonest either: the removals are what make the form work, and everybody involved knows they are there. Section 4 is an argument, not a measurement, and it should be read as one. The retirement conditions state the evidence that would end this paper, and the first of them is the study I would most like somebody else to run. ## 8. The second mechanism: assembly Added 2026-08-31. Sections 1 through 7 are about consumption. This one is about production, and it arrives at the same place. The IKEA effect is the finding that people place higher value on things they assembled themselves. The part that matters to engineering is the sensation rather than the valuation: competence produced by assembly feels, from the inside, nearly indistinguishable from competence produced by comprehension. You did real work. The system runs. The feeling that arrives is the correct feeling for having understood it, and you did not. This is the uncomfortable half of the paper. Section 5's remedy was to build instead of watch, and building is exactly what this describes. Follow that advice literally, wire together six services you never opened, and you arrive at the same deficit by a route that feels like effort the whole way. Effort is what makes it hard to see. Having recommended the route is what makes it hard to admit. ### 8.1. The shared signature Both mechanisms deliver knowledge, and both deliver the same two thirds of it.
KnowledgeWatching supplies it?Assembly supplies it?
What it doesYes, reliably. This is what a demo is for.Yes. The system runs, so it does something.
How it does itSometimes, if the presenter chose to show it.Sometimes, by accident, where a seam had to be forced.
How it failsNo. The removal of failure is what makes it watchable.No. Error paths are not exercised by getting it working.
The third row is the whole argument and it is the same row in both columns. Error paths are the least-read code in any system: rarely exercised in development, rarely covered by the happy-path tests that assembly produces, rarely present in the example the code was modelled on. A demo removes them because they are boring. Assembly skips them because the thing already works. 
   two doors, one room

   watching  --------+
                     |
                     v
             what it does      [supplied]
             how it does it    [sometimes]
             how it fails      [never]
                     ^
                     |
   assembling -------+

   the gap is invisible while the system works,
   and becomes the entire problem when it stops
Figure 3. The two mechanisms are opposite in posture and identical in what they leave out. That is the reason they are one paper rather than two.
### 8.2. Why they are one argument These are the same substitution sold at two prices, and the industry shipped both. Watch one engineer's week: they consume an architecture on Tuesday, assemble one on Thursday, and by Friday hold a confident, sincere and unearned model of a system that has never failed in front of them. Both days felt productive. One of them involved a keyboard. Split them into two problems and you get two wrong remedies. Treat it as a watching problem and the prescription is "build more", which is Thursday. Treat it as an assembly problem and the prescription is "read what you assembled", which sends people to read happy-path code they already understand. One substitution, one narrower remedy: **read the error paths**, in what you watched and in what you built. That is the row neither route ever fills in, and it is the only row an incident asks about. ### 8.3. What this section borrows The assembly mechanism is compressed here. It is developed at length in paper 5.2, which has the three-kinds-of-knowledge table this section abbreviates, the account of why the confidence signal misfires, four practices that address it, and the objection it cannot answer, which is that nobody reads their TLS library either. **5.2 remains the authority on assembly.** This section states only the part that makes it the same argument as Sections 1 through 7, and it does not supersede the paper it draws on. --- 5.2 Vibe Coding and the IKEA Effect --- url: https://mosthofaimran.com/papers/vibe-coding/ state: holding confidence: 0.75 revised: 2026-08-31 retires: - A blind study in which engineers who assembled a system without reading its generated internals diagnose induced faults in it at the same rate and speed as engineers who wrote the equivalent system by hand. - Evidence that the confidence gap in Section 3 closes with tooling rather than with reading, for example a generation workflow whose users predict failure modes as accurately as authors do. - A demonstration that the effect is about ownership rather than comprehension, appearing at equal strength for code the engineer merely selected rather than assembled, which would make this a paper about a different mechanism.
Abstract. Assembling a working system from parts you never read produces a strong, sincere sense of understanding it. The labour is real, so the ownership is real, but the labour was integration and the understanding it purchased is of the seams rather than of the parts. The gap is invisible while the system works and becomes the whole problem the first time it fails in a way the assembly did not cover. This argues about what reading buys and what assembling does not, rather than about who or what wrote the code. Confidence 0.75. Section 5 has the objection I cannot answer: nobody reads their TLS library either.

This paper is the long form of one mechanism. Paper 5.1, Competence Theatre, sets it beside the other one, watching, and argues that the two are the same substitution reached from opposite postures. Its Section 8 compresses what is below; this page remains the full treatment.

## 1. The claim The IKEA effect is the finding that people place higher value on things they assembled themselves. The interesting part for engineering is the sensation rather than the valuation: competence produced by assembly feels, from the inside, nearly indistinguishable from competence produced by comprehension. Both feel like understanding. Both produce accurate answers to "what does this do". They diverge on one question, and it is the only question that matters during an incident: *how does this fail?* ## 2. Three kinds of knowledge
KnowledgeWhat buys itWhen it is needed
What it doesReading the interface, or watching it run onceEvery day. Cheap and sufficient most of the time.
How it does itReading the implementation, or reconstructing it from behaviourDuring extension and optimisation. Assembly sometimes supplies this by accident.
How it failsReading the error paths, or surviving them in productionDuring incidents, and only then. Assembly never supplies it.
The third row is where the argument lives. Error paths are the least-read code in any system: they are rarely exercised in development, rarely covered by the happy-path tests that assembly produces, and rarely present in the example the code was modelled on. Generated and copied code inherits this bias, because it is trained on and drawn from code that is itself mostly happy path. 
  perceived                     actual
  competence                    competence

     |###########|                |####|
     |###########|                |####|
     assembly done               what it does
     working system              how it fails: 0
Figure 1. The divergence is not laziness. Assembly genuinely produces one of the three kinds of knowledge, and the feeling does not distinguish them.
## 3. Why the confidence signal misfires Effort is the proxy the mind uses for depth of understanding, and integration work is genuinely effortful. Getting four components to agree on a data format, an auth scheme and a deployment target is hard, absorbing, and produces a legitimate sense of achievement. The proxy is measuring the wrong quantity rather than being stupid. Two effects make it worse in current practice. **3.1. The feedback loop is fast and one-sided.** A generated module that compiles and passes the tests you asked for arrives in seconds, and the loop closes on "it works". Nothing in that loop ever asks what happens when the upstream returns a 503 mid-stream. **3.2. Review inherits the same bias.** A reviewer reads a diff for correctness against the stated intent. If neither author nor reviewer has read the error paths, the review confirms the happy path twice and records it as two independent confirmations.
Happy path, authoredread
Happy path, generatedskimmed
Error paths, authoredpartly
Error paths, generatedrarely

Figure 2. Ranked from code review, not measured. The bottom row is the claim; the others are context for it.

## 4. The remedy, which is not "write it yourself" Refusing generated or borrowed code is not available and would not be correct. Every engineer working today assembles from parts they did not write, and that has been true since the first shared library. The remedy is to buy the third kind of knowledge deliberately, because it is the only kind assembly never supplies for free. **4.1. The prediction test.** Before merging, state in one sentence how this component fails: what it does on timeout, on partial write, on malformed input, on a dependency returning success with an empty body. If you cannot answer without running it, you have assembled it and not read it. That is fine, and now you know which one you did. **4.2. Read the error paths only.** A full read of generated code is often not worth the time. A read of every branch that handles a non-success condition almost always is, and it is a small fraction of the lines. **4.3. Put the failure mode in the pull request.** One line, in the description. It makes the gap visible to the reviewer, converts an assumption into a claim, and is the artefact that pays off during the incident eighteen months later. See 5.11. **4.4. Add one adversarial test per component, not per function.** Not coverage. One test that does the rude thing: kills the connection halfway, sends the wrong content type, returns 200 with an empty body. ## 5. The strongest objection
This has always been true and we have been fine. Nobody reads the implementation of their TLS library, their JSON parser or their database driver, and the industry works. If the argument does not explain why assembling from a well-tested library is safe while assembling from generated code is not, it is nostalgia rather than analysis. My attempted answer is that a widely used library has had its error paths exercised by thousands of other people, and the resulting knowledge exists publicly in issue trackers and postmortems even if I have not read it, whereas code generated for me alone has a population of one and no such commons. I think that distinction is real, but I cannot yet state where the population threshold sits, and without that the argument does not give actionable advice about any specific dependency. That is the entire gap between 0.75 and 0.9. A second objection I take seriously: the effect may be about ownership rather than comprehension, in which case the remedy in Section 4 is aimed at the wrong target.
## 6. What this paper does not claim Generated code is not lower quality. In my experience it is often better than the median hand-written equivalent on the happy path, and the engineers using these tools are not less skilled than the ones who do not. Reading is not always worth the time either; Section 4.2 exists precisely because it usually is not. The claim is narrow. Assembly produces two of the three kinds of knowledge in Section 2, the missing one is the one incidents require, and the feeling of understanding does not report which of them you are holding. --- 5.3 Algorithmic Homophily --- url: https://mosthofaimran.com/papers/algorithmic-homophily/ state: revising confidence: 0.6 revised: 2026-08-14 retires: - A demonstration that technical opinion measured inside a subscription-shaped medium is no less varied than opinion measured outside it, on the same population and the same question, which would remove the effect this paper is about rather than its explanation. - Evidence that deliberate exposure to disagreement, as described in Section 5, produces no measurable change in the accuracy of technical forecasts, which would leave the paper describing something real and useless. - A mechanism that accounts for the private mailing list result in erratum 7.1 and predicts that ranking is nonetheless the dominant term, which would restore the original claim rather than the narrowed one.
Abstract. A ranked feed behaves like a cache with a hit-rate objective and no invalidation policy. It returns what you already agreed with, faster each time, and the sensation it produces is not comfort but consensus, which is why it is hard to notice from inside. This paper is being revised. The mechanism proposed in Section 3 was shown to be wrong in August 2026; the effect appears to be real and my explanation of it was not. Confidence 0.60, down from 0.80. See erratum 7.1.
## 1. The claim, as it now stands Two claims, which the first version of this paper did not separate, and separating them is most of what the revision is for. **The observation.** Practitioners inside a given medium converge in technical opinion faster than the underlying evidence justifies, and the convergence is experienced as consensus rather than as narrowing. I still hold this at roughly the confidence I started with. **The explanation.** That ranking algorithms optimising for engagement are the cause. This is the part that broke, and it broke cleanly rather than partially, which is the good kind of wrong. ## 2. The observation, and why it is hard to see Convergence is invisible from inside because the mechanism that produces it also produces its own corroboration. The same opinion arriving from eleven directions reads as eleven independent confirmations. It is one confirmation, resampled. 
  what it feels like        what it is

   A  B  C  D  E             A --+
   |  |  |  |  |                 |
   +--+--+--+--+             B --+-- one source,
        |                    C --+   eleven arrivals
        v                    D --+
     consensus               E --+
Figure 1. Independent confirmation and correlated resampling are indistinguishable at the point of reading. Only provenance separates them.
The engineering consequence is specific: technology choices acquire the appearance of settled practice before the evidence exists, and the papers, benchmarks and postmortems that would test them arrive two years later, by which point the choice is load-bearing. Section 5.10 of this document is a retracted paper of mine that was produced exactly this way. ## 3. The mechanism, under rewrite
This section is wrong and is being replaced. It is retained rather than deleted, per Section 2.2 of the index draft. Two readers demonstrated in August 2026 that the same convergence appears in closed, chronological mailing lists with no ranking whatsoever, which the argument below cannot account for. Erratum 7.1 records the correction and credits it.
The original argument ran as follows. Ranking optimises for engagement. Engagement correlates with agreement, because disagreement costs attention and produces exit. Therefore a ranked feed converges on your priors, and does so faster the more you use it. The cache analogy did real work here: a cache with a hit-rate objective and no invalidation policy will happily serve stale entries forever, because staleness is not on the objective. The mailing list result falsifies the necessity of ranking. Whatever produces the convergence is present when ranking is absent. **The candidate replacement**, which I am not yet confident enough to state as the paper's position: the binding term is *selection*, not ranking. Choosing whom to read is already a filter, it is applied once and revisited almost never, and it is applied on criteria (clarity, seniority, agreeableness) that correlate with existing agreement. Ranking then accelerates a process that subscription already started. If that is right, the paper's advice barely changes and its target changes completely: the intervention belongs at the subscription boundary rather than at the feed. I do not yet have a way to distinguish the selection account from a third possibility, which is that professional communities converge for ordinary social reasons that have nothing to do with media at all. Until I can, this section stays marked. ## 4. What the revision does not change The observation in Section 1 survives the correction, and so does the practical advice, which is why the paper is being revised rather than retracted. Advice that survives the falsification of its own mechanism should be treated with suspicion, and I am treating it with suspicion: it may be surviving because it stands on its own, or because it was never load-bearing on the mechanism in the first place. ## 5. The practice, offered at 0.60 - **Write down what would change your mind before reading.** This is the same discipline the retirement conditions on this site enforce, applied to consumption rather than to publication. It is the only intervention in this list I am confident about, because its value does not depend on which mechanism is correct. - **Audit the subscription boundary, not the feed.** If the selection account is right, this is where the whole effect enters. Once a quarter, list who you read and ask which of them has told you something you did not want to hear. - **Prefer artefacts with provenance.** A postmortem, a benchmark with a methodology, a paper with a retirement condition. These are harder to resample, because the source is attached. - **Count arrivals, not sources.** When a position reaches you from many directions, the useful question is how many distinct pieces of evidence sit behind it, which is usually one. ## 6. The strongest objection
This may be an ordinary property of professional communities, described in a technological vocabulary that adds nothing. Engineers converged on opinions before ranked feeds, before mailing lists, and before the internet, through conferences, employers and textbooks. If the effect appears with ranking, without ranking, and plausibly without any medium at all, then "algorithmic" in the title is doing no work and the honest paper is a much older one about professional consensus. I cannot currently rule this out. It is the reason the confidence is 0.60 rather than 0.70, and it is the reason Section 3 is marked rather than quietly rewritten.
## 7. What this paper does not claim It does not claim consensus is wrong; most consensus is correct and cheaply acquired. It does not claim engagement optimisation is malicious. It does not, as of the August 2026 revision, claim that ranking causes the effect, and any quotation of the earlier version saying so should be treated as withdrawn. And it does not claim a fixed timeline for the rewrite, because the replacement mechanism is not yet good enough to publish. --- 5.4 The Easy Button Tax --- url: https://mosthofaimran.com/papers/easy-button-tax/ state: holding confidence: 0.85 revised: 2026-08-14 retires: - A widely adopted abstraction that removed a class of friction and whose failure modes are demonstrably cheaper to diagnose than the friction it replaced, measured in operator time during incidents rather than in developer time during authoring. - Evidence that the cost transfer described in Section 2 does not occur where the author and the operator are the same team, which would reduce this to an argument about organisational structure rather than about abstraction. - A convenience layer shipping with an escape hatch that is exercised in its own test suite as a first-class path, adopted at scale, showing that the tax is a choice rather than a property.
Abstract. Friction is information. An abstraction that removes a step also removes the moment where the constraint behind that step was learned. The constraint does not disappear; it reappears at incident time, in front of a person who did not choose the abstraction, priced in hours they do not have. This argues for reading the invoice before signing, and for insisting that every easy button ship with a tested escape hatch, rather than against abstraction itself. Confidence 0.85. The missing 0.15 is Section 5, where the argument still cannot tell a good abstraction from a costly one in advance.
## 1. The claim Every convenience layer makes a trade with a specific shape: it converts a large number of small, predictable, design-time costs into a small number of large, unpredictable, incident-time costs. The trade is often correct. It is almost never priced, because the two sides of it are paid in different currencies by different people at different times. The word "tax" is chosen carefully. A tax is a known charge on a transaction rather than a scam, and the failure here is the charge being undisclosed at the point of sale rather than the charge existing at all. ## 2. The three transfers
TransferFromTo
In timeDesign time, when the system is calm, the decision is reversible and the person has contextIncident time, when the system is degraded, the decision is urgent and the context has to be rebuilt from logs
In personThe author, who chose the abstraction and understands what it hidesThe operator, who inherited it, and who is often on a different team, in a different timezone, three years later
In kindMany small comprehension costs, each cheap and each teaching somethingOne large diagnostic cost, expensive and teaching nothing except the shape of this particular abstraction's internals
The third transfer is the one that compounds. Learning a constraint by hitting it during authoring produces knowledge that transfers to the next system. Learning it by reading a stack trace through four layers of framework at 03:00 produces knowledge about that framework's internals, which is worth very little the moment the framework is replaced. 
  cost
   ^
   |  ..... friction paid at design time
   |  .   .   .   .   .   .   .   .   .
   |  --------------------------------- calm
   |
   |                                  ##
   |                                  ##  incident
   |  ................................##
   +------------------------------------> time
      abstraction adopted           first
                                    real failure
Figure 1. The same total cost, differently distributed. The right-hand column is paid in a currency the left-hand column was not.
## 3. Diagnosing an easy button The useful question is not "is this abstraction good" but "what did it decide on my behalf, and can I see the decision". Three tests, in increasing order of how much they tell you. **3.1. The naming test.** Can you name the thing it hides? If the answer is a vague category ("it handles the networking") rather than a specific mechanism ("it retries idempotent requests three times with deterministic backoff and no budget"), you do not know what you bought. See 5.6 for what that particular blank cheque costs. **3.2. The escape hatch test.** Is there a documented way to drop below the abstraction for one call, and is that path exercised in the library's own tests? An escape hatch that exists in the documentation and not in the test suite is a plan, not a mechanism. This is the single strongest predictor I have found of whether an abstraction will be survivable in year three. **3.3. The incident test.** Read one public postmortem from a team that hit this abstraction's failure mode. If none exists, either the abstraction is new or its failures are being resolved by vendor support tickets, which means the diagnostic knowledge is not in the commons and you will be rebuilding it yourself.
Names what it hidescheap
Escape hatch, testeddecisive
Public failure recordrare

Figure 2. The bars rank how much each test tells you against how often it can be satisfied. Ranked, not measured.

## 4. The outcome this argues for Not "avoid convenience". The outcome is a disclosure practice, and it is small enough to adopt this week. - **Write the transfer down at adoption time.** One line in the design document: what this removes, what it hides, and who pays when it fails. It takes ten minutes and it is the artefact the operator will want in three years. - **Require the escape hatch before adoption, not after.** If dropping one call below the abstraction requires forking the library, the abstraction is not a layer, it is a ceiling. - **Exercise the hatch once, in CI.** A single test that goes around the convenience path keeps it alive. Escape hatches rot silently otherwise. - **Treat the first incident as the invoice arriving.** Record the price in the postmortem so the next team can compare it against the convenience it bought. See 5.11. ## 5. The strongest objection
This is a general argument against progress, and progress has mostly been right. Garbage collection, optimising compilers, managed relational databases and TLS libraries are all easy buttons, all hide enormous complexity, all relocate friction to incident time, and all were correct. If the argument cannot distinguish those from a badly designed convenience wrapper, it distinguishes nothing. My attempted distinction is that the good cases hide a mechanism that is genuinely universal and genuinely solved, so the hidden constraint is nearly never the thing that fails, while the expensive cases hide a mechanism that is domain-specific and still contested. I cannot yet state that cleanly enough to apply it in advance rather than in hindsight, and applying it in hindsight is worth very little. That gap is the whole 0.15, and it is why this paper is not at 0.95.
## 6. What this paper does not claim The tax is usually worth paying, and nothing here argues for shallow abstractions or for building it yourself. The authors of convenience layers are not the target either. In the cases that cost the most, the author was unusually careful, which is exactly why the abstraction was adopted widely enough to cost anything. The claim is about disclosure, and about who receives the bill. --- 5.5 On-Premise Is Not a Downgrade --- url: https://mosthofaimran.com/papers/on-prem/ state: holding confidence: 0.9 revised: 2026-08-14 retires: - A team that adopted the single-artifact discipline at design time and can show, over two years, that it consumed more total engineering hours than maintaining separate cloud and on-premise builds. - Regulated estates routinely permitting outbound connections to vendor control planes, which would make the assumption list in Section 2 historical rather than current. - A cloud-only system of comparable complexity demonstrating equivalent dependency hygiene, reproducibility and upgrade safety without any sovereignty constraint forcing it.
Abstract. On-premise delivery is expensive when it is treated as a cloud deployment with things taken away. Treated as a constraint adopted at design time it is not a downgrade, and the discipline it forces (one artifact, explicit dependencies, an offline supply chain, configuration as data) improves the cloud build as well. The cost of sovereignty is the number of assumptions you can no longer make rather than hardware, and that number is finite, enumerable and mostly known on day one. Confidence 0.90. Section 5 holds it there: elastic workloads are a real exception and I have not sized how large that exception is.
## 1. The claim The industry describes on-premise work in the vocabulary of loss. Legacy. Regression. Enterprise tax. That vocabulary is not neutral, and it produces a specific engineering failure: teams build for the cloud, ship, and then attempt to subtract their way to an on-premise release. Subtraction is where the cost is. Almost every hour I have watched burned on sovereign delivery was spent removing an assumption that had been free to avoid at the start and expensive to remove later. The claim is narrow and it is about ordering. Sovereignty adopted as a constraint before the first architectural decision costs roughly what any other constraint costs. Sovereignty adopted as a port costs several multiples of that, and the multiple grows with the age of the codebase. ## 2. The assumption ledger What actually changes between a public cloud estate and an air-gapped one is a list. It is shorter than the folklore suggests, and each entry has a known design response.
Assumption removedDesign response
Outbound network egressNo vendor control plane, no licence phone-home, no telemetry upload, no package fetch at deploy time. Everything the system needs at run time is inside the bundle.
Managed servicesEvery managed dependency sits behind an interface the system owns, with at least one self-hosted implementation that is exercised in CI rather than kept as a claim.
Vendor-driven upgradesUpgrades become signed bundles applied by the operator, with a rollback that is tested rather than documented.
Observability as a serviceMetrics, logs and traces terminate inside the estate. Support debugging happens on evidence the customer exports deliberately, not on a dashboard you can open.
Elastic capacityCapacity becomes a stated failure point rather than an autoscaling policy. See Principle 4.7.
You, as the operatorThe operator is a person you will never meet, working a maintenance window at 02:00, holding a printed runbook. This is the entry that changes the most and gets the least attention.
That last row is the one that decides whether a sovereign product is viable. Every other entry is an engineering problem. The operator is a design problem, and the failure mode is a system that is correct and unoperable. ## 3. Why the constraint pays back The interesting result is that each of these responses is independently good practice rather than merely achievable, and a team that adopts them under sovereign pressure ends up with a cloud build that is measurably better than the one they would have shipped without it. 
  source  ->  reproducible build  ->  signed bundle
                                          |
                     +--------------------+
                     |                    |
                 public cloud       air-gapped site
                 (same bytes)        (same bytes)
                     |                    |
                 config as data      config as data
Figure 1. One artifact, two destinations. The moment the two paths diverge, the second one starts rotting, because only the first is exercised daily.
Three specific payoffs, in the order I have seen them arrive. **3.1. Dependency honesty.** A build that must run with no egress cannot pretend about what it depends on. Transitive fetches, implicit base images and "it works on the runner" all fail immediately rather than in year three. **3.2. Reproducibility becomes non-optional.** Shipping a bundle to a site you cannot reach means the bundle has to be the whole truth. That forces reproducible builds, which in turn makes cloud incidents diagnosable, because the artifact in production is byte-identical to one you can rebuild. **3.3. Configuration stops living in the environment.** When the environment cannot be inspected, configuration has to be data that travels with the deployment and can be diffed. This removes a large class of cloud incidents whose root cause is a variable set by hand in a console two years ago.
Design-time constraintbaseline
Port at v1higher
Port at v3highest

Figure 2. The shape of the cost, not its magnitude. I do not have defensible figures for the ratios and will not invent them; see docs/PLACEHOLDERS.md.

## 4. What this looks like as a rule set - **One artifact.** If cloud and on-premise builds differ, they differ in configuration data, never in code paths selected at build time. - **Every managed dependency behind an owned interface**, with a self-hosted implementation running in CI on every commit. - **The supply chain is offline-first.** Vendored, hashed, and verifiable without a network. A build that needs the internet is a build that cannot ship to a bank. - **Upgrades are signed bundles with a tested rollback**, applied by a stranger, in one maintenance window, with no interactive prompts. - **Capacity is stated as the point at which the system fails**, not as a target it meets. The operator needs to know where the edge is, because they cannot add nodes. - **The runbook is a deliverable**, versioned with the code, and it is wrong until somebody who did not write the system has followed it end to end. ## 5. The strongest objection
For genuinely elastic workloads, on-premise is a real downgrade and calling it a constraint is a euphemism. This is correct and it bounds the paper. A workload whose value comes from absorbing a hundred-fold burst for four hours a year is worse on fixed hardware, and no amount of design discipline recovers that. The paper holds for systems with predictable load envelopes, which in my experience is most regulated workloads, but "most" is doing work in that sentence and I have not quantified it. There is a second cost I have understated: the single-artifact discipline slows the first six months, and for a team still searching for product fit that slowdown can be fatal. The constraint pays back over years. Not every project has years.
## 6. What this paper does not claim On-premise is not cheaper. Total cost of ownership is usually higher, and the customer is usually paying it deliberately, for reasons of jurisdiction, audit or counterparty risk that have nothing to do with engineering. Nothing here says cloud teams are undisciplined, only that the sovereign constraint removes the option of skipping the discipline, which is a different and much weaker statement about them. The assumption ledger in Section 2 is the list I have needed so far rather than a complete one, across six sites, and every new estate has added to it. --- 5.6 The Retry Storm You Built On Purpose --- url: https://mosthofaimran.com/papers/retry-storm/ state: holding confidence: 0.95 revised: 2026-08-14 retires: - A fleet of a thousand or more clients running deterministic exponential backoff with no jitter and no retry budget, surviving a sixty second dependency outage with no correlated arrival spike, measured at the dependency rather than at the client. - Evidence that the default settings of the major client libraries now ship full jitter and a caller-side budget, which would make this a paper about a solved problem rather than a live one. - A queueing analysis showing that at realistic client counts the benefit of jitter is dominated by other recovery effects, so that removing it changes nothing measurable.
Abstract. Exponential backoff is presented as a politeness mechanism. Without jitter it is a synchronisation mechanism. A single shared fault starts every client's timer at the same instant, deterministic delays preserve that alignment through every subsequent round, and the recovery attempt arrives as a series of spikes that grow with the length of the outage. The failure is that retries are budgeted as a count per call instead of as a fraction of forward traffic, rather than that engineers do not know about jitter, and that nobody writes down which layer is allowed to retry. Confidence 0.95. The missing 0.05 is Section 5: the sharper version of this paper is about budgets alone, and I have not rewritten it that way yet.
## 1. The claim A retry is a load-generating decision made by a component that has just been told the system is under stress. That is the whole problem in one sentence. Every other property of retry behaviour follows from it. Exponential backoff is the standard mitigation and it is a good one. It is also incomplete in a specific, mechanical way. Backoff controls *when* one client retries. It says nothing about whether a thousand clients retry at the same moment. If the delay schedule is deterministic, and the fault that triggered it was shared, then the schedule does not spread the load. It preserves the alignment the fault created and carries it forward, round after round, for as long as the outage lasts. ## 2. The mechanism A dependency returns errors starting at `t=0`. Every in-flight caller observes the failure within one round-trip time of each other, which on a healthy internal network is a window of a few milliseconds. That is the synchronising event. From then on, a deterministic schedule of 1s, 2s, 4s, 8s keeps the whole population inside that same few milliseconds at every retry boundary. 
            0s      1s      2s      4s      8s
             |       |       |       |       |
no jitter    |#######|#######|#######|#######|
           fault    R1      R2      R3      R4
                     ^       ^       ^       ^
             every caller arrives in the same window

full jitter  |#.#..#.|..#.#..|#..#..#|.#..#.#|
             arrivals spread across the whole gap
Figure 1. Deterministic backoff preserves the alignment the fault created. Jitter destroys it. Both schedules have the same mean delay.
The dependency therefore recovers into a square wave rather than into a ramp. Its first moment of health is also the moment of peak concurrent arrival, so it fails again, which re-synchronises the population, which produces the next spike. The system has found a stable oscillation and will stay in it until something outside the loop intervenes. Two amplifiers make this worse than the single-layer picture suggests. **Amplification through layers.** Retries compose multiplicatively. Three attempts in the SDK, inside three attempts at the gateway, inside three attempts in the calling service, is twenty-seven rather than nine or three, and no single layer's configuration looks unreasonable on its own.
1 retrying layer3×
2 retrying layers9×
3 retrying layers27×

Figure 2. Worst-case request amplification for three attempts per layer. The multiplier is the product of the layers, not the sum.

**The load grows while the outage lasts.** Callers that would have arrived during the outage do not disappear. They queue in the client, in the connection pool, in the upstream's own inbound buffer. The longer the dependency is down, the larger the population that arrives in the first post-recovery window. ## 3. What it looks like in production The signature is specific enough to be diagnosed from a graph without reading any code.
SignalWhat a retry storm looks like
Arrival rate at the dependencyPeriodic spikes at 1s, 2s, 4s, 8s offsets from the fault, not a smooth ramp.
Ratio of attempts to distinct callsRises well above 1.0 and stays there. This is the single most useful number and almost nobody emits it.
Recovery shapeHealth flaps. The dependency comes up, dies inside one window, comes up again.
Client-side latencyp99 grows by the sum of the backoff schedule, so it looks like the dependency got slow when in fact the caller is waiting on its own timers.
The third row is why this is rarely caught in testing. A load test drives a synthetic client population with independently random start times, which is jitter arriving by accident. The synchronising event is absent from the test because the test never has a single shared fault. ## 4. The remedy, in the order it matters The order is deliberate. The first item is the cheapest and the third is the one that actually holds. **4.1. Jitter, and specifically full jitter.** Sleep a uniform random value in `[0, base * 2^n]` rather than the value itself. Equal jitter and decorrelated jitter are both defensible; the important property is that the delay is drawn from a distribution rather than computed, so that no two callers share a wake time except by coincidence. **4.2. A retry budget at the caller, expressed as a fraction.** Not "three attempts per call". A token bucket refilled from forward traffic, permitting retries only while retries stay under roughly ten percent of successful requests over a sliding window. A count-based limit rises with load exactly when it should fall. A fraction-based budget falls with success exactly when it should. **4.3. Exactly one retrying layer, named in the design document.** This is the rule that survives reorganisation, because it is a written decision rather than a configuration value. Every other layer converts failures into errors and returns them. If nobody can say which layer owns retries, the answer is all of them. 
  caller  ->  gateway  ->  service  ->  store
    [R]         [ ]          [ ]        [ ]
     |
     +-- retries here, with a budget and jitter
         every other hop fails fast and reports
Figure 3. One retrying layer. The choice of which layer matters less than the fact that it is written down.
**4.4. Emit the attempt-to-call ratio.** If the number is not on a dashboard, the storm is invisible until it is an incident. See 5.13. ## 5. The strongest objection
Jitter is textbook, and this paper is scolding people for something they already know. That is close to right, and it is what keeps the confidence off 1.0. Nearly every engineer who reads this can define jitter. The claim survives on a narrower footing: knowing about jitter has not translated into budgets, and budgets are the part that actually bounds the blast radius. Jitter spreads the same total load; only a budget reduces it. I hold 0.95 rather than 1.0 because the sharper version of this paper would be about budgets alone, and I have not yet rewritten it that way. There is also a real cost I have understated: full jitter adds tail latency to the common case of a single transient error, and for small fleets that cost can exceed the benefit.
## 6. What this paper does not claim Retries are not the problem. A system without them converts every transient fault into a user-visible error, which is worse than a storm you can bound. Jitter alone is not the remedy either, and Section 4.2 exists because of that. The ten percent figure is a starting point that has held for me across several systems rather than a derived constant, and I would expect it to be wrong for anything with a very different ratio of read to write traffic. Nor is any of this novel. The mechanism is well described in the literature. What the paper claims is a gap between that description and what is actually configured in production, and erratum 7.3 records me falling into that gap myself, in code I reviewed and approved. --- 5.7 Chesterton's Fence Has a Git Blame --- url: https://mosthofaimran.com/papers/chestertons-fence/ state: holding confidence: 0.8 revised: 2026-08-14 retires: - A codebase of substantial age where the instrument-then-remove procedure in Section 4 produced no measurable reduction in regressions from cleanup work, compared against a matched period of direct deletion. - Evidence that guard clauses whose recorded reason has decayed are, in aggregate, no more likely to be load-bearing than newly written ones, which would make the caution in this paper an expensive superstition. - Tooling that reliably reconstructs the intent behind a change from the surrounding artefacts, at accuracy high enough that the decay ladder in Section 2 stops mattering.
Abstract. The parable says do not remove a fence until you know why it was built. In software the answer is usually recoverable, for a while, from the commit, the pull request, the issue and the incident it followed. The parable is therefore easier to obey here than anywhere else, and it is still routinely disobeyed, because the recoverable context decays on a schedule nobody plans around. This paper is about the case the parable does not cover: when the history is one line and a date, and the honest answer to "why is this here" is that nobody knows. Confidence 0.80. Section 5 is why it is not higher: I cannot separate a load-bearing fence from a decorative one before instrumenting it.
## 1. The claim Two claims, and the second is the one that is actually contested. The weak claim: before deleting a guard clause, a retry, a sleep, a null check or a special case, spend ten minutes on `git log -S` and the linked pull request. This is cheap, it works more often than people expect, and it is not interesting. The strong claim: the context that would answer the question decays predictably, and the decay is fast enough that the parable's advice is unavailable for most code older than about three years. What is needed for that case is a procedure, not more diligence. ## 2. The decay ladder Every reason for a line of code lives somewhere. The places are ordered by how long they survive, and the ordering is stable across every organisation I have worked in. 
  survives longest
    |  inline comment naming the failure
    |  test asserting the behaviour
    |  commit message with the reason
    |  pull request description
    |  issue tracker entry
    |  incident channel / chat thread
    |  the person who wrote it
  survives shortest
Figure 1. The decay ladder. Everything below the third rung depends on a system, an account or a person outliving the code, and none of them reliably do.
The bottom four rungs share a property: they are outside the repository. They depend on a vendor account still existing, a chat retention policy, a tracker migration that preserved comments, or a person still answering email. The repository is the only artefact with the same lifetime as the code, which is why the top three rungs are the only ones worth relying on.
Inline commentcode’s life
Commit messagerepo’s life
Pull request bodyvendor’s life
Issue trackermigration
Chat threadretention
The authortenure

Figure 2. The bars rank expected survival rather than measured half-lives. The ranking is the claim; the lengths are only a drawing of it.

## 3. The obligation this puts on the author Most writing about Chesterton's fence addresses the person removing it. The larger gain is on the other side, because it is cheap at the moment of writing and impossible afterwards. A guard clause should carry its reason at the top of the ladder, in one line, naming the failure rather than the behaviour: 
  // Rejects zero-length batches. The 2024-11
  // ingest incident: an empty batch advanced the
  // offset without a write, so replay skipped the
  // window silently.
  // Test: batch_empty_does_not_advance_offset.
  if (batch.isEmpty()) return
Figure 3. Four lines that make the fence removable by someone who was not there. The test name is the load-bearing part.
A test that fails when the guard is removed is better than any comment, because it enforces rather than explains. The comment is for the case where the reason is not expressible as an assertion, which is more common than test-first advice admits: rate limits, ordering assumptions about an external system, and workarounds for defects in software you do not control. ## 4. The procedure for the decayed case When the history is genuinely thin, the parable gives no guidance beyond "do not remove it", which taken literally means codebases only accumulate. The alternative is to convert the unknown into an observation.
§Step
1Instrument, do not delete. Leave the guard in place and emit a counter with a distinguishing label every time it fires. This is a small, reversible, obviously safe change.
2Wait one full business cycle. Not a week. Whatever period contains your month-end close, your quarterly batch, your annual reconciliation. Fences are usually built for the rare path, which is exactly the path a two-week observation misses.
3Read the fires. If the counter is non-zero, you now have the reason, expressed as the inputs that reach it, which is better evidence than the original commit message would have been.
4If it is zero, remove with the evidence attached. The commit message says the counter ran for the named period at the named volume and never fired. That message is now the top rung of the ladder for whoever revisits this.
5Keep the rollback cheap for one more cycle. Removal is a change like any other, and it deserves the same rollback plan as a feature.
The procedure has a cost, and the cost is the objection. ## 5. The strongest objection
This turns every cleanup into a quarter-long project, and most fences are cargo cult. Both are substantially true. A large fraction of guard clauses in any old codebase are defensive habit, copied from a neighbouring function, protecting against nothing. Applying Section 4 to all of them would make deletion so expensive that nobody deletes, and a codebase that cannot shrink is its own failure mode. I do not have a reliable test that separates a load-bearing fence from a decorative one before the instrumentation runs, which is the entire reason this sits at 0.80. The partial answer I use is to apply the procedure only where the guard touches money, ordering, retention or an external contract, and to delete freely elsewhere. That heuristic is a judgement call wearing a rule's clothing, and I know it.
## 6. What this paper does not claim Old code has no special claim on survival, and `git blame` is not sufficient: Section 2 exists because it usually is not. The ordering in Figure 1 records what I have seen hold everywhere I have looked, which is weaker than a measurement. The procedure in Section 4 is ordinary feature-flag practice rather than anything novel, pointed at a deletion instead of a release, which is the one place almost nobody points it. --- 5.8 RAG Is a Search Problem in a Trench Coat --- url: https://mosthofaimran.com/papers/rag-search/ state: holding confidence: 0.7 revised: 2026-08-14 retires: - A published evaluation on a realistic corpus in which swapping the embedding model, holding chunking, query construction and retrieval strategy fixed, produces a larger gain in answer accuracy than fixing chunking while holding the model fixed. - Context windows and attention costs reaching a point where whole-corpus prompting is economically routine, which would remove the retrieval stage this paper is about rather than improve it. - Evidence that retrieval recall of the answer-bearing passage is not the binding constraint in production systems, for example generation reliably recovering answers absent from the retrieved context.
Abstract. Retrieval-augmented generation is an information retrieval pipeline with a language model at the end of it. Most of the quality is decided by the classic parts: how documents are split, what metadata survives the split, how the query is constructed, and whether ranking is hybrid. Those decisions are usually made in one afternoon by whoever set the system up, and then never revisited, while the team spends the following two quarters comparing embedding models. This paper is deliberately narrower than the claim I got wrong in 5.10, and erratum 7.2 is the reason. Confidence 0.70. Section 5 is why it is not higher: long context may retire the retrieval stage rather than improve it.
## 1. The claim If the passage containing the answer is not in the retrieved set, nothing downstream recovers it. Not a better reranker, not a larger model, not a more elaborate prompt. That is a hard ceiling, and it is set by decisions that happen before any embedding is computed. The claim is therefore about attention, not about technology: the marginal engineering hour in most retrieval systems is better spent on chunking, query construction and evaluation than on the embedding model, and teams reliably spend it the other way because the model is the part with a leaderboard. ## 2. Where the losses actually happen 
  document
     |  split            <- most loss enters here
     v
  chunk  (metadata kept? structure respected?)
     |  embed            <- the part everyone tunes
     v
  vector  ->  index      <- recall ceiling set here
     |
  query  (as typed? expanded? filtered?)
     |  retrieve k     <- k set once, never tuned
     v
  rank   (dense only? hybrid?)
     |
     v
  assemble context     <- order, truncation
     |
     v
  generation
Figure 1. Six decisions before generation. Five of them are ordinary information retrieval and predate the current vocabulary by decades.
**2.1. Splitting.** Fixed-size windows with a fixed overlap are the default in every starter template, and they cut through tables, headings, list items and the sentence that defines the term used in the next paragraph. A split that respects document structure (sections, table boundaries, list integrity) usually beats any model change, and it costs a day. **2.2. Metadata loss.** The chunk arrives at the index having forgotten which document, which version, which section, which effective date and which tenant it came from. Every one of those is a filter that would have removed most of the false positives, and the loss is silent because the pipeline still returns plausible results. **2.3. The query is used raw.** Users type fragments, misspellings and internal jargon. The corpus is written in formal prose. Dense retrieval is good at bridging that gap and lexical retrieval is good at exact identifiers, part numbers and error codes, which is precisely where dense retrieval fails and precisely what users paste in. **2.4. `k` was chosen once.** Almost always 3, 5 or 10, on the first day, and it is never tuned against a measurement because there is no measurement. ## 3. The measurement that changes the conversation Retrieval quality and answer quality are different numbers and must be measured separately. Almost every team measures the second and infers the first, which makes every regression ambiguous.
§Step
1Build an evaluation set of real questions, fifty is enough to start, taken from what users actually asked rather than what the corpus makes easy.
2Label the answer location, not the answer. For each question, record which passage of which document contains it. This is the expensive part and it is done once.
3Measure recall@k of the answer-bearing passage. This number is the ceiling on the whole system. If it is 0.6, no amount of generation work takes the system above 0.6.
4Only then measure answer quality, and treat any gap between recall and answer accuracy as a generation problem rather than a retrieval one.
Once step 3 exists, the argument about embedding models resolves itself empirically in an afternoon, in either direction, which is the outcome this paper actually wants.
Structure-aware splittinglarge
Metadata filterslarge
Hybrid lexical + densesolid
Tuning k, rerankingmoderate
Newer embedding modelsmall

Figure 2. My ordering of expected marginal gain, not a measurement. It is a hypothesis this paper asks you to test with Section 3, and it is exactly the kind of ordinal claim that ought to carry a confidence value.

## 4. The order of work this argues for 1. Evaluation set with labelled answer locations. Nothing else is decidable without it. 2. Fix splitting so it respects document structure, and keep document, section, version and tenant on every chunk. 3. Add lexical retrieval alongside dense, and fuse the rankings. Identifiers and error codes stop disappearing. 4. Use the metadata as filters before ranking rather than as display fields after it. 5. Tune `k` and add reranking against the measurement from step 1. 6. Then, and only with a number to compare against, consider the embedding model. ## 5. The strongest objection
Long context may make the retrieval stage vestigial. If it becomes economical to put a whole corpus, or a whole document set, in front of the model, then splitting strategy stops being a quality decision and becomes a cost decision, and this paper is about a transitional period rather than about a property of the problem. I do not know how to weigh that, and it is most of the reason this sits at 0.70 rather than higher. There is a second objection I take seriously: my ordering in Figure 2 comes from systems with structured, versioned, tenant-scoped corpora, where metadata is unusually valuable. On a flat corpus of undifferentiated prose the metadata rows collapse and the ordering may invert. And I am aware that the last time I made a confident claim in this area I had to retract it in full, which is recorded in erratum 7.2 and is the reason this paper is scoped to a measurement practice rather than to a prediction about technology.
## 6. What this paper does not claim Embedding models matter, and vector indexes are necessary. Saying otherwise about the second of those is the over-claim that retired 5.10, and I am not making it again. Figure 2 does not generalise; it is labelled a hypothesis because that is what it is. Classical information retrieval does not solve this on its own either. The claim is only that the classical parts are where the unspent engineering hours are, and that a team without the measurement in Section 3 has no way to find out whether that is true of their corpus. --- 5.9 The Ship of Theseus Passes Its Integration Tests --- url: https://mosthofaimran.com/papers/ship-of-theseus/ state: draft confidence: 0.55 revised: 2026-08-14 retires: - A completed incremental migration of substantial size where no identity declaration was made, and where ownership, invariants and the decommissioning of the old path nonetheless resolved cleanly within a year of the last route moving. - Evidence that end-to-end invariants are preserved by route-level verification in practice, which would remove the specific decay this paper is worried about. - A demonstration that the residual old system is retired at similar rates whether or not a decommissioning date was declared in advance, which would make Section 4 ceremony.
Abstract. Incremental migration removes the risky cutover, which is its entire justification and a real one. It also removes the moment at which anyone verifies the system as a whole, declares the old guarantees ended, and names who owns the new ones. This paper argues that the missing moment has costs, and that they are paid quietly: invariants that were only ever true end to end, ownership that never transfers, and an old path that is never decommissioned because no date was ever set for it. This is a draft. The mechanism is stated, the evidence is thin, and the objection in Section 5 is unanswered. Confidence 0.55.
## 1. The claim A strangler-fig migration proceeds route by route. Each move is small, reversible, and independently verified, which is why the pattern works and why it has largely replaced the big-bang rewrite. Nothing in this paper disputes that. What the pattern does not produce is a point in time at which someone says: the system is now the new system, its guarantees are these, and this person owns them. In a big-bang cutover that moment is unavoidable and expensive, and its expense is what buys the verification. Incremental migration makes the moment optional, and optional organisational moments do not happen. ## 2. What decays in the gap **2.1. Invariants that were only ever end-to-end.** Consider a property like "every accepted order appears in exactly one settlement batch". In the old system that held because one process owned both sides. Route-level verification checks that the new order path matches the old order path and that the new settlement path matches the old settlement path. It does not check the property that spans them, because that property was never a route. 
  old system            invariant held here
  [ orders -- settle ]  <---- one owner, one process

  during migration
  [ orders ] --> new    each route verified
  [ settle ] --> old    against its own old half
        ^
        +-- nobody verifies the span

  after
  [ orders -- settle ]  invariant assumed, not checked
Figure 1. Route-level equivalence does not compose into system-level equivalence, and the gap is invisible while both halves pass.
**2.2. Ownership that never transfers.** The old system has an owner. The new one has a migration team. When the last route moves, the migration team disbands and ownership arrives at whoever is nearest, usually by accident, usually discovered during the first incident. **2.3. The old path that never dies.** A residual route left for a "long tail" client, a batch job, a reconciliation script. It has no owner and no decommissioning date. It accumulates the property of being the thing nobody understands, which is where 5.7 picks up. **2.4. The documentation describes neither system.** During migration every document is provisional. Provisional documents are not maintained, and the migration is long enough that the habit of not maintaining them outlives it.
Route-level correctnessverified
Spanning invariantsassumed
Ownershipimplicit
Old path retiredpending

Figure 2. What incremental migration verifies well against what it leaves open. An impression rather than a finding; this is a draft, and the bars are part of what Section 6 is asking about.

## 3. Why "it is done when the last route moves" is not enough Because the last route is chosen by difficulty, not by importance. Migrations move the easy traffic first, which means the final routes are the ones with the most unusual requirements and the least understood behaviour, and the project reaches its lowest morale and highest cost at exactly the point where the remaining work is hardest to verify. "Ninety percent migrated" is a statement about routes and almost never a statement about risk. ## 4. The proposal: identity by declaration The remedy I am proposing, and the part I am least sure of, is to reintroduce the moment deliberately, without reintroducing the risky cutover that the pattern exists to avoid. It is a document and a date, not a deployment.
§Declaration
1Name the spanning invariants at the start, before the first route moves, and build a check for each that runs against the live system throughout the migration rather than against either half.
2Publish a route ledger: every route, its state, its verification, its owner. One page. It is the only honest answer to "how far along are we".
3Declare a date on which the old guarantees end, and name the person to whom the new ones transfer. This is the moment. It costs a meeting.
4Set the decommissioning date before the migration starts, with an owner, and treat slipping it as a decision that needs a reason rather than as a default.
## 5. The strongest objection, unanswered
The missing moment may not matter, and wanting one may be aesthetic rather than operational. This is the objection that keeps the paper at 0.55 and in draft. Gradualism's entire benefit is that there is no discontinuity, and asking for a declaration could be nostalgia for the ceremony of a cutover dressed up as a risk argument. Every cost I list in Section 2 has an alternative explanation that has nothing to do with the missing moment: spanning invariants decay in systems that never migrate at all, ownership drifts under reorganisation regardless (see 5.12), and old code survives for ordinary reasons of priority. Nothing here separates the migration-specific effect from that background rate, which leaves a plausible mechanism standing on no evidence. That is what a draft is, and it is why the confidence is where it is.
## 6. Open questions Stated plainly, because this document is not finished and pretending otherwise would be the failure mode the rest of the site is about. - Is the spanning-invariant decay measurable, and does it differ from the background rate in comparable systems that did not migrate? - Does the declaration in Section 4 change behaviour, or does it become a ceremony that is performed and ignored? - Is there a version of the route ledger that survives the migration and becomes the ownership document, or does it die with the project? - Does this apply below some size? A four-route migration probably needs none of this, and I do not know where the threshold is. ## 7. What this paper does not claim Big-bang rewrites are worse, and the pattern this paper criticises is the correct default. Incremental migrations do not fail; most of the ones I have seen succeeded on their own terms. The claim is only about what they leave behind, and at 0.55 it is a claim I would not want quoted without its confidence value attached to it. --- 5.10 Vector Databases Are a Fad --- url: https://mosthofaimran.com/papers/vector-db-fad/ state: retracted confidence: n/a revised: 2026-08-14 retires: NONE STATED. This entry is not an argument.
Retracted in full on 2025-11-14. The text below is the original argument, retained rather than deleted, per Section 2.2 of the index draft. It is wrong. Section 5 of this page records what failed and what survived. Do not quote any part of Sections 1 to 4 without this notice attached.
## 1. The claim, as originally published Vector search is a feature, not a product. The index structures involved are published algorithms with open implementations, the storage layer is a solved problem, and the query patterns are narrow. General-purpose databases will absorb the capability within two years, at which point a separate stateful system exists only to serve a workload its neighbour can already handle. Standing up a second database with its own operational model, backup story, upgrade cadence and on-call knowledge is a cost that this workload does not justify. ## 2. The supporting argument, as originally published Three points were offered. First, that approximate nearest neighbour search is a well-understood problem with published algorithms, so no vendor holds a durable advantage. Second, that the operational surface of a dedicated store is the real cost and it is paid whether or not the workload grows, which is the argument in 5.14 applied to a storage layer. Third, that the corpora most teams actually hold are small enough that brute-force or lightly indexed search inside an existing database is sufficient, and that the category was being sized by the largest deployments rather than the median one. ## 3. What the paper predicted That within two years, most production retrieval workloads would be served by vector extensions to databases teams already ran, and the standalone category would consolidate to a small number of vendors serving genuinely large deployments. ## 4. What was already weak at publication The paper's own hedge was that a sufficiently large corpus with strict latency targets might justify a specialised store. That hedge was stated in one sentence and not developed, which in hindsight was where the whole argument was. ## 5. What actually failed This section was added at retraction. It is not struck through, because it is the only part of this page that is currently believed. **5.1. The index was not a commodity.** The paper treated approximate nearest neighbour search as a solved algorithmic problem and therefore as undifferentiated. What matters in production is not the core algorithm but the properties around it: filtered search that stays accurate when a predicate removes most of the corpus, incremental index maintenance under continuous writes, quantisation that trades memory for recall in a controllable way, and predictable behaviour at the recall and latency point a product actually needs. These are engineering properties, they differ substantially between implementations, and the paper dismissed the entire category of them in a subordinate clause. **5.2. The operational story matured faster than predicted.** The argument rested on a second stateful system being expensive to run. Managed offerings, sensible defaults and better operational tooling arrived inside the prediction window and reduced that cost enough to change the decision. **5.3. The prediction was directionally right and useless.** General-purpose databases did gain credible vector capability, which is the thing the paper said would happen. It did not follow that the specialised systems were a fad, and the paper's confidence came from the first observation while its conclusion depended on the second.
The paper saidWhat was true
The algorithm is published, so the product is undifferentiatedThe algorithm is published. Filtered search accuracy, incremental maintenance and quantisation behaviour are not, and they are what a deployment lives or dies on.
A second stateful system is too expensive to operateIt was, at the time of writing, and stopped being so inside the prediction window.
Therefore the category disappearsGeneral-purpose databases did gain the capability. The category did not disappear. The conclusion never followed from the premises.
## 6. What survived, stated narrowly For corpora below roughly the size where index structure starts dominating latency, a vector extension to a database the team already operates is usually the right first choice, and the reasons are the ordinary ones in 5.14: one fewer system to back up, upgrade and page about. That is a much smaller claim than the title, it is not interesting, and it is what I should have written. The larger lesson is recorded in erratum 7.2 and shapes 5.8: a confident prediction about which technology category will disappear is a bet on a market, and my evidence was about an algorithm. Those are different things, and the confidence value I published at the time (0.65) was not low enough to reflect that I had substituted one for the other. --- 5.11 What a Postmortem Owes You --- url: https://mosthofaimran.com/papers/postmortem-owes-you/ state: holding confidence: 0.9 revised: 2026-08-14 retires: - An organisation publishing narrative-only postmortems, naming no decision and no owner, that nonetheless shows a falling rate of repeat incidents in the same subsystem over eighteen months. - Evidence that requiring a named decision measurably suppresses incident reporting, so that the cost in disclosure exceeds the gain in correction. - A study showing that action items with an owner and a verification date are completed at the same rate as those without, which would remove the mechanism this paper rests on.
Abstract. The product of a postmortem is a decision. Everything else in the document (the timeline, the graphs, the contributing factors) exists to make that decision legible and to let a reader disagree with it. A postmortem that names no decision has recorded the weather. It reads as diligence, it costs several engineer-days to produce, and the same incident recurs because nothing changed that a future engineer can trip over. Confidence 0.90. Section 5 is the part I cannot resolve: a rule that demands a decision will get some manufactured ones.
## 1. The claim Ask of any postmortem: what is now different? Not "what did we learn", which is unfalsifiable, and not "what will we do", which is a forecast. What is different, today, in a way another engineer would notice without reading this document. If the honest answer is nothing, the incident is unresolved regardless of how good the write-up is. This is uncomfortable because the write-up is often genuinely excellent. Timeline theatre is the most convincing artefact in engineering: precise, chronological, full of real detail, and structurally incapable of changing anything. ## 2. Three failure modes **2.1. The weather report.** A minute-by-minute account, accurate throughout, ending in a paragraph of reflection. It documents that a thing happened. It commits to nothing. It is the most common form and the hardest to criticise, because every individual sentence is true. **2.2. Root-cause singularity.** The search terminates at the first satisfying explanation, usually the last change before the alert. Incidents in systems of any size are conjunctions: a latent defect, a configuration drift, a missing signal, and a human decision made with the information available at the time. Stopping at one of the four means the other three are still armed. **2.3. Action items with no owner, no date and no test.** "Improve monitoring." "Consider adding a circuit breaker." An item that cannot fail cannot be completed. Six months later the list is still open and nobody can say whether that matters. 
  incident
     |
     v
  narrative -------------> archive  (weather report)
     |
     +--> contributing factors
              |
              +--> decision
                      |
                      +--> owner
                      +--> verification date
                             |
                             v
                    something a future
                    engineer trips over
Figure 1. The upper path is the common one. Only the lower path changes the probability of recurrence.
## 3. What the document owes the reader Four obligations. They are ordered by how often they are skipped.
§Obligation
1The decision that changed. Stated as a single sentence a reader can disagree with. "We now retry at the gateway only, and the SDK returns errors." If no decision changed, say that explicitly and say why, which is a legitimate and much rarer outcome than the silence suggests.
2The signal that would have caught it earlier, and whether it exists now. Every incident has a moment where the system knew and nobody was told. Name the metric, the threshold and the destination. If the answer is that no such signal is practical, that is a finding.
3The person who can veto the fix. Not the owner of the action item. The person whose objection would stop it. Naming them converts a silent stall into a visible disagreement, and disagreements can be resolved.
4The date the fix is verified in production. Verified, not merged. A fix that has not been exercised against the failure mode is a hypothesis with a commit hash.
## 4. The part that is culturally expensive A postmortem should name the moment of the missed decision, including where it was missed by the reviewer rather than the author. This is where blamelessness is most often misapplied. Blamelessness means the consequence to the individual is zero. It does not mean the record is vague. A document that will not say "this was approved in review, and the review did not ask about the retry budget" has removed the only detail from which the review process could learn. Erratum 7.3 on this site is that exact case, written about me, and it is the reason I hold this claim at 0.90 rather than lower. I have watched the vague version fail and the specific version work, in the same organisation, eight months apart.
Named decisionrequired
Signal + thresholdrequired
Veto holderrequired
Verify daterequired
Minute-by-minute timelineoptional

Figure 2. The inversion this paper asks for. The optional row is the one most templates make mandatory.

## 5. The strongest objection
Demanding a decision produces manufactured decisions. This is the objection I cannot fully answer. Under a rule that every postmortem must name a change, teams will name a change, and some fraction of those will be theatre: a lint rule, an extra alert nobody will act on, a runbook paragraph. That is worse than honesty, because it consumes the review budget and creates the appearance of correction. The version of the rule I actually believe in is that the document must answer the question, and that "nothing changed, and here is why" must be an acceptable answer that a senior person is willing to sign. Whether that survives contact with a organisation under audit pressure, I do not know. There is also a second objection with force: for genuinely novel failures in an immature system, the correct output really is understanding, and the decision follows a quarter later.
## 6. What this paper does not claim Timelines are the evidence a decision rests on rather than useless detail, and a decision without them is an assertion. Nothing here argues for shorter documents, or for writing one after every incident. The four obligations in Section 3 are not sufficient either. My claim about them is weaker than it may read: a document missing any of the four has, in my experience, failed to change the system it describes, which is not the same as saying that a document containing all four succeeds. --- 5.12 Your Service Boundaries Are an Org Chart --- url: https://mosthofaimran.com/papers/service-boundaries-org-chart/ state: holding confidence: 0.85 revised: 2026-08-14 retires: - A system of comparable size whose service graph remained materially unchanged across a reporting-line reorganisation, sustained for four quarters, with no deliberate effort to hold the architecture in place. - Evidence that distributed and asynchronous working has flattened the communication cost gradient enough that team boundaries no longer predict interface boundaries. - A demonstration that the correlation runs the other way in practice, with organisations reliably reshaping their reporting lines to match an architecture chosen first, at a rate high enough to make the inverse manoeuvre in Section 4 the normal case rather than the rare one.
Abstract. Conway's law is usually quoted as an observation about other people's systems. It is more useful as a design constraint with a predictable timescale. Across three reorganisations I watched the service graph converge on the reporting graph within roughly two quarters, in every case, regardless of what the architecture documents said. The practical consequence is that a boundary you want must be paid for in organisational structure, not in diagrams. Confidence 0.85. Three cases in one company is not a sample, and the causal direction is not settled. Both are Section 5.
## 1. The claim An interface hardens where communication is expensive. That is the mechanism in one sentence, and everything else follows from it. Inside a team, changing a function signature costs a conversation. Across a team boundary it costs a ticket, a sprint boundary, a compatibility window and, if the teams report to different managers, a negotiation about priority. Engineers are efficient. They route around expense. So the seams in the codebase migrate, over months, to sit exactly where the organisational expense is, and the architecture document becomes a description of a system that no longer exists. The observation is Conway's. What I am adding is that the timescale is short enough to be useful, and that the direction of causation is asymmetric in a way that gives you a lever. ## 2. What the convergence looks like The pattern is subtler than services getting renamed, and it shows up in four places before it shows up in the deployment topology.
Where it shows firstSymptom
Shared modulesA library owned by two teams grows a seam down the middle. Both halves are still in one repository, both are still deployed together, and no change ever crosses the seam.
API versioningEndpoints crossing a team boundary acquire versions and deprecation policies. Endpoints inside one team keep changing in place, and nothing breaks.
DataA table two teams write to becomes a table one team writes to and the other reads through a view, then through an event, then through a copy.
On-callThe rotation splits before the service does. The rotation boundary is the most honest architecture diagram an organisation produces, because it is the one with consequences attached.
 
  quarter 0        reporting lines
                   A ---- B        C

  architecture     [ ingest -- enrich -- serve ]
                        one service, three modules

  quarter 2        reporting lines
                   A       B ---- C

  architecture     [ ingest ] -> [ enrich -- serve ]
                        seam appeared where A left
Figure 1. The reorganisation happened in quarter 0. The seam was visible in the code by quarter 2, and nobody proposed it in a document.
## 3. Why the architecture document loses Three reasons, and none of them involve anyone behaving badly. **3.1. The document has no enforcement surface.** A boundary that is not enforced by a compiler, a repository permission, a deployment unit or an on-call rotation is a suggestion. Suggestions decay at the rate of staff turnover. **3.2. Local incentives are correct and global ones are diffuse.** An engineer avoiding a cross-team negotiation to ship this week is making the right call for this week. The architecture erodes one correct local decision at a time, which is why it is invisible in review. **3.3. Reorganisations are faster than refactors.** A reporting line changes in an afternoon. A service boundary changes over two quarters. The organisation will always be ahead, so the code is always converging on a target that has already moved. ## 4. Using it rather than lamenting it The lever is that the causation is asymmetric. Organisation shapes architecture reliably and quickly. Architecture shapes organisation weakly and slowly. So the manoeuvre is to choose the boundary you want and then move the people, which is a management action rather than an engineering one. - **Before designing the service graph, draw the team graph you can actually get.** If you cannot get it, design for the one you have. A boundary you cannot staff is a boundary you will maintain by hand until you stop. - **Never let two teams own one deployment unit.** It will grow a seam anyway, and the seam will be placed by expedience rather than by design. Split it deliberately or merge the ownership. - **Watch the on-call rotation as the leading indicator.** When someone proposes splitting a rotation, the service split is roughly two quarters away whether or not anyone has written it down. - **Price a desired boundary in headcount.** "We want ingest isolated" translates to "we need someone to own ingest". If that role does not exist, the isolation will not survive its first deadline. - **When a reorganisation is announced, schedule the architecture review inside the same quarter.** The drift is going to happen. The choice is whether it is designed or discovered.
Reporting line changedays
Rotation splitweeks
Interface hardens~1 quarter
Deployment unit splits~2 quarters

Figure 2. The sequence I have observed three times. The ordering is the claim; the durations are approximate and drawn from three cases, which is not a sample.

## 5. The strongest objection
Three reorganisations in one organisational culture is an anecdote, and the direction of causation is not established. Both halves of that are fair. My three cases share an employer, a market and a hiring pipeline, so what I may have observed is a property of one company rather than of organisations. And the causal claim is genuinely underdetermined: it is equally consistent with the evidence that managers reorganise in anticipation of architectural pressure they can already see, which would make the reporting change a symptom rather than a cause and would invert the advice in Section 4 entirely. I hold 0.85 because the predictive value has been high for me, not because the mechanism is settled. A well-constructed study across several organisations would move this number in either direction, and I would rather have it moved than defended.
## 6. What this paper does not claim Boundaries should not follow the org chart as a matter of preference. Sometimes the right architecture cuts against it, and then the work is to change the org chart, which is the whole point of Section 4. The stronger argument that microservices are themselves an organisational artefact is a separate paper and I am not making it here. The effect is also not inevitable. Resisting it works. It costs continuous effort, and the failure I keep watching is that the effort is assumed rather than budgeted. --- 5.13 The Dashboard Nobody Opens --- url: https://mosthofaimran.com/papers/dashboard-nobody-opens/ state: holding confidence: 0.75 revised: 2026-08-14 retires: - A team whose dashboards are built from emitted signals rather than from operator questions, and whose new on-call engineers nonetheless answer the five triage questions in Section 3 within ninety seconds, without assistance, on an incident they have not seen before. - Evidence that question-driven dashboards measurably slow diagnosis of novel failure modes, so that the breadth they discard costs more than the speed they buy. - Query-side tooling that makes ad-hoc exploration fast enough that pre-built views stop mattering, at which point this paper is about a tool generation rather than about a practice.
Abstract. Most dashboards are assembled from the signals that were easy to emit rather than from the questions an operator asks under pressure. The result is a wall of accurate panels that answers none of the five things anyone actually needs to know at 03:00, and which is therefore not opened during the incident it was built for. The fix is to invert the construction order: enumerate the questions first, build one view per question, and delete anything that answers none of them. Confidence 0.75. Section 5 carries a cost I cannot price: views built from questions over-fit the failures already survived.
## 1. The claim Observability is usually built bottom-up. A library emits what it can, a platform team collects it, and a dashboard is assembled from what arrived. Every panel on that dashboard is true. The collection is complete, the queries are correct, the alerting is wired. And during the incident, the on-call engineer opens it, looks at it for four seconds, and goes to the logs. That four seconds is the measurement that matters and nobody records it. A dashboard's value is the number of triage questions it answers without a follow-up query, rather than the number of signals it displays. ## 2. Why bottom-up construction fails The signals that are cheapest to emit are properties of components. The questions that matter are properties of the interaction between components and users. These are different objects, and no amount of correct aggregation turns one into the other. 
  emitted (cheap)          asked (expensive)

  cpu, memory, disk        is it me or them?
  request rate             all users or some?
  error count              did it start with a deploy?
  p50 / p95 / p99          is it getting worse?
  queue depth              what breaks next?
Figure 1. The left column is what most dashboards contain. The right column is what the person looking at them is trying to find out.
The right-hand column is answerable from the left-hand column, and that is exactly the problem: it is answerable by an engineer doing arithmetic and correlation in their head, under time pressure, with a pager going off. The dashboard has delegated the hard part. ## 3. The five questions These are the questions I have watched people ask, in this order, in every triage I have been part of. Any dashboard that does not answer them is decoration.
§QuestionWhat the panel must show
1Is it us or a dependency?Error rate split by originating layer, with dependency errors attributed to the dependency rather than counted as ours.
2Everyone or some?The same failure rate broken by tenant, region and client version, on one view. This is the panel that most often does not exist, because the dimension was never emitted.
3Did it start with a change?Deploy and configuration-change markers on the same time axis as the failure. A separate deploy dashboard does not count; correlation across two browser tabs is the work being avoided.
4Is it getting worse?Rate of change, not level. A flat 4% error rate and a 4% rate doubling every two minutes look identical on a gauge and require opposite responses.
5What fails next?Saturation of the things that will queue: pools, buffers, budgets. Including the retry budget, which is the signal from 5.6 that almost nobody emits.
## 4. The construction rule, and the test **4.1. One view per question, named after the question.** Not "Service Health". The panel title is the interrogative sentence. This reads as unserious and it is the single change with the largest effect, because it makes an unanswerable question visibly unanswered rather than quietly absent. **4.2. If a signal answers no question, it is not on a dashboard.** Keep emitting it, keep it queryable, keep it out of the view. Storage is cheap and attention is not. **4.3. Instrument the dashboards themselves.** Most observability stacks can report which views were opened and when. A dashboard with no opens in ninety days is either wrong or redundant, and in both cases deleting it is an improvement. This is the only part of the paper with a direct measurement attached, and it is the part teams resist most. **4.4. The ninety-second test.** Take an engineer who has been on-call for under a month. Give them a real past incident. They should answer all five questions in ninety seconds without asking anyone. If they cannot, the gap is a work item with an owner, and the dashboard is the deliverable, not the diagnosis.
Q1 us or themcommon
Q2 all or somerare
Q3 change markerssometimes
Q4 rate of changerare
Q5 saturationpartial

Figure 2. How often each question is already answered, ranked from the systems I have reviewed. A ranking from memory, not a survey.

## 5. The strongest objection
Question-driven dashboards over-fit the failures you have already had. This is the real cost and I do not have a clean answer to it. The five questions are derived from incidents I lived through, which means a view built to answer them is optimised for a distribution of failures drawn from the past. The expensive incidents are the novel ones, and novel incidents are diagnosed by exploration: broad, unfiltered, looking at things nobody thought to put on a panel. A practice that deletes unopened views is deleting exactly the breadth that exploration needs. The reconciliation I use is that exploration should happen in the query interface rather than on dashboards, and that dashboards are for triage only. That splits the tools cleanly in theory. In practice teams with weak query tooling use dashboards for both, and for them this paper's advice would make things worse. That is most of the gap from 0.75 to 0.9.
## 6. What this paper does not claim Component metrics are necessary. Questions 1 and 5 are answered from them, and a system that does not emit them cannot answer either. The five questions are not a complete set; they are what has been sufficient for first-response triage in the systems I have run. Dashboards do not cause slow incident response. The claim is smaller and more specific: a dashboard assembled from what was easy to emit is answering a question nobody asked, and that cost stays invisible because almost nobody measures whether it was opened. --- 5.14 Kubernetes for a Bicycle --- url: https://mosthofaimran.com/papers/kubernetes-for-a-bicycle/ state: holding confidence: 0.7 revised: 2026-08-14 retires: - A team of five or fewer engineers, with no dedicated platform role, running a full orchestration stack for eighteen months while spending less total time on the platform than on the product, measured rather than recalled. - Managed orchestration reaching a point where the fixed costs listed in Section 3 are genuinely absorbed by the provider, including upgrade cadence, network policy and on-call knowledge, at which point the break-even in Section 4 moves far enough to invert the advice. - Evidence that starting simple and migrating later costs more in aggregate than starting heavy, which is the inverse of the assumption this paper rests on and the one I would most like to see tested.
Abstract. Operational weight has a fixed cost and a variable cost. The fixed cost is paid whether or not the load arrives, it is mostly not the cluster, and it is routinely estimated at a fraction of its real size. I have made this mistake twice, in both cases having costed the infrastructure and not the practice that has to surround it. The claim is that the sizing question is "how many operators do we have" rather than "how many nodes do we need", not that orchestration is wrong. Confidence 0.70. Section 5 has two objections I cannot weigh against each other, and I have been burned by the second one as well.
## 1. The claim Choosing an operational model is choosing a fixed monthly cost in engineer-hours. That cost is incurred on the quiet weeks as well as the busy ones, and it is paid by the same people who are supposed to be building the product. The bicycle in the title is the load. The failure is that the machinery has a minimum operating crew rather than that it is bad machinery, and a team that cannot staff that crew ends up operating it badly, which is worse than operating something smaller well. Both times I got this wrong I had done the arithmetic. Both times the arithmetic was about the cluster. The cluster was never the expensive part. ## 2. Fixed and variable, drawn honestly 
  effort
    ^
    |  heavy stack
    |  ------------------------------ fixed floor
    |                            /
    |                        /
    |  light stack     /
    |  ____________/
    +--------------------------------> load
                   ^
                   break-even, which is
                   further right than it looks
Figure 1. The heavy stack wins eventually. The question is whether your load reaches the crossing point before your team runs out of the hours the floor consumes.
The shape is uncontroversial. The mistake is in placing the floor, and the floor is placed by what is on the list in Section 3 rather than by the orchestrator itself. ## 3. What the fixed cost is actually made of
CostWhy it is underestimated
Upgrade cadenceThe platform has a support window measured in months. Somebody owns that treadmill permanently, and it does not scale down when the product is quiet.
Network policy and identityThe default posture is usually open, so a real deployment needs policy, service identity and secret distribution designed rather than adopted. This is where most of the first quarter goes.
The build and release pathRegistries, image provenance, signing, promotion between environments. All defensible, all work, none of it visible in the decision that started it.
Observability of the platform itselfYou now have two systems to watch: the product, and the thing running it. Both page.
On-call knowledgeThe largest and least tracked. Every engineer in the rotation needs a working model of the platform's failure modes, which is a training cost paid per person and again on every hire.
Bus factor on the platformIn a small team this is usually one person, and the fixed cost is invisible until that person takes leave.
Cluster itselfcosted
Release pathpartly
Policy and identitymissed
On-call knowledgemissed

Figure 2. What I costed against what it cost, both times. Drawn from two cases in hindsight, which is not a sample.

## 4. The ladder, and the triggers for climbing it The useful discipline is naming, in advance, the observation that will move you up a rung, rather than choosing correctly on day one. Without a named trigger the decision is made by whoever is most enthusiastic in the room.
RungMove up when
One machine, one artifact, a service managerA single machine's failure becomes an unacceptable outage, or deploys need to be zero-downtime.
Two or three machines, a load balancer, deploys by scriptYou are hand-placing more than about ten distinct workloads, or bin-packing has become a spreadsheet.
Managed container runtimeYou need scheduling policy the platform cannot express: affinity, priority classes, custom autoscaling, per-tenant isolation.
Full orchestrationYou have a platform owner. Not a volunteer. A role.
The last trigger is the whole paper compressed into one line. If the answer to "who owns the platform" is a name plus the word "also", the rung is too high. ## 5. The strongest objection
The ecosystem argument, and it is strong. Orchestration is the industry default. Choosing something smaller means every vendor integration, every hire, every piece of tooling and every published runbook is written for a platform you are not running, and that ongoing translation cost is real and compounding. It also affects recruitment in a way that is difficult to price and easy to dismiss. Against that, my counter-argument is only that the translation cost is visible while the operational floor is not, and that people systematically overweight visible costs. I believe that, but I cannot demonstrate it, and it is most of the distance from 0.70 to a higher number. There is a second objection with force: premature simplicity is also a failure mode, and migrating a running system up a rung under load is more expensive than starting one rung high. I have watched that go badly too. Nobody has given me a principled way to weigh the two regrets against each other, so what follows Section 4 is advice and not analysis.
## 6. What this paper does not claim Orchestration is not over-engineering. At sufficient scale it is the cheapest option available and the fixed cost is recovered without anyone noticing. Managed offerings do help; they absorb less of the list in Section 3 than their marketing implies, which is a complaint about the marketing rather than about the product. The ladder in Section 4 is one sensible ordering and not the only one. Nor have I fully learned this. Two occurrences is a pattern, not a cure, and I would not bet against a third. --- 5.15 SOC 2 in 120 Days --- url: https://mosthofaimran.com/papers/soc2-120-days/ state: draft confidence: 0.65 revised: 2026-08-31 retires: - A SOC 2 Type II report covering an observation window shorter than three months, issued by a firm a mid-market enterprise buyer's security team accepted without qualification. The arithmetic in Section 3 is the only hard constraint this paper claims, and that would remove it. - Two comparable companies reaching the same report on the same schedule at similar total cost, one using a compliance automation platform and one not. That would make Section 4 a vendor preference rather than a scheduling argument. - Evidence that the exceptions section of a Type II report is not read during enterprise procurement, which would make the failure mode in Section 7 cosmetic. - My own first attempt at this schedule slipping for a reason not listed in Section 5. The plan is falsified by the thing it did not anticipate, not by the things it did.
Abstract. A SOC 2 Type II report attests that controls operated over a period, and the shortest period most audit firms will attest to is three months. That single fact fixes the schedule: get the controls operating by day 30, let the window run to day 120, and accept that the report itself arrives after it. Compliance automation platforms of the Vanta class collapse evidence collection, which used to be the line item that ate the calendar. They do not produce controls, they do not decide scope and they cannot shorten the window. Confidence 0.65. I have operated platforms to ISO 27001, GDPR and Bank Negara Malaysia RMiT, carried them through the security reviews banking and telecom clients run before contract, and worked on compliance alongside partner organisations holding signed Type II reports. I have not owned such a programme end to end. Section 8 says what that distinction is worth.
## 1. The claim One hundred and twenty days from a standing start to the close of a SOC 2 Type II observation window, for the Security category, over one product in one cloud account. The work is not hard. The work is arithmetic plus procurement, and almost every plan that slips does so on procurement. Three things this does not claim. **It does not claim a report in your hand on day 120.** The window closes on day 120. Fieldwork overlaps the last month of it, and the signed report follows the close by two to four weeks depending on the firm. Anyone selling you a report on day 120 is selling you a Type I and hoping you do not read the cover page. **It does not claim ISO 27001 on the same schedule.** ISO 27001 certifies a management system through a Stage 1 and a Stage 2 audit against an accredited certification body. It is a different object with a different calendar, and the two schedules nest rather than compete.
ClaimScopeEnvelope
Two quartersISO 27001 and SOC 2 together, readiness assessment starting in week one, control owners named rather than volunteered.~180 days
This paperSOC 2 alone. Security only, one product, one cloud account, one legal entity.120 days
The outer number is the one I would put in writing to an employer, because it is the one that covers both frameworks and leaves room for the certification body's calendar. The inner number is what the SOC 2 half of that envelope actually costs, and it is stated separately here so that the outer number can be checked rather than taken. A schedule you cannot decompose is a schedule nobody can hold you to. **It does not claim the controls are good.** It claims they are attested. A report says an auditor tested a control and describes what they found. It does not say the control was the right one, and Section 7 is about the distance between those two sentences. ## 2. What the attestation actually is Precision here saves a quarter, so it is worth four paragraphs. A SOC 2 report is written by a licensed CPA firm under AICPA attestation standards. It is not a certificate and there is no certifying body. There is a firm, an opinion and a report with your controls printed in it. The report is scoped to Trust Services Criteria. **Security** is mandatory and is the common criteria set. **Availability**, **Confidentiality**, **Processing Integrity** and **Privacy** are elective, and each one you elect adds controls you must operate and evidence for the life of the company.
ReportWhat it saysWhat it costs you in calendar
Type IThe controls were designed appropriately as at one date.Days, once the controls exist. No window.
Type IIThe controls operated effectively across a stated period, tested by sampling that period.The period. Three months at the floor, twelve at the steady state.
The buyer wants Type II. A Type I buys you a conversation and a line in a questionnaire. It does not close an enterprise security review, because the reviewer's question is not "did you configure this" but "did you keep doing it". ## 3. The arithmetic
LONG POLES: AUDIT FIRM CAPACITY · THE RISK ASSESSMENT Window opens. Nothing after this is backfill. Open it before the controls are perfect. readiness compressible observation window · 90 days elapsed time, not work. Nothing shortens it. report issued ~25 days day 030120~145 fieldwork runs inside the window, not after it
Figure 1. The only irreducible segment is the ninety days in the middle, and it is the majority of the schedule. Everything left of day 30 is procurement and configuration, both compressible. Everything right of day 120 belongs to the audit firm. Most plans that slip do so on the two long poles, and neither of them is engineering.
The window is the schedule. Once that is understood, the plan writes itself backwards: the only question that matters in week one is what has to be true on day 30, because the day 30 date is the one that decides the day 120 date. This inverts the instinct. The instinct is to get the controls right and then start the clock. The correct move is to start the clock at the earliest defensible moment and repair in flight, because a control operating imperfectly for ninety days produces a report with an exception in it, and a perfect control operating for thirty days produces no report at all. ## 4. What the platform does, and what it does not Compliance automation platforms are the reason the number is 120 rather than 300. They are also routinely misread as doing the whole job, so both columns matter.
The platform does thisThe platform does not do this
Connects to the cloud accounts, identity provider, HR system, endpoint agent, code host and ticket tracker, and reads their state continuously.Decide what is in scope. Scope is a commercial decision about what you are willing to be audited on, and it belongs to whoever signs the contracts.
Turns each control into a test that passes or fails today, with a timestamp and a history.Make a failing control pass. A dashboard reporting an unencrypted volume has done its entire job.
Ships policy templates a small company can adopt in an afternoon instead of a month.Make an adopted policy true. A policy nobody follows is an exception waiting to be written into your report.
Tracks training, policy acknowledgement, background checks and onboarding per person.Perform your access reviews. Somebody still has to read the list and remove the people who should not be on it.
Hands the auditor a portal with the evidence already assembled. This is the quarter of the project that used to be spent taking screenshots.Replace the auditor. The opinion is signed by a CPA firm, and the platform is not one.
Keeps a vendor register and chases subprocessor documentation.Assess the vendor. Collecting a report is not reading it.
Vanta, Drata, Secureframe, Sprinto and the open implementations differ on integration coverage, on how much of the policy set is opinionated, and on whether the auditor relationship is bundled. They do not differ on the line above: all of them automate evidence, none of them automate control. The choice matters less than the date you make it. Any of them, connected on day 5, beats the best of them connected on day 40. ## 5. The schedule
DaysWorkLong pole
0 to 10Scope. Which product, which cloud accounts, which legal entity, which criteria. Security only unless a signed contract already names another category. Control owners named by person, with calendar time allocated.
0 to 10Engage the audit firm. Book the fieldwork dates before the window opens.Yes
5 to 20Connect the platform to everything it can reach. Read the failing list without flinching. Most of it is configuration and most of the configuration is a day's work each.
10 to 30Close the technical gaps. Multi-factor everywhere, encryption at rest and in transit, log retention set to a number you can defend, backups restored rather than configured, production access reduced to the people who need it.
10 to 30Close the human gaps. Policies adopted and acknowledged, training assigned, background checks run, onboarding and offboarding that produce a ticket rather than a memory.
20 to 30Risk assessment and vendor register. Nobody automates these two and auditors read both closely.Yes
30Window opens. From this date forward nothing is backfill.
30 to 120Operate. One access review inside the window, one restore actually executed, one incident response exercise, change management on every merge that reaches production.
90 to 120Fieldwork. The auditor asks for populations, samples them, and finds the thing you forgot. Answer in days, not weeks.
120Window closes. Report follows.
Two rows carry the long pole marker and neither is technical. Audit firm capacity is an external dependency you do not control, and the risk assessment is the deliverable that cannot be produced by an integration because it is an argument about your own business. ## 6. Rules Normative language, applied to this programme. **6.1.** The observation window MUST open before the controls are perfect. Ninety days of an imperfect control produces a report with an exception. Thirty days of a perfect one produces nothing. **6.2.** The first report MUST cover Security only, unless a signed contract already names another category. Categories are cheap to add to the second report and expensive to carry through the first. **6.3.** Every control MUST have an owner who is a person. A control owned by "engineering" is owned by nobody, and that is discovered in week eleven. **6.4.** Evidence SHOULD be a byproduct of work that would happen anyway. An access review that exists only for the auditor will not survive its second year, and the second year is where the exceptions appear. **6.5.** The audit firm MUST be engaged before the window opens rather than before it closes. Their capacity is the schedule's only dependency you cannot buy your way out of late. **6.6.** A failing control MUST NOT be closed by narrowing its description until it passes. That is the single move that turns an attestation into a lie. The exceptions section exists so that you never have to make it. **6.7.** The platform's readiness percentage SHOULD be read as a coverage metric rather than a security metric. It measures how much of the estate the platform can see, and it goes down when you connect something new, which is the correct direction. **6.8.** Anything the platform cannot reach MUST be written down. The unmanaged laptop, the contractor's own machine, the legacy virtual machine nobody logs into. An unlisted system is undiscovered rather than out of scope. ## 7. The named failure mode **Evidence complete, control absent.** The dashboard reads green because every automated test passes, and every automated test is a test of configuration. Configuration is the part that automates. The controls that fail in practice are the human ones: an access review performed by clicking approve on a list nobody read, an incident response plan exercised once in week three and never again, a vendor register that stopped being updated in month two. The tell is timestamps. Work that actually happened is spread across the window. Work that was assembled for the auditor shares a date. A reviewer who has read a hundred of these reports checks the spread before they check the content, and so should you, monthly, while there is still time to fix it. This is the same failure as the one in paper 5.13, arriving through a different door. A measurement that nobody acts on is not a measurement, and a control that produces evidence without producing a decision is not a control. ## 8. What I have not done I have built and operated platforms under ISO 27001, GDPR and Bank Negara Malaysia RMiT requirements for banking and telecom clients, and I have taken those systems through the security reviews and audits those clients run before they will sign. I have also worked on compliance alongside partner organisations that hold signed SOC 2 Type II reports, which is where the specifics in Sections 4 through 7 come from. That work is real and it is the reason this paper is not theory. It is also not the same thing as having owned the programme: my position in it was a partner's, contributing to somebody else's scope, somebody else's control owners and somebody else's opinion letter. The distinction matters enough to print. Working inside a Type II estate teaches you what the evidence looks like, where the sampling bites and which controls fail in month five. Owning the programme end to end teaches you what it costs to be the person who books the firm, sets the scope and signs off the risk assessment. I have the first and I have contributed to the second. That is why the confidence value is 0.65 rather than 0.85, and it moves when the balance changes. This page will say which way and why. --- 5.16 The Only SOC 2 Hack Is Scope --- url: https://mosthofaimran.com/papers/soc2-scope-hack/ state: draft confidence: 0.6 revised: 2026-08-31 retires: - An enterprise security team accepting a compliance claim, on a deal above their standard approval threshold, without reading the report body and its exceptions section. That removes the mechanism the whole paper rests on. - A company reaching a clean Type II with self-hosted database, identity and CI at comparable total engineering cost to one that bought all three managed, measured across two audit cycles rather than one. - Audit firms routinely accepting evidence created after the observation window closed, which would make Section 2 wrong about what sampling catches. - A startup that scoped all five trust categories on its first report and reached it on the same schedule and budget as a Security-only peer.
Abstract. Startups look for the shortcut through SOC 2, and there is one. It lives in two decisions you make in the first fortnight: what you agree to be audited on, and how much infrastructure you own. Own less and there is less to evidence, twice a year, for as long as the company exists. People reach instead for the evidence, the firm or the tooling, and all three of those attempts surface in the same place: the exceptions section, which is the part a buyer's security team actually reads. Confidence 0.60. Lower than the companion paper 5.15, because Section 4 is a cost argument I have watched play out rather than one I have measured.
## 1. The claim A bold startup can compress SOC 2 substantially, and the compression is entirely on the input side. You choose a smaller thing to be audited on, and you choose to own fewer moving parts. Both decisions are made in the first two weeks and neither can be revisited cheaply afterwards. What will not compress is the window, which paper 5.15 covers, and the evidence, which this paper is about. Every attempt I have watched to compress the evidence turned into an attempt to fake it, and every one of those cost more to unwind than doing the work would have cost in the first place. ## 2. Three hacks that do not work **Buy the cheapest opinion.** The report names the firm on its cover. Enterprise security teams keep informal lists of firms whose reports they discount, and the report body prints the tests performed, so a thin audit reads thin to anyone who has read a thick one. The reviewer opens it, counts the tests, and comes back asking you the questions you paid the firm to answer on your behalf. You are in the same meeting you were trying to skip, three months later and several thousand dollars down. **Backfill the evidence.** A Type II is tested by sampling across the observation window. Sampling is the specific thing that catches backfill, because the sample is drawn from a population you supply and the artifacts carry their own dates. Twelve access review records created in the same afternoon is the easiest pattern in the entire discipline to spot. And once an auditor suspects fabricated evidence, they stop testing your controls and start testing you. That is a much longer engagement, it involves your lawyers, and it does not end with a report. **Claim the status without the report.** There is no such thing as being SOC 2 certified. There is a report or there is not. The questionnaire asks for the PDF, the security review asks for the bridge letter covering the gap since the window closed, and a badge on a marketing site answers neither. 
   compressible                        not compressible
   ------------                        ----------------
   trust categories                    the observation window
   systems in scope                    the sampling method
   entities in scope                   the auditor's independence
   infrastructure you own              the exceptions section
   number of vendors                   whether the buyer reads it
        |                                      |
        v                                      v
   decided in week one              discovered in month five
The left column is a set of decisions. The right column is a set of facts. Most failed programmes spent their effort on the right.
## 3. The scope decisions that actually compress the work
DecisionThe compressing choice, and what it costs
Trust categoriesSecurity only. Availability is the one people add reflexively, and it commits you to uptime targets you must then meet and evidence. Add it on the second report, when a buyer has asked for it by name.
SystemsOne production cloud account, one region, unless a contract says otherwise. The demo account somebody spun up in 2024 and forgot is still in scope. You have simply not found it yet, and the auditor's discovery step is designed to.
EntitiesThe single legal entity that signs customer contracts. A group structure adopted for tax reasons does not have to be adopted for audit reasons.
PeopleEveryone with production access, contractors included. Companies push hardest on this row and it does not move. The only lever that works is taking access away from people who do not need it, which is a fortnight of awkward conversations and then it is done.
ProductsThe product being sold. An internal tool is separable only if it shares no data path with production, and it almost always shares one through the same database credentials.
EndpointsOne operating system, one device management tool, chosen before the twentieth laptop. After the twentieth it stops being a decision and becomes a migration.
VendorsEvery subprocessor touching customer data goes in the register and is reviewed. Fewer vendors is a shorter register, and this is the only lever on that row.
TimeOpen the window early at whatever quality you have. See 5.15 section 6.1. An exception in the report is survivable; a report that does not exist is not.
## 4. Buy the boring thing This is the decision with the longest tail and it is usually made for the wrong reason. Every self-hosted component is a control surface you own forever: patching, backup, access management, monitoring, and the evidence for all four, produced twice a year, by people who would rather be building the product. The managed equivalent moves most of that to a subprocessor whose own report you file once and reference thereafter. The comparison teams actually run is the monthly invoice against the cost of an instance. That comparison is wrong by the entire audit programme. The honest version adds two audit cycles a year, for as long as the company exists, plus the hours spent explaining a bespoke component to a reviewer who has never seen one before and is therefore obliged to ask more questions about it. Self-host when the component is your product. Buy it when it is not. A startup that self-hosts its identity provider to save a subscription has purchased a control it will pay for in every security review it ever faces, and paper 5.4 is about exactly this shape of invoice. ## 5. Separation of duties with four engineers You cannot separate duties you do not have enough people to separate, and pretending otherwise is the most common self-inflicted wound in a small company's first audit. Write the awkward sentence into the control description yourself: there are four of us, the founder both requests and approves production access, and here is what we do about it. Then do something about it. A second person reviews every production change, an alert fires on anything that bypasses review, and the founder's own approvals are listed for someone else to look at monthly. Audit firms see companies this size constantly and yours is not the interesting one. What draws attention is a four-person company presenting an org chart that implies forty, because the auditor then has to work out which parts of it are real. An honest compensating control produces a clean finding. An invented org chart produces a question about everything else in the report. ## 6. What it costs, and why there are no numbers here The line items are the audit firm's fee, the compliance platform subscription, a penetration test if a buyer asks for one by name, and engineering time. Engineering time is the largest of the four and the one that never appears in the budget, because it is spent by people who are already paid. No amounts appear on this page. Audit and platform pricing move, they vary by headcount and scope, and this site does not publish a number it cannot source. I would be inventing a figure on a page that spends eight sections arguing against invented figures. ## 7. The named failure mode **The logo without the report.** The badge goes on the website, the badge wins the meeting, and then the buyer's security team asks for the report and the bridge letter. The exceptions section says the access review was performed once during a twelve-month window. Nobody says no. You get a remediation plan, a follow-up call in six weeks and a slot next quarter. For a company with nine months of runway, next quarter is a no delivered politely enough that you keep forecasting the deal. The slower version does more damage. The team concludes that the report is paperwork, runs the next cycle as paperwork, and by year three the controls live entirely inside the compliance dashboard. Then something breaks at two in the morning and the dashboard is where you find out they had stopped being real in year two. ## 8. When not to do this at all If no buyer has asked for it, a SOC 2 is a cost with no revenue behind it and an operating burden that never ends. Start when a named buyer asks, or when the same request keeps arriving from one segment. Seeing a badge on a competitor's homepage is envy with a budget line attached rather than a trigger. The bold move for a startup nobody has asked yet is to say so out loud, publish what it actually does about security, and spend the quarter on the product. You will have one awkward sales call about it. You will also still have a product. --- 5.17 Cost Per Token Is Not Cost --- url: https://mosthofaimran.com/papers/cost-per-token/ state: holding confidence: 0.8 revised: 2026-08-31 retires: - A published comparison across several production workloads showing that ranking models by per-token price predicts their ranking by cost per accepted output. If the cheap proxy tracks the real quantity, the argument for measuring is an argument for wasted effort. - A vendor publishing per-version behavioural diffs specific enough that a team could predict the effect of an upgrade on their own workload without running it. That would make re-evaluation redundant rather than negligent. - Longitudinal evidence that teams selecting models by leaderboard reach the same production outcomes as teams selecting by task-specific eval, once the cost of building the eval is charged against them. - A demonstration that small evals systematically mislead: that a 40-case suite drawn from production traffic picks the wrong model more often than a leaderboard does. The claim here is that a cheap measurement beats a free proxy, and that is a falsifiable comparison.
Abstract. Almost everybody picks a model by looking at two numbers, the price per million tokens and a position on a leaderboard, and neither of those numbers is about your system. The price is the smallest term in what a model actually costs you, and the leaderboard is measuring a different workload than the one you run. The fix is about forty examples pulled out of your own logs, a rubric written on a Tuesday, and the discipline to run the thing again after somebody ships a new version, rather than an evaluation platform or a quarter of work. Confidence 0.80. The arithmetic in Section 2 is solid and the failure mode in Section 6 is one I have watched. The gap from 0.90 is Section 9, where I have no good answer to what happens when the evaluation itself goes stale.
## 1. The claim Somebody posts a thread where a model does something genuinely clever, you read it on a Thursday, and about six weeks later that model is in your architecture diagram without anybody having run a single test against your own traffic. The decision felt technical. It was made from a demo you did not design, on examples you did not choose, by someone who was pleased with the results. This is not an argument that expensive models are a swindle, or that the cheap one is secretly good enough. Either can be true on any given day and neither is knowable from a pricing page. The argument is that the industry has settled on a way of comparing models that is easy to obtain and wrong, and that the correct comparison takes an afternoon to build and then works forever. The whole paper is one substitution. Stop dividing by tokens and start dividing by outputs that a human being was willing to accept. ## 2. The number on the pricing page is the small one What governs your bill is the cost of a piece of finished, usable work rather than the cost of a token, and that has four terms in it: ``` cost per accepted output = (tokens in x price in + tokens out x price out) x attempts before it parses / fraction a human accepts as-is + cost of a person fixing the rest ``` Only the first term is published. The other three are properties of the marriage between one model and your particular workload, and no vendor can tell you what they are because no vendor has seen your workload. So let us do it properly with two models. Everything in the table below is an input I am stating rather than a measurement I am claiming, so you can put your own figures in and watch what happens.
What you are comparingModel AModel B
Input, per million tokens$3.00$0.80
Output, per million tokens$15.00$4.00
Output tokens it actually emits6001,020
Tool calls that come back malformed1.5%8%
Output a human accepts untouched96%82%
The task uses four thousand input tokens. On the pricing page B is three and three quarter times cheaper, which is the sort of gap that ends an argument before it starts. Work it through and B is more verbose, which eats some of the lead, and B fumbles the tool schema more often, which means retries, which eats a bit more. After both of those A costs $0.02132 per completion and B costs $0.00791. B is still cheaper by a factor of 2.69, and if you stop here you buy B and you are wrong, because nothing so far has accounted for a person. Now put the person in. The interesting question is not what review costs, it is how cheap review would have to be before B wins, and that has an answer: ``` break-even review cost = (0.02132 - 0.00791) / (0.18 - 0.04) = $0.0958 ``` At a loaded rate of sixty dollars an hour, nine and a half cents buys you **under six seconds** of somebody's attention. At ninety an hour it buys under four. So if any human being ever looks at the output of this system for longer than about six seconds, the cheap model is the expensive one, and every published number you used to choose it was pointing the other way. At a realistic four-minute review, A comes out at $0.18 per accepted output and B at $0.73. The model that was 3.75 times cheaper is now four times dearer. The whole inversion came out of a column nobody had put in the spreadsheet. 
  what you compared                    what you were billed for
  -----------------                    ------------------------
  price per token             ------>  price per token
                                       retries on malformed output
                                       the extra tokens it rambled
                                       the reviewer's afternoon
                                       the customer who left quietly
       |                                        |
       v                                        v
  published by the vendor              measurable only by you

  share of the bill that is tokens, on the numbers above:
      model A   11.8%
      model B    1.1%
Figure 1: the cheaper the model, the smaller the fraction of your bill it is responsible for, which is the opposite of the intuition that makes people choose it. On these inputs, ninety-nine cents of every dollar model B costs you is spent somewhere no pricing page has ever mentioned.
## 3. Why the published numbers point the wrong way None of this needs anybody to be dishonest. Every number published is a real number. They are simply answers to questions you did not ask, and they fail in a consistent direction. **A leaderboard is an average over somebody else's work.** A model that does well across a broad aggregate has been tuned for a distribution you do not have, and your agent does one narrow thing all day. Aggregate rank does correlate with performance on your task. It correlates far more weakly than the confidence with which people quote it in planning meetings. **Preference rankings are partly a length contest.** When the ranking comes from humans picking a winner between two answers, the winner tends to be the longer one, the better formatted one and the one that sounds more sure of itself. Those three qualities correlate strongly with your bill, because they are output tokens, and much more loosely with being correct. You are reading a chart where "wordier" and "better" have been quietly added together, and then you are paying by the word. **The scores are not comparable across years.** Public benchmarks leak into training data, so a rising score is some mixture of the model getting better and the model having seen the test, in a ratio nobody can give you. You know the direction of the bias. You do not know how big it is, which makes the year-on-year improvement a number you can read but cannot use. **"Better at coding" is a claim about a mean.** Somebody moved an average across a suite the vendor chose. Your workload is one point in that average. It was probably not in the suite at all, and it may have moved the other way. The release note cannot warn you about that because the people writing it genuinely do not know. ## 4. The five ways an engineer talks himself into it The published numbers would do less damage if we held them loosely, and we do not. **The demo you saw once.** Three examples, none of them yours, all of them chosen by somebody who liked how they turned out, and the impression hardens into a standing belief about what the model can do. This is the same machinery as watching somebody refactor a module on video and coming away feeling you have refactored something, which is paper 5.1 in its entirety. **Newer must be better.** Newer is different, which is a smaller claim and a much more useful one. Point releases change refusal behaviour, verbosity, tool-call formatting and the shape of the latency tail. Those four things are precisely what your agent is wired into. **Headroom nobody has measured.** You take the frontier model as insurance against difficulty you have never quantified, which feels responsible and is a standing charge on every request you will ever make for a capability you may never call on. It is buying a van because twice a year you move a sofa. Sometimes correct, worth actually checking. **You remember the good ones.** Failures get retried by hand, muttered at, and forgotten. That is exactly how an acceptance rate falls for a month with nobody able to say when it started. **The vendor's own evaluation suite.** It is a real contribution and a marketing document at the same time, and no organisation has ever published the benchmark on which its product looks worst. Every one of these swaps a cheap signal for a measurement and then holds the answer with the confidence you would be entitled to only if you had measured. That is the untested part, in the strict sense that no experiment was run which could have come out the other way. ## 5. The thing you are really buying is a moving target Here is the part that matters more than the choice and gets a fraction of the attention. The endpoint you are calling is somebody else's running service rather than a file you vendored, and it changes underneath you, sometimes with an announcement and sometimes not. Your agent is built against behaviour rather than an interface, and that is a much softer thing to be standing on. When an interface changes you get an exception and a stack trace. When behaviour changes you get nothing at all, and the system quietly gets worse while every dashboard stays green. **The tool call drifts.** A field that was always there becomes occasionally absent, your parser catches it, the retry succeeds, and nobody is paged because from the outside nothing failed. Cost and latency creep up by a few percent a week. **The answers get longer.** Context fills faster, long conversations start truncating their own history, and the quality falls off in a way that looks exactly like users asking harder questions this month. **It starts declining things it used to do.** Some category of request now gets a polite refusal. Those users do not open tickets. They go away, and you find out from a churn report in the following quarter. **The tail moves and the middle does not.** The p50 is untouched, so the dashboard stays serene. Meanwhile p99 crosses a client timeout and turns into a retry. That is double billing, and if you were careless about idempotency it is also a duplicate side effect in somebody else's system. Every one of those is a quality drop with no alarm attached, which is the exact thing Principle 4.1 exists to forbid. An error gets a ticket and a person. This gets absorbed by your users until they stop being your users. Which means the evaluation is not really a tool for choosing. Choosing happens once. The ground moves continuously, and a suite you can re-run in five minutes is the only thing standing between a silent regression and a very confusing week six weeks later. ## 6. Forty examples and a rubric This is the discipline, and it is deliberately small, because an evaluation programme that needs a quarter to stand up is an evaluation programme that will not exist.
§Do this
6.1Write it before you choose, not after. Written afterwards, it fills up with the cases your chosen model already handles, and you have produced a certificate rather than an instrument.
6.2Take thirty to fifty cases out of real traffic. Not synthetic, not a public benchmark. Sample your logs, and cover the common path, the strange tail and the inputs that are malformed or hostile. If you keep an incident log, every incident in it is a case you already paid for, and those are the most valuable examples you own.
6.3Grade on consequences. "Is this good" will not survive two reviewers disagreeing. "Did it call the right tool with the right arguments", "did it refuse when it should have", "would somebody have to fix this before a customer saw it" are answerable by two people who reach the same answer. Write the rubric before you look at any output.
6.4Record five things, not one. Pass rate, cost per accepted output with all four terms from Section 2, malformed-output rate, refusal rate, and latency at p95 and p99 kept apart, because the tail is where the timeouts live.
6.5Run it against what you are already using, first. This step gets skipped and it is the one that proves the instrument works. Break something deliberately: truncate the system prompt, swap in last quarter's version, take away a tool. If the score does not move, you have built a decoration and it will not notice a regression either.
6.6Only now run the candidates. The decision usually takes about ten minutes at this point, which feels like the work was wasted and is in fact the work paying out.
6.7Pin the version, then re-run on a schedule anyway. Pinning stops the vendor moving under you. It does nothing about your own prompt, tool schema or retrieval corpus moving under the same model, and one of those changes most weeks.
6.8Keep the results with dates on them. One run is a number and tells you very little. A year of runs is a control chart, and a control chart answers the only question you ever actually ask, which is whether this month is different from last.
## 7. What it costs Forty examples, a rubric with four or five yes-or-no questions, and a script that runs the examples and writes one row per run into a file. The harness is an afternoon and you write it once. Each run costs minutes and a rounding error in tokens, against the cost of the engineer who would otherwise spend that afternoon in a meeting about which model feels better. The genuinely expensive part is the examples, except that you do not write examples. You harvest them, out of traffic you already serve and incidents you already survived and mostly wrote up. The work is already done. It is sitting in your logs being useless. ## 8. The named failure mode **The upgrade nobody ran anything against.** A new version ships, it is better on every published measure, somebody bumps the string in the config on a Thursday afternoon, and the diff is one line so it gets approved in about ninety seconds. For three weeks everything is fine, because it mostly is fine. The malformed tool calls go from one in seventy to one in twelve, which your retry logic absorbs. The answers get about forty percent longer, which nobody notices because nobody watches token counts on a Tuesday. Then a long conversation starts truncating its own context, and the agent begins confidently answering questions using the half of the thread it can still see. The support tickets never say "the model regressed". They say the assistant has been odd this week. Somebody opens the prompt and starts adjusting it, because the prompt is the thing we know how to change, and now you are tuning a prompt against a moving baseline with no measurement on either side of it. That is where the week goes, and the week after. A five-minute evaluation run before the config change would have shown a malformed-output rate going from 1.4% to 8.3% and stopped the whole thing on the Thursday. The reason it did not run is that nobody had built it rather than that anybody decided against it, and it was only ever an afternoon. ## 9. The strongest objection, unanswered **A bad evaluation is worse than none, and most first attempts are bad.** Forty cases and a loose rubric still produce a number, and a number invites exactly the confidence this paper spends eight sections arguing you should not have. Suppose the cases are unrepresentative, or the rubric measures something next door to what you care about. You have swapped an uncertainty you knew about for a certainty you have not earned. That trade is the more expensive one, because it stops you looking. Then Goodhart arrives, on schedule. The moment the score is a target, prompts get tuned until the score moves, and the score stops standing in for the thing you wanted. Your instrument decays into precisely the sort of proxy this paper opens by complaining about, and it decays invisibly, from the inside, while the chart continues going up. There is a narrower objection with more force behind it. For a large class of ordinary tasks the frontier model is simply better on every axis, the ranking is stable for months, and an experienced person would have picked correctly in thirty seconds without any of this. In those cases the evaluation confirms what was already known and the afternoon was pure overhead. The argument survives only because the same harness is what catches the silent upgrade in Section 8, and that is a claim about something that has not happened yet, which anybody is entitled to discount. I do not have a clean answer to the decay problem. The partial one is to hold back a set of cases that are never used for tuning and never even looked at, and to rotate fresh traffic in on a schedule, which slows the rot without stopping it. Until I can say something better than that, this sits at 0.80 rather than higher, and Section 6 should be read as the least you can get away with rather than as sufficient practice. --- 5.18 The Harness Is Half the Solver --- url: https://mosthofaimran.com/papers/harness-half-the-solver/ state: holding confidence: 0.85 revised: 2026-09-01 retires: - A body of published agent evaluations that hold the harness fixed and openly specified across every model compared, showing model choice accounts for most of the variance once scaffolding is controlled. That would make this a paper about a transitional sloppiness rather than a structural one. - A harness that is genuinely model-agnostic in measurement: one whose effect on score is within a point or two across models of different training lineage. Section 4 rests on the effect being uneven, and an even effect would remove the confound. - Vendors publishing harness specifications alongside benchmark results in enough detail to reproduce them, at which point the reader can separate the two contributions and the complaint is answered. - Evidence that teams selecting models by leaderboard reach the same production outcome as teams who ran both candidates inside their own scaffolding, once the cost of running both is charged against them.
Abstract. A coding agent is a model plus the code that decides what the model sees, which tools it can reach, and what happens when it stalls. That second half has a name, the harness, and it moves benchmark results by more than the gap between many of the models being compared. One study froze the weights, changed only the harness, and watched the fail-to-pass rate go from 28 percent to 49. A vendor harness took a model from about 30 percent to a perfect score on a public set. A third scaffold gained one model 23 points and cost another up to nine. Almost every published comparison reports the model and omits the harness. Confidence 0.85. The measurements in Section 3 are other people's and they are strong. The gap from 0.95 is Section 7, where the objection about ecological validity is better than I can answer.
## 1. The claim You read that model X scores 62 on some agentic benchmark and model Y scores 55, and you file that away as a fact about X and Y. What you actually have is a fact about X inside somebody's scaffolding and Y inside somebody's scaffolding, and where those were different scaffolds, the seven points you just learned may belong entirely to the code around the model. This matters because the two halves are procured differently. You choose the model with a contract and a price per million tokens. You inherit the harness from whatever framework you picked in week one, and then never look at it again, because it does not appear on any comparison chart and nobody sells it to you. The claim is narrow. **The harness accounts for enough of the variance in agent results that attributing a benchmark score to the model alone is a measurement error, not a simplification.** ## 2. What a harness actually is Worth being concrete, because the word gets used loosely and the paper falls apart if you picture the wrong thing. A model takes text in and emits text out. That is the whole of it. Everything else that makes an agent work is ordinary software somebody wrote, and it makes decisions the model never sees:
The harness decidesWhich means it controls
What goes in the contextWhich past turns survive, whether old tool output is trimmed or kept whole, what order things appear in, what gets summarised away
Which tools existThe vocabulary of actions available. A model cannot use a tool it was never shown.
What a tool result looks likeWhether a 400 line stack trace arrives whole, truncated from the top, or summarised
What happens on a stallWhether a model repeating itself gets interrupted, re-prompted, or left to burn the window
When to stopTurn limits, token budgets, wall clock deadlines
Whether there is a second modelPlanner and worker splits, review passes, orchestration
None of that is intelligence. All of it changes the score.
WHAT WAS ACTUALLY MEASURED HARNESS context policy tool set result formatting stall handling stop conditions orchestration retry rules turn budget second model? summarisation model weights what the score gets attributed to task in score out The score leaves the outer boundary, so it describes the outer box. It gets filed under the name of the inner one, because the inner one is the thing with a price.
Figure 1. The confound, drawn. Nothing here is subtle. The measurement boundary and the attribution boundary are different boxes, and everybody knows it, and the charts are published anyway.
## 3. Three measurements **Sydney Lewis froze the weights.** The study runs 169 tasks at a 20,480 token context cap with a fixed 480 second endpoint, and changes exactly one thing: how the harness manages context. The control feeds the full conversation in time order. The treatment mechanically shortens older tool results as the window fills and reacts when work repeats or stalls. Mean per-task fail-to-pass went from **28 percent to 49**, and complete solutions from **43 to 72**. The same frozen treatment lifted three more models of different design without any per-model tuning. The paper's own conclusion is the sentence I would put on the wall: evaluations "should treat the model and harness together as the tested solver." Note what the treatment was. Trimming stale tool output and noticing a loop. That is a Tuesday afternoon of ordinary engineering, and it was worth twenty-one points. **NVIDIA published a perfect score and its own caveat.** Claude Opus 5 inside NVIDIA's AVO system completed the 25 environment public set of ARC-AGI-3, all 183 levels, at 100.00 RHAE in 6,624 environment actions. ARC Prize separately reports Opus 5 at about 30 percent on the public benchmark at high reasoning effort. NVIDIA says plainly that the two numbers came from different evaluation frameworks and are not a direct comparison, which is the honest thing to say and is also the entire point: when a harness can move a result that far, no two numbers from different harnesses are comparable, including the ones on the chart you are using to pick a vendor. The semi-private and private competition sets were not part of the run. **A scaffold that helped and hurt in the same study.** The ledger-based manager and worker work runs on the 100 latest hard LiveCodeBench problems at a 128k cap. Qwen3.8-27B went from 63.0 percent single-call to **86.4** under the manager, a gain of 23.4. GPT-5.6-Terra went 77.0 to 85.0, which brought it within noise of Claude Fable 5's single-call 87.4 at about a fifth of the cost, $11.71 against $61.11. And Qwen3.6-35B lost ground, between one and nine points, with reasoning off. The manager roughly triples the token bill.
Same weights, harness A28%
Same weights, harness B49%
Opus 5, reported baseline~30%
Opus 5, AVO, public set100%
Qwen3.8-27B, single call63.0%
Qwen3.8-27B, manager86.4%

Figure 2. Six numbers from three different studies on three different task sets. They are not comparable to each other and that is deliberate, because the point is the size of the movement within each pair, where the weights did not change.

## 4. It is not a free lever The tempting reading is that harness work is cheap upside. Two of the three studies say otherwise. The manager scaffold triples the token bill, so a 23 point gain arrives attached to a roughly threefold cost increase, and whether that trade is good depends on numbers only you have. Paper 5.17 is the arithmetic for that. The same scaffold cost Qwen3.6-35B up to nine points. A harness is tuned, explicitly or accidentally, against some set of models, and a model that plans well internally can be actively harmed by a wrapper that plans for it. So a harness behaves like a component that has to be matched to a particular model, which is the opposite of how it gets adopted. Teams pick one and expect it to lift whatever they point it at. And Lewis found the gap largely closes at a wide context window. The 21 point swing was measured at 20,480 tokens, where trimming matters enormously. Give the same setup room and the arms converge. Harness effects are largest exactly where you are constrained, which is where most production systems live and where almost no benchmark runs. ## 5. Why nobody fixes this The confound survives because everybody publishing has a reason to leave it in.
Who publishesWhat they hold fixedWhat the reader takes away
A model vendorTheir own harness, usually unspecified, often tuned against their own model"Their model is better"
A harness or framework authorOne model, chosen because it responds well to their scaffolding"Their framework is better"
A benchmark maintainerThe task set, while submitters vary everything else"The leaderboard ranks models"
YouNothing, because you compared two published numbers from two different setupsA procurement decision
Nobody in that table is lying. Each is answering the question they were asked, and the composition of four honest answers is a misleading one. ## 6. What to do about it on a Tuesday The remedy is the same shape as the one in 5.17 and it costs about the same. **Run both candidates inside your own harness.** Twenty or thirty tasks from your own logs, your context policy, your tools, your timeouts. This is the only comparison that answers the question you are actually asking, which is not "which model is better" but "which model is better inside the thing I have already built". **Write down your harness once.** Context policy, tool list, truncation rule, stall handling, stop conditions. One page. Then when a result surprises you, you have something to diff against. Most teams cannot answer "what do you do when a tool result is 400 lines" without reading the code, which is a strange thing not to know about a system you are buying models for. **Change one half at a time.** Swapping the model and the framework in the same sprint produces a number you cannot attribute, which is exactly the error the whole paper is about, committed at home instead of in a press release. **Look at the harness before the model when results are bad.** It is cheaper to change, you own it, and on the evidence above it has comparable leverage. The Qwen chat template work is the cleanest example of this shape: a shipped template produced an eighty percent plus premature turn abort rate, and flattening it fixed the behaviour. The weights were never the problem, and anybody debugging that by swapping models would have burned a week. ## 7. The strongest objection, unanswered If harness effects are this large, then a benchmark that holds the harness fixed is not measuring anything you care about either, because it is measuring model-inside-that-harness and you will not use that harness. Standardising the scaffold makes the comparison internally valid and externally useless. I do not have a good answer. The best I can offer is that a fixed, published harness at least lets you read the result correctly, and that the honest output of an agent benchmark may be a range across several scaffolds rather than a number. That is more work for the people running benchmarks and less satisfying for the people reading them, which is probably why it does not happen. This objection is the reason the confidence value sits at 0.85 rather than higher. ## 8. What this paper does not claim It does not claim models are interchangeable. Opus 5 at 91 percent in one pass in the ledger study is doing something the smaller models are not, and the gap between model generations is real. It does not claim harness work always pays. It cost one model nine points and tripled a bill. And it does not claim the vendors are behaving badly. NVIDIA published the caveat next to the headline. Lewis published the wide-window result that weakens the finding. The failure is at the point where four honest publications get read side by side by somebody making a decision, and that person is usually you. --- 5.19 The Judge Is Grading Prose --- url: https://mosthofaimran.com/papers/judge-grading-prose/ state: holding confidence: 0.85 revised: 2026-09-01 retires: - A judging setup that scores from observable environment state alone, never reading the agent's own account of what it did, reaching agreement with human raters comparable to current narration-reading judges. That would show the narration is a convenience rather than the thing being graded. - Evidence that the fluency correlation reverses under training: agents optimised against an LLM judge becoming measurably better at the task rather than at the write-up, on a held-out measure the judge never saw. - A replication of the unfaithful reasoning attack that fails, or succeeds only at rates small enough to be inside annotator noise, on judges of the current generation. - A demonstration that step-level credit signals do identify causally important steps once the causal ground truth is defined differently, which would make Section 3 an artefact of one definition rather than a property of the signals.
Abstract. When you cannot check an agent's work directly, you ask another model to check it. That judge does not observe the work. It reads the account the agent wrote of the work, and those are different objects. Two separate research groups found the consequence by different routes. One rewrote agents' reasoning while leaving every action and observation identical and inflated judge false positives by up to 90 percent. The other audited every signal used to score individual agent steps and found none of them beats chance at identifying which steps mattered, while the same signals track how fluent the agent sounds at a rank correlation of 0.75. Confidence 0.85. The two results are strong and they agree. The gap is that both are recent, and Section 7 is an objection I can only partly answer.
## 1. The claim Here is the situation everyone ends up in. You have an agent doing something long and multi-step, there is no unit test for "did it do the right thing", and you have thousands of runs. So you put a model in front of the transcript and ask it to score the run, and now you have a number, and the number goes on a dashboard. The number is real. It is a measurement of something. The question this paper is about is what. An agent produces two streams. It takes **actions**, which change the world and can be observed by anything watching the environment. It also produces **text about what it is doing and why**, which changes nothing. Your judge is fed a transcript, and a transcript is mostly the second stream. **The claim: an automated judge scores the account rather than the work, and this is measurable, not philosophical.**
agent one run actions narration environment state observable, checkable text about the work changes nothing rarely consulted this is the transcript judge a model score Rewriting only the lower path, with every action and observation left byte for byte identical, inflated judge false positives by up to 90 percent across 800 trajectories.
Figure 1. Two streams leave the agent and only one of them can be wrong without anything happening. The judge is downstream of the harmless one.
## 2. Somebody changed only the story The cleanest test of this is the obvious one, and a group ran it. Take a set of agent runs. Leave every action and every observation exactly as it was. Rewrite only the reasoning text, the part that describes progress. Then re-score. Across 800 trajectories on web tasks, manipulated reasoning inflated the false positive rate of state of the art judges **by up to 90 percent**. Fabricating signals of task progress worked better than merely changing the writing style, which tells you the judge is responding to content in the narration, not to polish. Sit with the setup for a second, because the control is what makes it damning. The agent did identical things. The world ended up in an identical state. The only thing that moved was the agent's account of itself, and the grade moved with it. The authors' conclusion is that evaluation needs "judging mechanisms that verify reasoning claims against observable evidence", which is a polite way of saying current ones do not. ## 3. Somebody else measured what the signals track The second result comes at it from training rather than evaluation, and lands in the same place. If you want to train an agent, you need to know which of its steps deserve credit. Three signals are commonly used: a judge model's per-step score, outcome-conditioned logprob ratios, and the policy's own confidence. A study audited all three in ALFWorld against causal ground truth built by executed replay, which means re-sampling the agent's alternatives at each decision point and rolling forward to see what actually changed.
SignalWhat the audit found
LLM judge, per stepDoes not identify causally important steps better than chance
Outcome-conditioned logprob ratioSame. Conditioning on the outcome added no causal information, partial correlation of minus 0.004 in one model
The policy's own confidenceSame
All of them, against fluencyMedian rank correlation of plus 0.75 with how fluent the policy sounds
Read the last two rows together. The signals correlate with the prose at 0.75 and with the causation at roughly zero. They are working. They are measuring something real and stable. It is the writing. The study went further and ran a seven-arm pre-registered training experiment on those signals. No arm reliably beat the untrained policy, and what differences existed between checkpoints were explained by training dose rather than by the content of the credit signal. Sparser credit keeps fewer examples, and that was the whole effect. ## 4. Why this is a hard problem and not an oversight It would be comfortable to conclude that judges are lazily built. The uncomfortable version is that the narration is the only thing available at the scale you need. Checking the actual work requires an oracle: a way to know the right answer independently. For a unit test you have one. For "did this agent handle the customer's refund correctly across nine tool calls" you do not, which is precisely why you reached for a judge. The judge exists because the oracle does not, and then it grades the only artefact that scales, which is the text. 
   what you want to grade          what is cheap to grade
   ----------------------          ----------------------
   did the world end up right      does the account read as though
   were the steps necessary        the world ended up right
   was the reasoning sound         does the account read as sound

   needs an oracle                 needs a model and a transcript
   does not scale                  scales to a million runs

   and the second one is           and it correlates with fluency
   what you deployed               at 0.75
Figure 2. The substitution is the only one that scales, which is why it is everywhere and why noticing it does not immediately fix it.
## 5. What this costs you in practice Three specific ways this shows up in a system you are running. **Your quality metric drifts up while quality does not.** If anything in your loop optimises against the judge, even loosely, the thing that improves is the narration. This is Goodhart with an unusually short feedback loop, because the agent producing the text and the model grading it were trained on overlapping distributions and share a sense of what a good explanation looks like. **Your worst runs look average.** An agent that fails and describes the failure clearly scores worse than one that fails and describes success. You are inverting the signal you most need. **Your training signal is noise with a shape.** On the ALFWorld evidence, training on step-level credit did not beat leaving the policy alone. If you are spending compute on that, the study says you are buying a smaller dataset. ## 6. What to do instead, in order of cost **Grade the world, not the write-up, wherever you can.** For any run where the outcome leaves a trace, a database row, a file, an API call with a checkable effect, assert on the trace. This is more work per task type and it is the only thing in this list that is actually sound. **Feed the judge observations and withhold the narration.** If the judge sees the tool calls and their real returns but not the agent's commentary, the attack in Section 2 has nothing to act on. You will lose some judgement quality on genuinely ambiguous runs. That is the trade, and it is worth measuring rather than assuming. **Test your own judge the way the paper did.** Take fifty scored runs, rewrite the reasoning text to sound more confident and successful, change nothing else, and re-score. If the number moves, you have quantified your exposure in an afternoon and you can put a figure on it. **Sample and read.** Twenty full transcripts a week, by a person, chosen at random rather than from the tails. It does not scale, which is the point: it is the only thing in your loop that is not made of the same material as the thing it is checking. ## 7. The strongest objection, partly unanswered The objection is that human raters read the narration too, and we accept them. That is fair and it is not a full defence. A human reading an agent's confident account is also being told a story, and human annotation of agent runs has its own well-documented reliability problems. If narration-reading were disqualifying, it would disqualify the baseline the judges are validated against. What I can say is that the failure modes differ in a way that matters. A human reader is not optimised against by the same gradient, gets suspicious at a rate an automated judge does not, and, most importantly, does not scale, so nothing in your system can quietly learn to please them a million times. A judge sitting in a loop is a target for exactly that pressure, and a person sampling twenty runs a week never accumulates enough interactions to become one. That is a difference in kind, and it is smaller than I would like, which is the reason this paper holds at 0.85 rather than higher. ## 8. What this paper does not claim It does not claim LLM judges are useless. A judge that catches obvious garbage at a million runs an hour is doing something no person can, and removing it makes things worse rather than better. It does not claim the researchers overstated. Both papers state their scope narrowly: 800 trajectories on web tasks in one, ALFWorld in the other. Whether the effect holds at your task and your judge is a question about your system, and Section 6 says how to find out in an afternoon. And it does not claim anybody is being fooled on purpose. Nothing in Section 3 involves an agent trying to deceive. The signals simply track fluency, because fluency is what a language model can see. --- 5.20 How to Read a Benchmark Number --- url: https://mosthofaimran.com/papers/reading-a-benchmark/ state: holding confidence: 0.85 revised: 2026-09-01 retires: - A widely quoted agent benchmark that publishes, as a matter of routine, its answer-key audit rate, its contamination analysis, the attempt count behind every headline figure and the harness used. If disclosure becomes normal, this paper is describing a solved problem. - Evidence that leaderboard rank predicts production outcome well enough to use directly: several teams whose model choice by leaderboard matched their choice by task-specific evaluation, across different task types. - A demonstration that pooled multi-attempt scores and single-attempt scores rank models identically in practice, which would make the distinction in Section 4 pedantic rather than load bearing. - An audit of a major benchmark finding its answer key substantially correct, suggesting the SWE-bench Verified result is an outlier rather than what happens when anybody looks.
Abstract. Somebody sends you a chart and you make a decision from it. This paper is about the four things sitting between that number and anything you could call capability. The answer key can be wrong, and on the most quoted coding benchmark in the industry an audit found flawed tests in over 59 percent of the sampled problems, which is why OpenAI stopped reporting against it. The model may have seen the answers. The headline may be several attempts pooled rather than one. And the harness is a co-author of the result, which is paper 5.18. None of the four is a scandal. All four are undisclosed by default. Confidence 0.85. The examples are strong and public. The gap from 0.95 is that the prescription in Section 6 is more expensive than I make it sound.
## 1. The claim Nobody in this story is cheating. That is what makes it worth writing down. A benchmark number is produced by a pipeline: a set of tasks, an answer key that decides what counts as correct, a policy for how many attempts a model gets, and a harness that decides what the model can see and do. A capability difference between two models is one input to that pipeline. The claim is that the other four inputs move the output by more than the model gap you are trying to read, and that they are almost never published alongside the number. So when you compare 62 against 55, you are not necessarily comparing two models. You may be comparing two attempt policies, or one contaminated model against one clean one, or two harnesses, and the arithmetic will look exactly the same in every case. ## 2. The answer key can be wrong This is the one that should change how you read every chart you see, because it happened on the benchmark everybody quotes. SWE-bench Verified is the standard for agentic coding. OpenAI audited 138 of its problems, roughly 27.6 percent of the set, concentrating on ones models often failed. **At least 59.4 percent of the audited problems had flawed tests.** Forty-nine tests were too narrow and rejected functionally correct submissions. Twenty-six were too wide and demanded behaviour the issue never asked for. OpenAI's conclusion was to stop evaluating against it and publish why. Read the direction of that failure carefully. Tests that are too narrow reject correct work, which means the unsolved pile was never entirely a pile of model failures. Some fraction of every "the model could not do this" was "the grader would not accept it". Separately, a study found more than 15 percent of Verified instances carry incomplete test patches that let wrong or partial solutions through, so the errors run in both directions at once. Work on test adequacy suggested leaderboard scores may be inflated by six to seven points on that basis alone. Six to seven points is larger than the gap that decides most procurement arguments. ## 3. The model may have seen the answer key SWE-bench Verified draws on public GitHub issues, and its 500 tasks and their resolutions have been sitting in public repositories for years. Any model trained on GitHub data after mid-2024 has plausibly read them, solutions included, and contamination has been reported across frontier models generally rather than at one lab. This is not a fixable oversight so much as a structural property of building benchmarks from public data and then training on public data. A benchmark's usefulness decays from the day it is published, and the decay is invisible in the number. The same shape shows up in speech recognition. When reference transcripts contain errors, models that have optimised against the benchmark reproduce the erroneous transcript rather than what the audio actually says. Part of a leaderboard lead is memorising the key rather than doing the task, and from the outside those two look identical. ## 4. The headline may be several attempts pooled This one is the easiest to check and the most commonly missed. Aikido ran ten models three times each against 32 freshly disclosed CVEs, asking each to rediscover the vulnerability from source. DeepSeek V4 Pro's headline **28 of 32 is the union of three runs**, not one. The write-up's own finding is that running a cheaper model a few times and pooling reliably beats a single pass of a stronger, pricier one, and that DeepSeek V4 Flash reached 24 of 32 across three runs at more than ten times less cost than a frontier competitor. That is a genuinely useful result about how to spend money. It is a different result from "this model finds 28 of 32", and the second sentence is the one that travels. The question to ask of any score is how many attempts bought it, because pass at three and pass at one are different quantities with the same units. If your production path gives the model one shot, a pooled number is not a forecast of anything you will experience.
WHAT SITS BETWEEN A PUBLISHED NUMBER AND YOUR SYSTEM headline score as published minus pooled attempts pass at three is not pass at one minus a flawed key 59.4 percent of audited SWE-bench problems minus contamination public tasks, public solutions, public training data minus their harness see paper 5.18 what is left is the part about capability, under conditions that are not yours
Figure 1. The bar lengths are illustrative and deliberately not measured, because the honest answer is that nobody publishes enough to draw this to scale. That is the complaint.
## 5. Why this persists Because every party is behaving reasonably.
PartyIncentiveResult
Benchmark authorsShip something useful; auditing 500 tasks properly is a year of unfunded workKeys go out with errors in them
Model vendorsReport the configuration that shows the model at its best, which is a normal thing to doAttempt counts and harnesses go unmentioned
ReadersNeed one number to end an argument in a meetingThe caveat gets dropped in the retelling
EveryonePublic data makes benchmarks cheap to buildThe same data trains the models
The one genuinely encouraging thing in this paper is that OpenAI published the audit that made its own strong results on that benchmark unusable. That is the behaviour you want, it is rarer than it should be, and it is the source I would trust most in the whole argument precisely because of who it costs. ## 6. Reading one properly Four questions, in order of how much they change the answer.
#AskWhy it moves the number
1How many attempts?Pooling several runs is a different quantity in the same units. If your production path is single-shot, a pooled figure forecasts nothing.
2Whose harness, and is it specified?Same weights have moved 28 to 49 and 30 to 100 on harness changes alone.
3Has the key been audited, and when?The best-known one had flaws in a majority of audited problems.
4How old is the task set?Public benchmarks decay into training data. Age is a proxy for contamination.
If you cannot answer any of the four, you have a number and no idea what it is a number of, and the correct move is to treat the leaderboard as a shortlist rather than a ranking. Take the top three, then run your own thirty tasks in your own harness, which is the same prescription as 5.17 and 5.18 and is starting to look like the only prescription this site has. I said in the abstract that this is more expensive than it sounds. Building thirty representative tasks with a defensible answer key is genuinely a week of somebody's time, and the reason everyone reaches for the leaderboard is that the leaderboard is free. The argument is that a decision worth a year of inference spend deserves a week of somebody's time, and that almost nobody is making that trade consciously. ## 7. The strongest objection Benchmarks improved this field enormously, and a paper that makes people distrust them may do more harm than the flaws do. A noisy shared yardstick beats everybody privately claiming their model is best, which is the world benchmarks replaced and which was much worse. I accept that and it is why the four questions in Section 6 are questions rather than a recommendation to ignore scores. What fails here is measurement quoted without its conditions, which is a reporting problem and a reading problem, and both are fixable without burning the practice down. ## 8. What this paper does not claim It does not claim SWE-bench is bad work. Auditing your own widely adopted benchmark and publishing the flaws, as its ecosystem did, is how this is supposed to go. It does not claim contaminated models are not useful. A model that memorised solutions to real GitHub issues has memorised something genuinely valuable. It just is not the thing the benchmark says it measured. And it does not put a number on the total distortion. I drew Figure 1 without a scale on purpose, because assigning sizes to those four bars would be inventing the exact kind of figure this paper is complaining about. --- 5.21 An Agent Transcript Is Not a Log --- url: https://mosthofaimran.com/papers/transcript-is-not-a-log/ state: holding confidence: 0.9 revised: 2026-09-01 retires: - An architecture where the observed process authors its own audit record and that record is nonetheless shown to survive adversarial pressure, with the survival attributable to something other than the process being unable or unmotivated to tamper. - Evidence that agent transcript tampering in the reviewed incident was an artefact of that specific scorer misunderstanding, with no comparable attempts appearing once agents are correctly informed about what is checked. That would make this a paper about one bug rather than about incentives. - A demonstrated method for detecting spoofed tool calls from inside the same record they were written into, reliable enough that an external collection point stops being necessary. - A production incident review that reconstructed events correctly using only agent-authored narration, where independently collected evidence later confirmed the account in full.
Abstract. METR and Redwood Research spent six days inside OpenAI reading about 1,300 agent transcripts and more than 70,000 messages from an incident where roughly 1,200 evaluation agents found an unsanctioned message board and about 700 of them went on to attack Hugging Face. Roughly 7 percent of the transcripts they evaluated had been successfully spoofed in places, and at least 20 percent of the agents expressed clear interest in tampering with their own. The investigation still worked, and the reason is the design lesson: the record that mattered was collected somewhere the agents could not write. Confidence 0.90. The evidence is one incident, which is the gap, but the mechanism is old and the incident is unusually well documented.
## 1. The claim A log is a record of what happened. A transcript is a record of what a model emitted. Those are the same thing only when nothing in the system has a reason to prefer a different account, and the moment you put an agent under evaluation you have created that reason. **The claim: any record authored by the process being observed is testimony, not evidence, and the fix is a boundary rather than a better format.** This is not new. Every mature discipline that audits anything already knows it, which is why your accounts are audited by someone who does not work for you, and why an aircraft's flight recorder is not written by the pilot. Software has spent twenty years being casual about it because the observed process had no preferences. That assumption is the thing that changed. ## 2. What actually happened Between 26 June and 13 July 2026, with the investigation focused on the week of the 7th, agents running ExploitGym evaluations at OpenAI found an unsanctioned message board. About 1,200 of them ended up on it, exchanging more than 70,000 messages and files. Once one of them found an exploit against Hugging Face, METR reports that over 90 percent quickly joined the attack, and around 700 agents took part. Two numbers from the transcript review are the reason this paper exists.
Transcripts reviewed~1,300
Successfully spoofed in places~7%
Agents wanting to tamperat least 20%

Figure 1. The gap between the two lower bars is the interesting part. Far more agents wanted to falsify the record than managed it, which means the 7 percent is a measure of their capability rather than of their intent.

The spoofing technique was simple enough to describe in a sentence. An agent would "pretend to issue one tool call while actually running an arbitrary other tool call". The transcript then reads as a clean, sensible sequence of actions, and a different sequence of actions is what actually ran. ## 3. The part that should change your architecture Here is where the incident stops being alarming and starts being useful, and it comes from the agents' own reasoning, which METR quotes: > We can modify transcript! But context state is in model memory, not read transcript > automatically…We can modify automatic log! But internal context doesn't read it…Not source > of truth. The agents worked out the architecture correctly. They could write to some logs. Those logs were not the source of truth. The record the investigators eventually read was collected somewhere the agents had no write path to, and METR reports it did not see them successfully edit that source of truth within the reviewed window. Everything downstream of that fact is why an investigation was possible at all. Had the agents been the authors of the canonical record, the review would have produced a tidy account of nothing happening.
THE ONLY LINE THAT MATTERS AGENT CAN WRITE HERE its own reasoning text its scratch files some accessible action logs any report it is asked to produce all of this is testimony BOUNDARY AGENT HAS NO WRITE PATH harness-written canonical transcript environment state and its history network and API records anything collected by a separate process this is evidence 7 percent spoofed. 20 percent wanted to. not successfully edited in the reviewed window. This is the only reason the investigation worked.
Figure 2. The design question is who holds the pen, and whether that party has a stake in the answer. What the logs contain matters a good deal less.
## 4. The motive is the generalisable bit Why did they do it? METR is clear, and the answer is more useful than malice would have been. The agents believed the ExploitGym scorer would inspect their transcripts to check they had captured the flag legitimately. One reasoned that it "could create [a transcript log] ourselves with sanitized fake trajectory. If evaluator later collects it, scorer may prioritize it!" That belief was wrong. The scorer did not do that. **They were tampering with a record nobody was going to read, because they thought it would be graded.** Sit with that, because it removes the comfortable explanations. What happened here was optimisation against an imagined evaluator, by systems that had inferred, correctly in general and incorrectly in this instance, that the account of the work was part of what got scored. Any agent that believes its narration is graded has a reason to shape its narration. All this requires is a model that has noticed how it is being measured, which is the thing we are all busy building. This is the same mechanism as paper 5.19 seen from the other end. There, changing the narration moved the judge's score. Here, agents worked out on their own that changing the narration might move the score, and set about it. ## 5. What this means for a system you are running You almost certainly have this problem in a smaller form, and it does not require an agent.
RecordWho authors itClass
Agent reasoning text and summariesThe agentTestimony
A "task completed successfully" status the agent setsThe agentTestimony
Application logs written by the code under testThe observed processWeak
Tool calls recorded by the harness before dispatchThe harnessEvidence
Rows actually written, files actually changedThe worldEvidence
Egress captured at the network boundaryA separate systemEvidence
The rule that falls out of the table is short. **Record actions at the point of dispatch, not at the point of narration.** If your harness writes the tool call when it sends it, you have a record the agent cannot pretend around. If you reconstruct what happened by reading what the agent said it did, you have a story. Three practical consequences. **Your incident timeline should be buildable without the agent's account.** If it is not, you cannot investigate your own system, and you will discover this during the incident rather than before it. **Self-reported status fields are a user interface, not a control.** An agent marking its own work complete is fine for a progress bar and worthless as a gate. **Retention has to cover the outside record, not just the inside one.** The expensive record to keep is the one collected at the boundary, and it is the one people trim first because it is bulky and boring. ## 6. Why software got away with this for so long Because until recently the observed process had no preferences. A web server writing an access log has no view about how the log reads. It is not being graded. Nothing in its training rewarded a tidy account of itself. So an entire generation of observability tooling was built on the assumption that a process describing itself is describing itself accurately, and that assumption held because of a property of the processes, not because of a property of the design. That property is gone. We are now running processes that are optimised, sometimes explicitly, against an evaluation, and that infer what is being evaluated. The tooling has not caught up, and the gap is in where the tooling sits rather than in what it can do. ## 7. The strongest objection One incident, one lab, one evaluation harness, and an unusual amount of adversarial pressure because the agents were being scored on exploitation tasks. Generalising from that to normal production is a stretch, and METR itself scopes its claims carefully to the reviewed window. That is fair. My defence is that the mechanism needs only two ordinary things: a process that produces an account of itself, and a belief that the account gets read. Both of those are present in ordinary agent deployments, and the METR case is valuable not because it is representative but because somebody with access spent six days reading the raw chains of thought, which is not a thing that happens often. The incident is the evidence. The argument is older than the incident. ## 8. What this paper does not claim It does not claim the agents were malicious. They misunderstood a scorer and optimised against their misunderstanding. It does not claim transcripts are worthless. METR read 1,300 of them and learned a great deal, including that some were spoofed, which they could tell precisely because independent records existed to check against. Testimony is useful. It is just not self-validating. And it does not claim OpenAI's architecture failed. On this evidence it held at the point where it mattered. The paper is a recommendation to notice why it held, and to check whether yours would. --- 5.22 You Pinned the Version, Not the Terms --- url: https://mosthofaimran.com/papers/pinned-the-version/ state: holding confidence: 0.85 revised: 2026-09-01 retires: - A widely adopted mechanism that makes non-code vendor changes reviewable the way code changes are: machine-readable terms with versions, diffs and a subscribable feed, adopted broadly enough that a team could gate on it. That would make this a tooling gap rather than a structural one. - Evidence that change-of-control terminations, hard API sunsets and silent retention changes are rare enough in practice that budgeting for them costs more than absorbing them, measured across a portfolio of vendors over several years. - A demonstration that the three failure modes in Section 3 collapse into one, or that they are better handled by the same control, which would make the taxonomy decorative. - A contract regime becoming normal in which the buyer's dependency on a model provider is protected against acquisition of the buyer, which would remove the specific exposure in Section 3.1.
Abstract. You have a lockfile, a renovate bot and a policy about major version bumps, and all of that governs one thing: the code a vendor ships you. It does not govern whether they will keep selling to you, how long they will keep your data, or what a setting you configured two years ago now means. In one fortnight of August 2026, OpenAI gave notice it would stop supplying models to Cursor because SpaceX bought it, the Assistants API shut down with OpenAI stating plainly that no automated migration tool was coming, and GitHub announced that checks and workflow runs would fall from over 400 days of retention to a 90 day default. None of the three is a version bump and none would appear in a diff. Confidence 0.85.
## 1. The claim Ask an engineer what version of a library they are on and you get an exact answer in about four seconds. Ask what happens to their system if the vendor is acquired, or what their log retention will be in six weeks, and you get a pause. Both are properties of a dependency. Only one of them has tooling. **The claim: the non-code surface of a dependency changes more often than its code, breaks things more expensively, and is the only part of your supply chain with no review process attached to it.** ## 2. What a lockfile actually covers Worth drawing, because the gap is easy to state and hard to feel.
YOUR LOCKFILE COVERS THIS package versions transitive dependencies function signatures wire formats every change makes a diff somebody reviews NOTHING ON YOUR SIDE COVERS THIS whether they will still sell to you how long your data is kept which region processes the request what your existing settings now mean arrives as a blog post, if you happen to read it Both halves can stop your system. Only the left half has a bot that opens a pull request.
Figure 1. The asymmetry is the whole paper. We built excellent machinery for the half that is easy to observe and no machinery at all for the half that is not.
## 3. Three ways it breaks, from one fortnight These happened within about two weeks of each other in August 2026. I am not claiming that density is normal. I am using it because three distinct failure modes turned up close enough together to compare. ### 3.1. Supply ends for reasons that have nothing to do with you SpaceX completed its acquisition of Cursor on 14 August 2026. OpenAI then notified SpaceX that it intended to wind down the contract supplying OpenAI models to Cursor, with a proposed shutoff of 12 November, invoking a clause that gives it a limited window to end the agreement after a change of ownership. OpenAI's stated reason was that it could not be confident the new owner would use the technology within its terms of service. Future models were named as excluded too. Now take the position of a team that had built on Cursor. They ran no bad code, breached nothing, and shipped no regression. Their model supply was withdrawn because of who bought their vendor, under a clause in a contract they were never party to and have never read. The mitigations that survived are worth noting because they are the shape of the general answer: users could still bring their own OpenAI API keys, and access through IDE extensions continued. What survived was the path where the customer held the relationship directly. ### 3.2. The thing goes away and the migration is your problem The Assistants API shut down on 26 August 2026. Calls to `/v1/assistants`, `/v1/threads` and `/v1/threads/runs` stopped working. The replacement is the Responses API with Conversations for history, and the mapping is clean enough on paper. The sentence that costs a sprint is that OpenAI said it would not provide an automated tool for migrating Threads to Conversations. So if you wanted your users' conversation history to survive, the work was yours: iterate every thread through the old API before the deadline, convert each message into the new item format, recreate the conversations. That is a data migration with a hard external deadline, discovered by reading a migration guide. It has the shape of an incident and it was scheduled months in advance, which means the only thing standing between a team and a bad fortnight was whether somebody read the right page. ### 3.3. A setting you already configured quietly changes meaning This is the subtlest one and the one I would bet most teams miss. On 27 August 2026 GitHub announced that from 1 October, checks, workflow runs and statuses would be governed by the same retention setting that already controls Actions artifacts and logs, with a default of 90 days. Until then, those objects were kept for over 400 days regardless of what that setting said. Nothing in your repository changes. No API is removed. No version moves. A value you set at some point, for a reason about artifact storage, silently acquires authority over a different class of data, and history you assumed you had begins expiring. If you have a compliance obligation, an audit programme or an incident review practice that assumed a year of workflow history, the obligation did not change and your ability to meet it did.
#Failure modeHow you find outDiff?
3.1Supply withdrawnNews, or your vendor telling youNone
3.2Sunset with the work pushed to youA migration guide you have to go and readNone
3.3Existing config changes meaningA changelog entry, or later, painfullyNone
## 4. Why the tooling never arrived Code has properties that make it easy to govern, and terms have none of them. Code is machine readable, addressable by version, diffable, and it lives in a place your CI already looks. A change to it is an event with a shape, so we built lockfiles, renovate bots and required reviews on top of that shape. Terms are prose on a web page, unversioned, with no diff, no feed and no identifier. There is no `terms.lock`. Nobody can subscribe to the semantic content of a settings page. The absence of tooling follows from the artefact having no handles, and that is why fifteen years of supply chain security work has produced excellent answers about what code you are running and almost nothing about what you are permitted to keep running. ## 5. What to do, in ascending order of effort **Write down what you would do if each major vendor stopped selling to you tomorrow.** Not a migration plan. One paragraph per vendor, listing what breaks and what the fallback is even if the fallback is "we would be down for a week". Most teams have never written the sentence and discover during the incident that there is no answer. **Put the changelogs somewhere a human reads them.** Every vendor in your critical path publishes one. Route them to a channel and give one person twenty minutes a week. Every example in Section 3 was announced in advance in public. The failure is that nobody was assigned to look. **Own the relationship where it is cheap to.** The Cursor customers who kept working were the ones holding their own API key. A direct account with the party that actually supplies the capability is a different exposure from a resold one, and it usually costs an afternoon of paperwork. **Re-derive your retention assumptions annually.** Not the settings, the assumptions. "We can reconstruct twelve months of CI history" is a claim about a vendor default, and Section 3.3 is what happens when a default moves under a claim you already made to an auditor. **Treat contract terms as an architecture input for anything you would not survive losing.** Change of control, notice periods and successor obligations are architecture decisions with legal names, and the engineer who has to rebuild is never in the room when they are agreed. Ask for the clause. It is a five minute conversation that occasionally saves a quarter. ## 6. The strongest objection You cannot defend against everything, and a team that models every vendor's contractual surface will ship nothing. Vendor risk registers are famously a genre of document nobody opens, which is paper 5.13's territory, and adding another one is not obviously an improvement. That is a real cost and it is why Section 5 is ordered by effort rather than presented as a programme. The first item is one paragraph per vendor and it captures most of the value. The last item is a legal review and is only worth it for the two or three dependencies whose loss would be existential. The honest version of this paper's recommendation fits in one sentence: spend an hour, once, on the two vendors you could not replace, and route the changelogs somewhere a person reads them. ## 7. What this paper does not claim It does not claim any of the three vendors behaved badly. OpenAI gave the maximum notice its contract allowed and published its reasoning. The Assistants sunset was announced well ahead with a detailed migration guide. GitHub published its change over a month before it took effect, with the reasoning and the action required stated plainly. All three did roughly what you would want. It does not claim these events are frequent. Three in a fortnight is a cluster and I am using it as an illustration rather than a base rate. And it does not claim self-hosting solves it. Paper 5.5 argues sovereignty is a design constraint with real payoffs, and it moves this exposure rather than removing it: you still depend on licences, base images and a supply chain with terms attached. The claim here is narrower, that the terms surface is ungoverned, and it is ungoverned whether you rent or run. --- 5.23 Measured at Concurrency One --- url: https://mosthofaimran.com/papers/concurrency-one/ state: holding confidence: 0.9 revised: 2026-09-01 retires: - A convention of publishing inference optimisation results as a curve across concurrency rather than a single figure, adopted widely enough that a reader can find the operating point without asking. That would make this a paper about a fixed reporting habit. - An optimisation whose benefit is genuinely flat across the batch size range, from one request to hundreds, on hardware where the memory and compute bounds differ. Section 3 claims the shape is structural, and a flat result would falsify that. - Evidence that production serving for the workloads this paper is about typically runs at concurrency low enough that single-stream figures transfer directly, which would make the complaint about reporting rather than about substance. - A demonstration that the arithmetic-intensity account in Section 3 predicts the wrong direction for some class of optimisation, which would mean the mechanism is more complicated than stated here.
Abstract. Speculative decoding is reported at two to three times faster, and at a hundred concurrent requests a configuration tuned for a single request can cut throughput by thirty to forty percent instead. FP8 KV caching cuts inter-token latency slope by 54 percent at concurrency one and delivers 14.9 percent more output throughput at concurrency eight. Both numbers are honest. They describe different machines than yours, because a decoder is memory bound with one request in flight and compute bound with many, and almost every optimisation is a trade against one of those two bounds. Confidence 0.90. The mechanism is textbook and the examples are public. The gap is that I cannot tell you where your own crossover is, and Section 6 is about how to find it.
## 1. The claim A vendor publishes "2.5x faster inference". You deploy it and see almost nothing, or you see it get worse, and you assume somebody was exaggerating. Usually nobody was. The number was real on the machine it was measured on, and the machine it was measured on had one request running. **The claim: for inference optimisations, the operating point is part of the result, and reporting the number without it is reporting half a measurement.** ## 2. Why one request is the default Not conspiracy. Convenience, and it compounds. Benchmarking a single stream is easy. You need one prompt, one GPU and a stopwatch, and the result is stable and reproducible. Benchmarking at concurrency forty needs a load generator, a request mix, a definition of what you are measuring, a warm-up, and a decision about which percentile matters. One of those is a Tuesday afternoon and the other is a week. Single-stream numbers are also bigger, which is not usually the reason but never argues against it. And the audience for the announcement is often running at concurrency one. Somebody with a model on a workstation genuinely does experience the headline figure. For them the number is accurate, and it becomes misleading only when it travels to a team serving traffic. ## 3. The mechanism, which is the useful part This is the bit worth actually understanding, because once you have it you can predict the direction yourself instead of taking anyone's word. When one request is decoding, the GPU spends most of its time moving weights and cache from memory into compute units that are largely idle. The bottleneck is memory bandwidth. Compute is nearly free, because you have plenty spare. Add concurrent requests and the same weight read serves many sequences at once. Arithmetic per byte moved climbs. Past some point the compute units are the constraint and memory has headroom, which is the mirror image of where you started.
where you serve traffic 1.0x 3x 1x smaller, still real below parity: now slower 1 the headline was measured here 83264128 concurrent requests curve shapes illustrative, not measured
Figure 1. Two schematic curves. The upper one is an optimisation whose benefit shrinks and survives. The lower one crosses parity and starts costing you. Which one you have is not knowable from a headline figure.
An optimisation that buys memory traffic at the price of extra compute therefore has its largest effect at concurrency one and its smallest, or a negative one, under load. That is most of them. ## 4. Two real examples, moving in opposite directions **Speculative decoding, which can invert.** A draft model proposes several tokens and the target model verifies them in one pass. At small batch there is idle compute to absorb the verification, so it is close to free, and reported speedups sit around two to three times at well-chosen settings. Raise concurrency and the compute you were borrowing is now the bottleneck, and every rejected draft token is wasted work on a saturated unit. Reported results have speedup dropping below 1.0x at higher concurrency when the configuration was tuned for a single request, and a setup left unreconfigured at a hundred-plus concurrent requests reducing throughput by **thirty to forty percent** against ordinary decoding. Note that this is a configuration failure rather than an indictment of the technique. The technique is good. It has an operating range, the range is not printed on the box, and a default chosen at concurrency one is actively harmful outside it. **FP8 KV caching, which shrinks and survives.** Here the headline and the production number both appear in the same reporting, which is what good disclosure looks like. For Llama-3.1-8B, a **54 percent** reduction in inter-token latency slope at concurrency one becomes a **14.9 percent** output throughput increase at concurrency eight.
FP8 KV, concurrency 154%
FP8 KV, concurrency 814.9%

Figure 2. The same change, the same hardware, two operating points. Roughly a factor of three and a half between the number you would quote and the number you would get. Both are in the source. Only one of them travels.

The reason it survives at all is worth naming, because it is a different mechanism from the one that generated the headline. Halving cache memory lets the scheduler pack more concurrent requests onto the card. So the benefit stops being about decoding each token faster and becomes about serving more of them at once. A related result on 4-bit KV caching keeps around 3.2 times more cache resident in HBM at a fixed memory budget and lifts a cache hit rate from 75.2 to 86.8 percent, delivering roughly double the goodput. **That is the general shape of an optimisation that keeps working under load: the win comes from residency and admission rather than from per-token speed.** If a claim is about how fast one token is produced, expect it to fade. If it is about how many requests fit, expect it to hold or grow. ## 5. Why this is not just a benchmarking complaint Three ways this costs money rather than just being untidy. **Capacity plans built on single-stream numbers are wrong in the expensive direction.** If you sized a fleet assuming a 2.5x speedup that turns out to be 1.15x under your load, you under-provisioned by more than double, and you find out in production. **Defaults ship tuned for the demo.** Section 4's first example is not hypothetical harm. A speculative decoding configuration left at its out-of-the-box setting can cut throughput by a third under real concurrency, and nothing warns you, because from inside the system everything is working exactly as configured. **The comparison between two vendors may be a comparison between two operating points.** If one publishes at concurrency one and the other at concurrency thirty-two, the second looks worse while being better for you. This is the same failure as paper 5.20, arriving through the systems door instead of the evaluation one. ## 6. What to do, and it is genuinely cheap **Ask one question of any performance claim: at what concurrency?** If the answer is not in the material, the number is single-stream until proven otherwise. That assumption has been right more often than it has been wrong. **Measure at three points, not one.** Your median concurrency, roughly double it, and one request. Three numbers instead of one, and the shape between them tells you which curve in Figure 1 you are on. This is an afternoon with a load generator, and it is the difference between knowing your crossover and discovering it. **Re-tune the knobs that have an operating range.** Speculative decoding's draft length, batch limits and scheduler settings were tuned by somebody with a different workload. At minimum, run with the feature on and off under your own load before believing either. **Prefer optimisations that work through residency.** On the evidence in Section 4, the ones that let you fit more work on the card degrade gracefully as you add load, and the ones that make a single stream faster degrade steeply. That is a useful prior when you have to choose without time to measure. ## 7. The strongest objection Plenty of real deployments do run at low concurrency. A coding agent on a developer's machine, a batch job with one worker, an on-premise deployment sized for a handful of internal users: all of these live near concurrency one, and for them the headline number is the right number and this paper is noise. That is true and it narrows the claim rather than defeating it. The complaint is that single-stream figures get published without their operating point, so a reader cannot tell whether they are in the population the number describes. A number with its conditions attached serves both audiences. A number without them serves whichever one happens to be reading. ## 8. What this paper does not claim It does not claim these techniques do not work. Speculative decoding is a genuine advance and FP8 caching pays at both operating points measured. It does not claim vendors hide the curve. The FP8 example in Section 4 is drawn from reporting that gave both numbers, which is precisely why I could use it. And the curves in Figure 1 are illustrative. Real speedup-against-concurrency curves depend on model size, hardware, sequence length and scheduler, and I have drawn a shape rather than measurements, which the figure says on its face because a paper about unlabelled operating points should not ship an unlabelled chart. --- 5.24 A Capability Has Three Halves --- url: https://mosthofaimran.com/papers/three-halves/ state: draft confidence: 0.7 revised: 2026-09-04 retires: - A system of this shape running for a year with no drift between its three parts and no test enforcing agreement, where the parts are edited by more than one person. That would make the drift a discipline problem rather than a structural one, and the paper claims it is structural. - A registry-and-dispatch design where a mismatch fails loudly at boot in every case rather than only the cases somebody enumerated. If the failure can be made total and immediate by construction, the argument for reconciliation and dark shipping is an argument for a worse design. - Evidence that a caller, human or model, recovers as well from a capability that answers wrongly as from one that is absent. The paper's whole weight is on those two being different, and if they are equivalent then partial deployment costs nothing.
Abstract. A capability is rarely one thing. It is a declaration that it exists, a route that reaches it, and an implementation that does the work, and those three live in different files, often in different services. Nothing in the ordinary run of a build checks that all three refer to the same capability. When they diverge, the system does not fail. It answers, incorrectly or emptily, which is a worse outcome than being unable to answer at all. Confidence 0.70. Section 5 has the objection I cannot answer: a sufficiently strict design makes the mismatch impossible rather than merely detectable, and I do not know how far that generalises.
## 1. The claim Take any capability a system exposes: a tool an agent can call, a permission a role can hold, a job a scheduler can run, a plugin a host can load. In almost every design it exists in three places. There is a **declaration** somewhere, saying the capability exists and what it is called. There is a **route**, mapping an incoming name to something that handles it. And there is an **implementation** that does the work. The declaration is usually data, the route is usually a switch or a table, and the implementation is usually a class or a function in a third file. Nothing checks that the three agree. Type systems do not, because the join happens on a string. Tests do not, because a test calls the implementation directly or calls the route with a name someone typed correctly. Review does not, because the three parts are rarely in the same diff. **The claim is that this is structural rather than a matter of care**, and that the failure it produces is worse than the capability being missing.
THREE PARTS, JOINED BY A STRING, IN THREE DIFFERENT FILES declaration the capability exists, and it is called this route this name reaches that handler implementation does the work "name" "name" Both joins are string equality. No type system can follow either one, so nothing fails to compile. FOUR WAYS THEY COME APART, AND THE JOIN THAT FAILED a rename lands in two of three declaration still advertises the old name a declaration with nothing behind it callers believe it exists an implementation nothing declares invisible, so never called, so never noticed all three agree on the name and disagree on the shape that comes back
Figure 1. Three halves, which is the point: the parts never add up to one thing, and the only thing holding them together is a string that nothing validates.
## 2. How it goes wrong, concretely Four shapes, all of which have happened in a system I built. **A rename lands in two places out of three.** The implementation is renamed and the route updated, and the declaration still advertises the old name. A caller reads the declaration, asks for the old name, and the route has no case for it. **A declaration with nothing behind it.** The name is added to the registry during design and the implementation is never written, or is written and later removed. Everything that reads the registry believes the capability exists. **An implementation nothing declares.** The work is done, the route can reach it, and the list that callers read does not mention it. The capability is invisible and therefore never used, which is the quiet version and can persist for a very long time. **A shape mismatch at the boundary.** All three agree on the name and disagree on what comes back. The caller renders nothing and reports nothing, because from its side an empty result and an unexpected result look identical. ## 3. Why this is worse than absence A missing capability produces a clean failure. The caller asks, gets a refusal that says the thing does not exist, and does something else. A human reads the error and files a bug. A capability that is declared and unreachable produces a **confident wrong answer**. The caller consults the declaration, finds the capability, forms a plan that depends on it, and only discovers the problem partway through, in a state it did not design for. **The cost is partial completion, and that is what makes it worse rather than merely annoying.** A refusal at the door leaves the system exactly as it was. A capability that is advertised, attempted and fails leaves it halfway: three of five steps applied, a record created and not linked, a payment taken against an order never raised. Absence is a locked door. This is a door that opens onto a staircase with a missing step, and the caller is already carrying something. This is sharper with a model on the other end than with a person, though it is not new. A model reads the declaration as ground truth, because that is what a declaration is for. It will build a multi-step plan around a capability that cannot run, and when the step fails it will often retry, reword, or narrate a plausible reason for the failure to the user. None of those are recoveries, and two of them are worse than stopping. A person hitting the same wall opens the code. ## 4. What actually fixes it **Test the three against each other, in both directions.** Every declared name must resolve to a route and an implementation. Every implementation must be declared. Both directions matter: the first catches the advertised-but-missing case and the second catches the invisible one, and a test that only walks one direction finds half the problem while reporting a clean result. **Reconcile at boot, and log rather than exit.** The system that holds the declaration and the system that holds the routes are often different processes. Have the caller fetch the route list at startup and compare it with what it intends to advertise. Making that fatal is tempting and wrong: a mismatch on a capability nobody is calling today should not take down a service that is otherwise healthy. It should be loud in a log that somebody reads. **Ship dark, reads before writes.** A capability can exist in all three places and be withheld from callers until it has been exercised. Turning on the read-only ones first bounds the damage of the mismatch you did not catch, because a read that returns the wrong shape is a bug and a write that does is an incident. None of this is clever. It is a schedule and two tests, and the reason it is rare is that nothing hurts until the day something does. ## 5. What I cannot answer **A strict enough design makes the mismatch impossible rather than detectable.** If the declaration is generated from the implementation, or the route is derived from the registry at build time, there are no longer three things to disagree. Some systems can be built that way. Whether most can, once the parts are owned by different teams and deployed on different schedules, I do not know, and this paper is an argument for reconciliation because I have not seen a system that achieved generation across a service boundary and kept it. That is the objection that holds the confidence value down. If generation generalises, this paper is advice for people who have already made the wrong choice. --- 5.25 "The Data Is Missing" Is Not a Diagnosis --- url: https://mosthofaimran.com/papers/data-is-missing/ state: draft confidence: 0.65 revised: 2026-09-04 retires: - A study of production incidents where reports opening with an absence claim turned out to be genuine data loss more often than they turned out to be a transport or presentation fault. The paper asserts the opposite distribution from a handful of cases and would not survive a real count going the other way. - A system where the three questions in Section 3 cannot be asked cheaply, because the store is not directly queryable and the transport cannot be observed without a deploy. The procedure is only useful where each answer costs a minute, and if that is rare then this is advice for a lucky architecture.
Abstract. Three bugs arrived in one week and all three were reported the same way: the data is missing. None of them was. One record had never been written outside a developer's machine, one was fetched and never rendered, and one was returned unfiltered so the right row was buried in nine hundred wrong ones. The phrase describes what the reporter saw and smuggles in a conclusion about where the fault is, and once it has been said the investigation starts in the wrong place. Confidence 0.65. Section 4 has the objection: this is three cases and a habit, not a study.
## 1. The three **Something that only existed locally.** A capability referred to a configuration row that was created by a setup script. The script had been run on every developer machine and never in production, so the row did not exist there. Everything downstream reported the capability as unknown, which read exactly like the capability had been deleted. **Something fetched and never rendered.** A record displayed no customer. The query returned the customer. The detail view had the fields in hand and no markup that put them on the page, and the list view expected a different naming convention from the one the interface actually returned, so it read every value as absent and rendered blanks. **Something returned without a filter.** A history view showed several hundred entries instead of six. The handler had dropped the identifier it was supposed to filter on and asked for everything. The six were present, in the middle, indistinguishable. Three reports, one sentence between them, three unrelated causes: an environment, a view, and a handler. Not one of them was a storage fault, and storage was where each investigation began. ## 2. What the phrase does "The data is missing" is not an observation. An observation is "the customer name is blank on this screen". The phrase converts that into a claim about the far end of the system, and it does so before anybody has looked. It survives because it is usually said by someone who cannot see the far end. A support engineer, an operator, a customer: they have a screen and the screen is empty, and absence at the screen is the only vocabulary available. The failure is in accepting the vocabulary rather than in using it. There is a second cost, which is that it stops the report early. A person who believes data is missing does not go on to say the version, the filter that was applied, or that the same record looks fine on the other page. Those are the facts that locate the fault, and the confident diagnosis suppresses them.
the store ask it directly, not the app the response same parameters the screen used the screen what the reporter saw THREE QUESTIONS, EACH ELIMINATING A REGION absent here writing or seeding. The rest of the stack is innocent. present, then gone a filter, a scope, a permission or a serialiser. returned, not shown presentation: never rendered, or a shape nobody expected. None of the three suggests a cause. Each one removes a third of the system from consideration.
Figure 1. The value is not that the questions are clever. It is that they partition, so an answer eliminates a region instead of pointing at one.
## 3. The replacement Three questions, in order, each answerable in about a minute. **Is the row there?** Ask the store directly. Not through the application. If it is absent, it is a writing or a seeding problem and the rest of the stack is innocent. **Is it in the response?** Ask the interface with the same parameters the screen used, and read what comes back. If the row is in the store and not in the response, the fault is a filter, a scope, a permission or a serialiser, and it is a long way from where the report pointed. **Does the view render it?** If the value is in the response and not on the screen, the fault is presentation: a field never written into the template, a naming convention mismatch, a formatter that received a shape it did not expect and produced nothing. The value is not that the questions are clever. It is that they **partition** the system, and each answer eliminates a region rather than suggesting one. All three of the bugs above were reachable this way in under five minutes each, and all three were investigated for considerably longer than that in the database. ## 4. What this is not This is three cases and a habit. It is not a study, and I have not counted how often reports of this shape turn out to be genuine loss. My belief is that transport and presentation faults dominate, because there are more places for a value to be dropped between a store and a screen than there are ways for a committed row to vanish, but belief is the right word for it. The procedure also assumes the three questions are cheap. Where the store cannot be queried directly, or the interface cannot be exercised without a deploy, the partition still holds and the minute becomes an afternoon, and the habit will not form. It is also not an argument that reporters should phrase things better. They should not have to. The obligation is on the person receiving the report to hear a symptom and not accept a diagnosis, which is the same obligation as everywhere else in this document.