Audit evidence programme

1. The constraint

An audit does not inspect your systems. It inspects a paper trail about your systems, covering a window that has already closed.

That sentence decides the architecture. By the time an auditor or a client’s security team asks, the period they care about is in the past, and no amount of engineering effort in the present can produce evidence for a Tuesday in March that nobody recorded. Evidence is a by-product you either captured at the time or did not, and nothing written afterwards substitutes for it.

The failure mode this produces is well known to anyone who has watched it: a team is told an audit is in eight weeks, and eight weeks of engineering goes into reconstructing a story about the previous six months. The story is usually true. It is also unverifiable, and an auditor who cannot verify it has to record an exception, which is the thing everyone was trying to avoid.

2. The decision everything follows from

Evidence is emitted by the pipeline that does the work. Not collected by a person, not screenshotted, not assembled in a spreadsheet during audit season.

A control and the evidence that it ran are two different deliverables. Teams build the first and assume the second. Scanning every image is a control. Being able to show, fourteen months later, that image sha256:... was scanned on a given date, by a named pipeline version, with a recorded outcome, is evidence, and it does not exist unless somebody decided it should.

Enforced at the release gate rather than documented as policy. A build that skips the scan does not produce a warning, it fails. This matters more for evidence than for security: a control with an exception path has to have every use of that path explained to an auditor, and “we can override it in an emergency” turns one control into a sampling exercise across every emergency you ever had.

Retained append-only, because a mutable evidence store is not evidence. If a record can be edited after the fact, its value in an audit is the value of somebody’s word, which is what the record existed to replace.

3. Why this is delivery infrastructure, not a compliance function

The reason to build it this way is the client’s security team, who are the real gate and who arrive during procurement rather than after.

For banking and telecom buyers, a security review sits between a signed intent and a contract. It is run by people whose job is to find the reason to say no, on a schedule set by the buyer, and it asks for artefacts rather than assurances. A team that can answer it from a store that already exists answers in days. A team that cannot spends three weeks producing documents, during which the deal does not move and the engineering roadmap does not either.

That is the actual return. The compliance framework names the controls, and the evidence machinery decides whether answering for them costs a fortnight per deal or an afternoon. Every regulated customer asks the same questions in a different order, which is exactly the shape of problem that rewards building once.

4. Where SOC 2 fits

The Trust Services Criteria overlap heavily with what ISO 27001 already requires, and the evidence a well-built programme emits serves both with different labelling. Access reviews, change management, monitoring and vendor risk are the same underlying records.

The part that does not transfer is the observation window. SOC 2 Type II is not an assessment of your controls today, it is an assessment of whether they operated continuously across a period, and that period cannot be compressed by working harder. Paper 5.15 is the argument for treating that window as the schedule and everything else as procurement. Paper 5.16 is the argument that the only real compression is owning less infrastructure to evidence.

Both grew out of this work, and both are published as arguments with confidence values and retirement conditions rather than as claims of a certification.

5. No figures, and why the page says so

Section 3 admits a system with the numbers it produced. This note has none, and none are invented to fill the gap.

The numbers that would matter are the ones this programme was never instrumented to produce: time from a client security questionnaire arriving to it being answered, the proportion of controls with pipeline-generated evidence against those attested by a person, and the count of exceptions raised across review cycles. Those are the right measurements. Not collecting them is a real gap in the programme rather than a gap in this write-up, and stating it is more useful than a plausible figure would be.

6. What I would do differently

Instrument the programme itself. Everything above measures the systems under it and nothing measures the machinery. A programme whose whole argument is that evidence must be generated rather than asserted, and which then asserts its own effectiveness, has a hole in the middle of it. The three measurements in section 5 are cheap and I did not build them.

Write the exceptions register before the first review, not after the first exception. An exception that is recorded, explained and has a remediation date is a normal artefact of a mature programme. The same exception, discovered by an auditor, is a finding. The difference is entirely who wrote it down first, and it costs nothing to be the one who did.

Test controls against known-bad input from the start. Failure 6.1 is open and it is the one that keeps its shape no matter how good the evidence trail gets. A scanner that runs on every build and is silently misconfigured produces flawless evidence of nothing. Feeding a deliberately vulnerable dependency through the pipeline once a quarter would catch it, and it is an afternoon of work against a class of failure that an audit is structurally incapable of detecting.