Towards the Dark Factory: The Wall
Part 2 of 5 in the “Towards the Dark Factory” series
In Part 1 we borrowed Saumya Tyagi’s vocabulary: between the layer that generates code and the layer that validates it stands the wall. In his article the wall protects holdout scenarios, which he takes from StrongDM’s software factory: acceptance tests the coding agent never sees, judged by a separate evaluator against a throwaway deployment. The generator cannot game a test it has never read.
Our wall is built differently, and this part is about what it is made of, why it holds, and what it cannot see. The short version: nothing in our wall reads prose. It is made of llvm-cov, a mutation runner, a replay script, a catalogue drift test, the type checker, and one agent whose only job is to disbelieve another agent. The one place we do use holdout data, we will name.
Why not holdout scenarios
We thought about holdout scenarios and did not adopt them, for a reason that is specific to this app. Sestava is an iPhone timer. Its hardest behaviour is a workout engine with time semantics (ticks, catch-up after a suspension, a clock adjusted backwards), a state machine driven by voice cues and haptics, and decoders for an open plan format. An LLM evaluator poking an API of an ephemeral deployment is a good fit for a web service; it is a poor fit for “the rest step must not advance when now is earlier than the step’s start”. Those are properties, and properties want deterministic tests.
So the wall is white-box. The same agent that writes the code writes the tests, and that is a real weakness we should say out loud: a test written by the author of the code proves the code does what the author thought. What makes the tests mean something is the layer below them.
Layer 1: coverage, 100%, no exclusions
scripts/coverage-gate.sh runs the package’s tests instrumented, exports the profile through llvm-cov, and fails unless lines, functions and regions are each at 100% for Sources/. There is no exclusion mechanism. The contract file says “Coverage is 100% with no exclusions” and the gate has no flag to add one.
The gate’s own strictness found a bug in the gate. During the session-formats wave it reported a false zero-count line on the core package on 2 of 5 runs, with no code change between them. With a 99% target a person would have shrugged and re-run; with a 100% target and no exclusions there was no room to shrug, so it was investigated. Swift Testing runs test functions as concurrent tasks inside one process, and its LLVM coverage counters are plain non-atomic increments (Clang’s -fprofile-update documents the default); two tests hitting the same source location at the same instant can lose the only increment that line ever gets. The instrumented pass now runs --no-parallel (plain swift test stays parallel), and 20 consecutive runs on macOS plus 10 inside the Linux container reported 100% after the fix. The cost is about two seconds. The gate’s comment block explains all of this, so the next agent does not “optimise” it away.
Layer 2: the mutation runner
Coverage says a line ran. It does not say a test would notice if the line were wrong. For that we wrote a mutation runner, in the repo, on swift-syntax, on the first evening (the commit is timestamped 22:02). It mutates Sources/ one site at a time (relational operators flipped, arithmetic swapped, && for ||, boolean literals, integer increments and decrements, negated conditions, one deleted statement in a function body) and runs the test suite on each mutant.
A mutant that survives means no test noticed the change, and a surviving mutant fails the build. The only way out is an allowlist, one id per line with a justification on the same line, for mutants that provably cannot change behaviour, and the runner reports stale entries so the list cannot rot. As of 30 September the core allowlist has entries like EngineReducer.swift:302:35:integerDecrement:1->0, with the sentence explaining why the loop returns the same index either way.
The numbers at the typed-errors wave’s pull request of 30 September: 1,730 mutants on the core package, 1,706 killed, 8 timed out (counted as killed; usually an infinite loop), 16 survived and allowlisted; 1,622 on the app-logic package, 1,615 killed, 4 allowlisted survivors. Stillborn mutants, the ones that do not compile, are not counted at all. On pull requests CI runs the runner over changed files only; on main and weekly it runs everything.
The runner has its own tests, and two of its rules came from watching it fail. It runs the unmodified suite once as a baseline and aborts if that suite fails, times out or reports zero tests, rather than scoring mutants against a broken baseline. And when a mutant’s test run hits the timeout, the runner kills the whole process tree and counts the mutant as timed out only if it can confirm the run ended; otherwise it retries on a fresh copy and, failing that, fails the whole gate with unconfirmedRun. That rule exists because a lost termination notice once left an orphaned test helper idling for three hours, and because one CI job on Linux waited on a pipe held by an orphan until GitHub cancelled it at 240 minutes.
Layer 3: plans replayed onto a clean checkout
Part 1 said the unit of work is a spec and a plan, each with an adversarial review. Reviewing a plan by reading it is level 2 thinking. Our rule, since day six: a plan is reviewed as a file. A script extracts its code blocks, replays them onto a clean checkout, and the result is compared with the tree the plan’s writer verified, byte for byte. The rule was written the day a reviewer ran the extracted code and found that one block in the plan did not parse as written and two of its diffs were refused by git apply; the writer’s verified tree had been fine, the document describing it had not.
Since then, “replay byte-identical” is a line in a plan’s review verdict, and code in a plan counts as verified only when its tests ran.
Layer 4: the reviewer who disbelieves
Every task in a run gets an independent reviewer: a different agent, a read-only worktree, the diff, the brief and the implementer’s report. The rule that matters: the reviewer re-runs the implementer’s reproductions and never takes its word. An implementer’s report says “29 of the new tests were red on the old code”; the reviewer checks out the old code and counts.
This layer is where the wall is most like Tyagi’s: the reviewer did not write the code and does not know how it was produced beyond the diff. It is also where the rounds pay for themselves. On the blog’s pull request four review rounds each found what the round before had missed (an empty blog linked from every page, accented tags dropped from a sidebar, feeds in English defaults, then one regression the fix had introduced); on the Apple Watch plan two rounds did the same. A reviewer’s minor findings do not block a merge; they are parked as issues with a label, which is Part 4.
Layer 5: types where prose would drift
Two rules from the contract file are less about tests than about making certain mistakes unrepresentable.
Typed errors carry location as data; the UI never matches on error text. A decode error from the plan format is a typed reason with a JSON path, not a sentence. In the wave of 30 September this was carried through the whole app: every English String error detail in the app-logic package became a typed value rendered from a LocalizedText key, one owner at a time, so that a Czech translation is a catalogue entry and not a second if on a string.
All user-facing sentences are LocalizedText from Copy; run mise run copy-catalog after adding one; CatalogTests fails on drift. A tool regenerates the string catalogue from the Copy definitions, and a test compares the two. That wave’s gate line reads 0 added, 518 keys, 0 stale. A sentence written straight into a view does not compile against the type; a key added to Copy without the catalogue fails the test.
Two smaller bricks: swift format lint --strict on every pull request and push to main, and a schema gate that validates the format’s example plans against the JSON Schema and, as importantly, asserts that a folder of invalid fixtures is rejected.
The one holdout rule
Here is the place where we do what StrongDM’s scenarios do. The contract file has one rule about data the tuner never saw:
A change tuned against examples is measured on examples the tuning never saw, committed before the tuning starts.
It came from a feature, “Describe a workout”, that asks the on-device language model to turn a sentence into a plan, and from a bug: the model read reps as seconds. An agent tuned the instructions against the 16 known cases and raised the pass rate from 24% to 44%. On 8 cases it had never seen, the rate went from 11% to 13%. Without the unseen cases the change would have shipped as a fix. The next attempt moved the checks outside the model (numbers must appear in the description, repeated items are refused) and was measured on a second holdout set, where wrong plans fell from 61% to 24%. That one is the fix going out.
This is the wall in Tyagi’s exact sense, and what StrongDM compares to a holdout set in model training: the generator blind to the evaluation data. We use it exactly where it belongs: where the thing being changed is a prompt and the thing being measured is a model’s behaviour. Everywhere else the tests are white-box, and the mutation gate is what keeps them honest.
Three things the gates caught that a review had missed
The wall earns its keep on the days it disagrees with a reviewer.
Two site-wide accessibility defects. The first Lighthouse and axe run over the website found that white text on the brand red measured 3.6:1, under WCAG AA’s 4.5:1, and that links inside running text were told apart by colour alone. Three design handoffs and every review, human and agent, had passed both. Two CSS tokens changed, and the budgets now run on every site change.
Twelve surviving mutants after eight reviewed tasks. The session-formats wave on the core package went through eight tasks, each implemented, reviewed, fixed and re-reviewed, then a full mutation gate: 12 survivors outside the allowlist. An agent wrote the tests that killed ten of them and justified two as equivalent; the review of that diff was its own round. Reviewed code with 100% coverage still had a dozen places where a flipped comparison changed nothing a test could see.
Every draft blog post would have failed the site. A smoke test added for one issue checks every page in the sitemap for a description of 70 to 160 characters and a title of at most 60 including the site’s suffix. Measured against the ten draft posts that four review rounds had just approved: all ten descriptions too long (166 to 235 characters), five titles over the limit. The reviews had read the posts; nobody had measured them.
What the wall cannot see
The gates run on a Mac and on Linux. They cannot feel a haptic, hear a cue over a podcast, or read Czech. When the step-change vibration turned out to be unnoticeable in a pocket, no test had been wrong; the unit tests cover the choice of technique and the engine’s life cycle, never what is felt, and the pull request that fixed it shipped with a 24-line device checklist for a human with an iPhone. When a blog draft claimed a feature the roadmap had but the app did not, the validator was a reader comparing the copy with the code, and the ruling on what counts as true at launch was a person’s. That is the human’s remaining work, and it is Part 5.
Before that, Part 3: with a wall like this, the agents can be trusted to set their own pace, and what that looks like on one Mac with two Swift slots and one simulator.
This is Part 2 of the Towards the Dark Factory series. Part 1: Level 2 to Level 3 in a Week. Part 3: The Agent Sets the Pace (coming on 16 October). Part 4: The Agent Feeds the Backlog (coming on 23 October). Part 5: Where the Human Still Is (coming on 30 October).
Related: Architecture Tests as a Seatbelt Against Agentic Drift · Linting the Prompt Layer
