Abstract
Eva X is the lab's social distribution engine: a pipeline that scans, analyses, generates, gates and publishes content for a single operator. An internal audit in July found one of its three networks implemented but not wired into the publishing path — and failing with an explicit runtime error instead of pretending to have posted. We recorded that as a virtue. A second audit found the general case, and it was the opposite: with the system in live mode and no credentials present, the network client itself returned success, and a post was written to the database as published with an invented platform ID. This note documents the failure mode, the architecture we built around it — an explicit mock | dry_run | live switch, the removal of the false-success path from all three clients, and an atomic per-day cost reservation — and the honest status of the system, which is that no real post has ever been published through it.
1. The research question
The pipeline question is the one the whole portfolio keeps asking: how far does the automation go alone, and where does the human enter. The answer here is the pair approve → publish — content is generated and classified without supervision, and nothing leaves without a human gate.
But building it surfaced a second question, and it is the one this note is actually about: can a system be made structurally incapable of reporting work it did not do? Not "does it usually tell the truth" — can the lie be removed from the set of things the code is able to express?
2. The failure mode
An autonomous publisher has an obvious risk (it posts something it should not) and a quieter one (it says it posted, and did not). The second is worse, and it is worse for a specific reason: the first is visible to everyone, immediately. The second is visible to no one. The dashboard is green, the row says published, the operator moves on, and the gap between the record and reality only opens wider as the system runs.
We hit that failure in our own code. The path was this: when publishing was requested without a usable credential, the network client short-circuited and returned success. Nothing downstream knew to disbelieve it. The post was stored as published with a fabricated identifier, because the storage layer's contract was "a published post has an ID," and something had to fill the field.
The uncomfortable part is what it says about the thing we had already praised. The honest failure we found in July — one network refusing to fake a post — was not a property of the system. It was an accident of that network being incomplete in a way that happened to fail loudly. The same codebase, one layer down, would lie.
3. The publish switch
The fix was not to patch the client. It was to make the publishing mode an explicit, first-class piece of configuration with three values and no ambiguous default.
mock — the default. No network call is made. The action is logged as simulated, and the post is stored with status mock_published, with the platform identifier and publication timestamp left null. This is the whole design in one detail: the simulated path is not allowed to invent the field that only a real publication can produce.
dry_run — also no network call, but the system assembles what it would have sent: the rendered text, a real check of whether a usable credential exists, and an estimated cost for that specific post. "What would leave here, and what would it cost."
live — the client is actually called, after format validation and after a budget reservation succeeds.
The value is read once at startup, and an unrecognised value degrades to mock with a warning rather than being interpreted. A misconfigured deployment publishes nothing; it does not guess.
4. Removing the lie
With the switch in place, the false-success path was removed from all three network clients. In live mode with a missing or incomplete credential, publication now fails explicitly and names what is missing. There is no branch left that returns success without a network round trip.
This is the generalisation of the July finding, and it is the sentence worth keeping: the honest failure stopped being one network's accident and became a property of the publishing path. Any network, any missing credential, any misconfiguration — the system reports what happened, or it reports an error. It has no way left to report a post it did not send.
5. Cost as a gate, and one deliberate imperfection
Publishing to a paid API is an action with a price, so the same path enforces a daily budget. The check is an atomic script in the coordination store that sums settled spend plus in-flight reservations before authorising a post.
The reason it is atomic is documented in the code, and it is a real defect that was caught rather than a theoretical concern: two concurrent publications from the same batch — the normal behaviour of a scheduler publishing in a group — both passed the budget check before either one accounted for its own spend. Sequential logic gave the wrong answer for the system's own default workload.
Two details we want on the record because they are choices, not oversights. First, simulated and real spend live in separate buckets — a week of dry runs can never inflate the real number. Second, if the coordination store is unavailable, the recorded decision is to let the publication proceed without accounting for it. That is availability chosen over accounting, deliberately, and written down as a decision so that nobody later reads it as a bug.
6. What we measured
The rebuild ran as twelve sequential contracts over roughly ten days in August: 53 commits since the July audit. The publishing and cost paths — 679 and 498 lines — were later read line by line by a reviewer who was not the author, which is how the false-success path was found.
Alongside it: eight system invariants became automated tests; a scheduler defect that aborted 39% of its cycles was fixed; the backend and frontend suites run green (93 backend tests with one expected failure, 49 frontend); browser-level smoke tests verify rendering with pixel evidence; and a single-operator identifier that had been hardcoded was closed out across the nineteen remaining call sites. The database moved from schema-on-boot to a real migration baseline.
7. Status — and it is the honest part
The service is deployed on lab-owned hardware, survives operator logout, and is reachable on the internal network. It is tested. Its human approval gate exists as code and as interface, not as a diagram.
And no real post has ever been published through it. None of the three networks has a production account connected. One of them has a complete in-app authorisation flow; the other two are connected by pasting a token obtained outside the product. The publish mode has never been switched away from mock — verified in the deployed configuration, not assumed.
So "in production" here describes the infrastructure and nothing else. We are stating that plainly because the alternative — letting "deployed" imply "in use" — is the same class of claim the system itself was rebuilt to stop making.
8. Limitations
The audit was internal: a reviewer inside the lab reading code, not an external assessment. The learn stage of the published pipeline still does not exist as a service — the pipeline diagram is ahead of the code, and has been since July. Git history was not rewritten, so credentials removed from the working tree may persist in earlier commits; that is a residual risk we have accepted and are naming rather than implying it away. The audit also produced a defect list that is not yet closed. And there is no evidence at all about live behaviour under real conditions — by construction, because the mode has never been enabled.
9. Why publish a publisher that has never published
Because the finding was never the product. It is the failure mode, and we found it twice in the same codebase: once as a virtue, once as a defect, and only the second time did we understand it was the same thing. A system that reports success it did not earn is more dangerous than one that fails, because failure is information and a false record is the absence of information disguised as its presence. The design rule that came out of it is the one we will carry into the next system: honesty has to be structural. If a component can say it did something it did not do, eventually it will, and nobody will be watching at that moment.
10. Next
Connect a single network in dry_run and compare estimated cost against the real invoice — the estimate is untested against reality, and an untested estimate is another number nobody is checking. Close the open defect list from the audit before enabling live on anything. Then decide whether the missing learn stage is worth building, or whether the published pipeline description should shrink to match the code.
Research home: blackicelabs.ca/research