Case study

Five services, one contract, no downtime

Five production services that could not talk to each other, migrated onto a published contract one at a time, with no flag day and no lost leads. What follows includes what it cost and what is still unfinished.

The finding

Five services were already in production, each shipped on its own. In order they are the inbound revenue pipeline: Inkwell captures the form submission, Webhook Relay delivers it with signing and retries, Switchyard normalises and routes it, Distill checks it against what the CRM already holds, Splitstream records which variant produced it.

The console was meant to trace a lead across all five. Reading the source to build it turned up something more interesting than the console: they could not talk to each other.

Inkwell’s destinations sent a fixed header set with no Authorization support. Webhook Relay’s ingest sits behind an API key and requires an envelope Inkwell could not produce. That handoff failed on both auth and shape.
Webhook Relay signed with one HMAC scheme. Switchyard verified a different one. That handoff failed on signature.
No correlation column existed anywhere, so “which delivery came from this submission?” had no answer at any layer.
Each service had independently reinvented idempotency, signing and error shapes — five implementations of three problems, none interoperable.

Nothing was broken. Every service passed its own tests and served its own users. They had simply never been asked to work together — and five services that each work perfectly and cannot be composed is a different kind of failure than an outage. It does not page anyone and it does not fix itself.

What the contract had to be

Adopt standards, do not invent one. W3C Trace Context, Standard Webhooks, RFC 9457 problem+json, IETF idempotency semantics. A bespoke contract would have solved the same problem and been worth nothing outside these repos. The signature verifier that came out of it works against Svix, Clerk and Resend, because they implement the same spec.

Verifying those specs against the published documents rather than from memory was a gate of its own, and it caught four errors in what the plan had asserted. The costly one: Standard Webhooks secrets are base64 with a whsec_ prefix and the HMAC key is the decoded bytes. Getting that wrong produces signatures no compliant verifier accepts — and it looks like a configuration problem for as long as you let it.

Additive, never replacement. Every service keeps its original scheme documented and working. The contract is an additional negotiated mode, selected per source and per destination. There is no flag day anywhere in this program.

Exact pins, one service at a time. A shared package across six services is a single point of failure — a bad patch release breaks signature verification everywhere at once. Every consumer pins an exact version, upgrades roll out one service at a time, and a release is never tagged and adopted in the same change.

Prove it at the queue boundary first. Every hop here crosses a queue, so HTTP-only propagation would have produced five disconnected root spans — a design that looks correct on a diagram and is useless in practice. The gate was a throwaway app proving context survives HTTP → dispatch → worker → outbound HTTP as one trace, run on the database queue driver with the ambient scope cleared between dispatch and execution. On the sync driver it would have passed vacuously. That is the trap, not the test.

Ship the console against the broken version first

The console went live before any production service was touched, with explicit bridge classes standing in for the two impossible handoffs, unlisted and noindex.

The reason that mattered was not risk. It was that running the conformance probe against the fleet before the contract existed produced a red baseline: all five services failing trace.http.echo, measured in production, recorded with a timestamp. Without it there is nothing to move from and the migration is a claim. With it, the dashboard went 5 failing → 4 → 3 → 2 → 1 → 0, one service at a time, each step measured against deployed code.

Zero downtime by construction, not by care

The rule: every default preserves current behaviour, so each adoption deploy is a no-op on the wire and all of the risk lives in a separate per-source cutover on a different day.

New columns are nullable, so a rollback can leave them. New scheme fields default to the service’s existing scheme. New capabilities are opt-in per source and per destination. Switchyard — the service carrying real leads — was deployed with all six production sources verified still on the old scheme with dual-accept off, and an identical lead count before and after. The deploys were not the risky part because they were built not to be.

Dual-accept, and the failure that actually matters

The obvious risk in a signing cutover is that a bad signature gets accepted. That is not the risk that matters. The risk that matters is that a good lead is silently rejected because sender and receiver disagreed for ten minutes during a cutover. A dropped enquiry is indistinguishable from no enquiry. Nobody files a ticket; the prospect just never hears back.

So every inbound surface gained a dual-accept window, tested in both directions — including the rollback direction, where config has already flipped and the sender has not. Accepting either scheme is not accepting anything: with dual-accept on, a garbage old-scheme signature, a wrong-secret new-scheme signature, an hour-old replay and an unsigned request all still return 401.

Every request also increments a persistent per-source, per-scheme counter in the database, because removing these windows has to be evidence-based and log retention on this host is 30 days with a size cap that can truncate sooner. Inferring “no old-scheme traffic” from logs that may have rotated is not evidence.

The canary, and why isolation was the load-bearing part

The procedure is: synthetic source first, dual-accept before scheme, highest-value source last in a low-traffic window. Building it as a command rather than a checklist mattered for one reason — a cutover verified once is a demonstration, not a canary. The probe runs hourly from cron, so the soak day is observed rather than merely elapsed.

The part that took the most thought was isolation. CRM routing and destination fan-out are configured per workspace, not per source, so a canary in the live workspace would create a real CRM task on every hourly probe. The canary provisions its own workspace with neither, and re-asserts that on every run rather than trusting it was true at setup. Verified after cutover: production workspace unchanged at exactly 68 leads, all six real sources untouched, zero CRM tasks created.

The rollback that breaks the thing it was meant to save

Code rolls back with a symlink swap and a php-fpm restart; migrations add nullable columns only. That covers code and schema, and it is the part everyone writes down. It does not cover configuration — and configuration is where the risk was deliberately put.

If a destination has been switched to the new scheme and the code reverts to a release that does not implement it, the reverted code cannot sign what that destination now expects. The rollback is clean, the deploy is green, and the data plane is down — caused by the rollback itself.

So every scheme change records its previous value at the moment of change, and the rollback reverts both as one step. Dual-accept makes the forward cutover safe; this makes the reverse cutover safe. Neither substitutes for the other, and the second is the one that gets skipped because the first feels like it covered it.

What five migrations actually found

The migrations were the cheap part. Adopting a contract meant exercising code paths that had never been exercised, which turned up more than a dozen production defects in services that were reporting healthy. Nearly all of them are the same species.

A failure suppressed into a success

One service’s semantic dedup had never once worked in production, behind four stacked defects that each converted a failure into a success: a Python worker’s virtualenv living inside the atomic release directory, so every deploy broke it; the workers directory missing from the deploy transfer list; a request shape the worker rejected with a 422 on every call; and a bare catch (Throwable) that made the 422 look like a healthy degrade. Plus a deploy step restarting a program that did not exist, under ignoreError, printing success every time. The endpoint returned 200 throughout.

A queue with no worker

The capture service had no queue worker configured at all, ever, while its queue connection was Redis. What sat in its own infrastructure directory was another service’s supervisor config copied verbatim — program name and path included — so provisioning from it would have started a second worker for the other service and still left this one with none. Zero deliveries across every submission the service had ever received. Ingest is synchronous, so every submission returned 200 and the service looked healthy.

Configuration read through env() outside a config file

An API key pepper was correctly set in .env and resolved to an empty string in production, because config:cache runs on every deploy. Consequence: no authenticated request to that service could ever have succeeded — and nothing looked broken, because no key had ever been minted to try.

Green CI that does not run the code

One service’s CI ran a single test suite; its entire feature directory had never executed. Another pinned sqlite in its PHPUnit config, which cannot parse PARTITION BY, so 17 of 25 tests errored before a line of application code ran. Both were green in the sense that matters to a badge.

A load harness that measures itself

The concurrency check drove six copies of one payload, and the capture service dedupes identical submissions inside a 60-second window. So one real submission and five deduplication hits — and since a deduplicated submission still returns 200, the first hop read healthy on all six while only one was real. A load harness that trips the system’s own idempotency is measuring the idempotency.

And the console exempting itself

This console publishes the conformance matrix and was the one service that never installed the contract package. Its own row read not_applicableacross every requirement — not because the requirements did not apply, but because nothing had pointed the probe at it. A cell that says “does not apply” and means “nobody looked” is the same defect as all the others, wearing a dashboard. It is fixed; the row is now measured like every other.

The common shape is not carelessness. Each of these produced a success signal — a 200, a green check, a restarted service, an n/a — and a success signal is where nobody looks.

Where it ended up

	Before	After
Run-time conformance checks failing	5 of 5 services	0 of 18 checks
Services conformant	0	6 of 6
Services deferred	—	0
Hops in a traced run	6	6
Trace depth	1	5
Hops brokered by the console	6	0
Production leads lost or corrupted	—	0

Identical hop count with a different depth is the whole result. The same work happens either way; the difference is who called whom. Before, every hop was a child of the root span because the console made every call, so the waterfall renders flat. After, each hop parents to the one before it. Depth is computed by walking stored parent/child links, not read off a label — a flat fan-out is depth 1 whatever it is called.

Test coverage across the five services went from 425 tests to 493, and this console from 79 to 109. Two packages are published: the contract, and a conformance suite that ships assertions against an interface each service implements, because a generic package cannot know one service’s job class from another’s.

The evidence that matters most is not on the dashboard. On the morning after the riskiest deploy, a real enquiry arrived from one of the live marketing sites feeding this pipeline — not a probe, not a synthetic source. It was parsed, scored and routed into a CRM task one second after it landed, carrying a trace id, a correlation id and a production trace class. Nothing about it was special, which is the point.

See the before / after Check the matrix

What is not done

Three things, stated because a case study that ends on the good numbers is marketing.

“Cut the highest-value source last” turned out to have no referent. The plan carried it as the final step for a day after the canary went clean. It is wrong: the six sources carrying real leads do not use the signed-webhook path at all. They authenticate with a bearer token against the JSON API, and each was provisioned with an inbound signing secret no sender has ever used — zero signed requests across all six, per the same counters the sunset rule reads. Flipping their scheme would have changed an expectation on a route with no traffic and no sender, and would have let the step be marked done without anything happening. Every source that does receive signed webhooks runs the new scheme and has soaked clean. The cutover is complete; the plan item describing it was not.

Dual-accept is still on everywhere, and stays on until the counters show 30 consecutive quiet days per source. That is the rule working, not a gap.

Half the conformance matrix is honestly empty.Build-time conformance is meant to come from each service’s CI posting its result here. None does yet, so all 18 build-time cells read not_reported — rendered as its own state, never as a pass. An empty cell shown green would be exactly the defect this program keeps finding.

What transfers

Little of this is specific to these five services. Capture the goldens and commit them beforetouching anything, because a regression oracle written afterwards can be retrofitted to whatever the new code produces. Verify a claim by grepping for the symbol it says it uses, not the file it lives in — three items in this program were marked complete and were not. Check what CI actually runs. Give every degrade path a healthy-path test, or the degrade path becomes camouflage. Confirm the deploy target, because copied scripts keep the donor’s defaults.

And treat not_applicable as a first-class state worth defending. Honest N/A beats fabricated compliance and beats a false red — but only when it means the requirement does not apply, and never when it means nobody checked.