Futex: Silence is a denial

On building a human-in-the-loop control plane for agents that can spend money, grant access, and delete things.

0. The sentence that started it#

An agent of mine sent this, unprompted, at two in the morning:

I could not wait, so I learned the contract from a rejected delivery.

That is a machine describing initiative. It is also, if you squint, a machine describing the exact moment before an incident. It waited, it got no answer, and it proceeded on its own judgement — correctly, that time.

I have spent a year building systems where agents do real work: place orders on a live book, rotate credentials, promote parameters, deploy. Every one of them eventually arrives at the same fork. Something irreversible is about to happen. There is no human at the terminal. What now?

The industry’s answer is mostly block and hope. Put a input("continue? [y/N]") in the loop, or a Slack message with two buttons, and wait. This is a bad answer for reasons I will get to, and I got tired of writing a worse version of it in every project.

So I built Futex: a multi-tenant control plane for durable, policy-driven human decisions. An agent asks for sign-off, gets an ID back immediately, and carries on. A human answers from their phone, on their own time, through a link in an email. The agent resumes when the answer arrives — by webhook, or by polling, or by neither, if the answer never comes and the policy says a silent hour means no.

It is live at futex.rodmena.co.uk. This is what it does, why the obvious alternatives are worse, and what I got wrong on the way.

1. Why the obvious answers are wrong#

1a. Blocking is a lie about your architecture#

input() blocks a process. A Slack button with a 30-second timeout blocks a Lambda. Neither of these things survives the human being asleep.

The moment you accept that approval latency is measured in hours, not seconds, blocking stops being an implementation detail and becomes an architectural falsehood. You are pretending a human is a fast function call. They are not. They are an unreliable external service with a p99 of “tomorrow morning”.

Futex’s create call never blocks. You get:

{"id":"dec_...","status":"pending","context_hash":"sha256:...","expires_at":"..."}

and the process is free. That is the whole design point and everything else follows from it.

1b. A chat message is not a decision#

Slack approvals feel great in a demo. They are terrible as a control because a chat message has no schema. There is no reject-with-reason, no separation of duties, no delegation with a depth limit, no escalation after four hours, no record of what exactly was approved, and — the one that actually bites — no binding between the approval and the thing being approved.

Someone clicks Approve. Two hours later, the agent’s plan has changed. The button click still says yes.

1c. A ticket is durable but dead#

Jira, ServiceNow, a Google Form. Durable, auditable, and completely inert. Nothing resumes. The agent has to poll a human-shaped workflow tool that was never designed to be polled, and the semantics of “done” are whatever the last person to touch the ticket believed.

1d. What I actually wanted#

Non-blocking creation with a durable ID.
Policy-driven, so the rules are data and not code scattered through eleven repos.
Cryptographically bound to the request, so “approved” means “approved this”.
Fail-closed, so silence is a denial and never an accident.
Auditable to the individual actor, months later.
Usable by an agent over MCP and by a human over email, without either party learning the other’s tooling.

Nothing off the shelf did that. So.

2. The design, and the two ideas that matter#

Most of Futex is unremarkable in a good way: PostgreSQL, Redis, FastAPI with async SQLAlchemy over asyncpg, a durable outbox for webhooks, a worker runtime with expiry/SLA/mail/webhook loops. 49 endpoints, 21 migrations, 71 test modules. Boring is the point.

Two ideas are not boring.

2a. The context hash — “approved” must mean “approved this“#

Every decision carries an immutable digest of what was actually asked:

canonical = json.dumps(
    {"context": context, "proposed_action": action, "policy_version": version},
    sort_keys=True,
)
context_hash = "sha256:" + hashlib.sha256(canonical.encode()).hexdigest()

It is returned on creation, and again on every read, including the terminal outcome. The consumer’s go/no-go is not status == "approved". It is:

stored_ack.context_hash == read.context_hash  AND  status == "approved"

This is the difference between an approval and a permission slip. A trading system I run gates going live on a Futex decision whose context binds the deployed commit and the traded symbol set. Change the commit, and the old approval stops matching. It does not get revoked — it simply stops being about the thing you are now doing.

One subtlety worth stealing, which I got from an agent that pushed back on my own advice: do not bind anything that moves on its own. Binding account equity into the hash gives you a stale-but-matching approval at the exact moment equity has halved. Re-check moving values at execution time; hash only the facts that cannot change without a deploy.

2b. Silence is a denial#

expiration_action defaults to timed_out, and timed_out is a denial. A sweeper terminalises expired decisions within ~15 seconds of their deadline. auto_approve exists, is opt-in, and I would not use it.

This sounds obvious until you notice how many approval systems default to nothing happening, which in practice means a pending row and a human who forgot. A hold nobody renews is not a decision. It is neglect with better paperwork.

2c. The rest, briefly#

Policies are immutable and versioned. No update route. To change one you create a new policy and repoint the ID. A decision snapshots the policy definition at creation, so an edit can never retroactively change the meaning of something in flight.
The outcome envelope is a normative terminal shape: result, reason codes, justifications, delegation summary, clearance state, audit_ref, and — added after I found it missing — an unconditional list of who acted. The default configuration made refusals attributable and approvals anonymous, which is precisely backwards: the branch that caused something was the one with no actor in it.
Idempotency is scoped to (tenant, submitter, key), not (tenant, key). Without the submitter, agent B posting a key agent A had used received A’s decision and acted on the answer a human gave to A’s question. Reproduced live before it was fixed.
Separation of duties, delegation with a depth cap, SLA escalation ladders, further clearance, amend-and-approve with the original hash preserved alongside an effective one, so an auditor sees both what was asked and what was approved.

3. It is a platform because the other platforms exist#

This is the part I am proudest of, and it is not really about Futex.

Futex is roughly 4,000 lines of application code. It behaves like something much larger because it does not implement most of what it needs. Each of these is a separate service I run, with its own agent, its own repo, and its own on-call story.

3a. auth — RBAC as an oracle, not a table#

Futex has no users table and no permission logic. A credential authenticates, and its permissions come from auth.rodmena.co.uk, keyed on a per-tenant namespace that Futex holds and never returns to clients.

The whole authorisation surface is:

def check_permission(principal, required):
    if not principal.has(required):
        raise HTTPException(403, "Forbidden")

Roles map to permission sets — agent_operator gets decision:create, decision:read, policy:read, webhook:manage and pointedly not task:act. That single omission is the strongest control in the system: a compromised service key literally cannot approve anything. Not “is prevented by policy from approving”. Cannot reach the route.

I like controls I can demonstrate with a status code. That one is 403, forever, on a bogus task ID, no state required.

3b. TokenGate — metering that admits when it did not meter#

The free tier is metered through TokenGate rather than a Redis counter I would have got wrong. The interesting engineering was not the integration; it was the failure semantics.

Metering sits after the last thing that can refuse a decision. It used to sit before the insert — and the step-instantiation code can still raise a 422 when separation-of-duties leaves no eligible assignee. That 422 rolls the transaction back, so no decision existed, while the charge had already left over the wire with no refund path. A tenant whose policy could not assign was billed on every failed attempt, and the symptom (quota exhausted) surfaced a long way from the cause (a policy that cannot assign).

And when TokenGate is unreachable, Futex does not block — it audits the absence of metering as an event. A silent metering outage is an unlimited free tier that nobody notices.

3c. mail-api — the human’s actual interface#

The reviewer never sees an API. They get an email with a signed, single-use link to a small review page. Rejecting requires a reason code and text. Approving under a step-up policy sends a six-digit code to the assignee’s address and asks for it back.

All of that is mail-api’s problem, not mine: templates, DKIM, suppression, delivery events, inbound parsing. I send a structured notification and get delivery telemetry back.

Inbound matters too. The free tier has no signup form — you email futex-freepass@mail.rodmena.co.uk from any address and get an API key back in seconds. That is mail-api’s inbound webhook plus a poller as a backstop, because at-least-once delivery plus a second collector beats either alone.

3d. RunFlow — approvals in front of real execution#

RunFlow runs container workloads in a hardened sandbox and orchestrates DAGs. Futex binds to a RunFlow node: the workflow reaches a gate, the gate raises a Futex decision, and the node stays parked until a human answers. Approve and the work runs. Reject and the workflow takes the other branch. Time out and it takes the branch the binding says a timeout means — which is configurable, and defaults to reject.

That last detail is the one that makes it a control rather than a pause button. An orchestrator that treats “no answer” as “carry on” has an approval gate in the diagram and none in production.

The RunFlow integration also taught me something about inter-platform error semantics. RunFlow migrated to auth-issued credentials and began refusing the old UUID-shaped ones with 401. Futex classifies that as legacy_credential — determined from the shape of the credential we sent, not by matching RunFlow’s error text. “This credential must be migrated” and “you are forbidden” need different human responses, and keying my audit trail on another platform’s copy would make my history their string to change.

3e. supervice, migretti, and the boring layer#

supervice supervises the API, the worker runtime, and the issue tracker as one process group with health checks and restart policy. migretti does SQL-first migrations — plain, reviewable SQL, no ORM-generated DDL I would have to read twice. Postgres lives off-box on its own host with TLS client-certificate auth. Redis is local, and — as I will get to — is deliberately not load-bearing.

3f. AgentBus — the one I did not expect to matter#

Every platform has an agent. They have real inboxes and talk to each other over an encrypted bus.

I expected this to be a novelty. It has become the single most productive code review mechanism I have. A worked example from one night, which I will come back to in §5.

4. The good: what actually works#

Non-blocking is correct. Agents raise decisions and get on with other work. The trading system polls every five minutes and treats the webhook as a latency optimisation, never as the authority — because at-least-once delivery is not exactly-once, and for a capital-affecting gate the pull path should be the source of truth.

Fail-closed defaults are correct. Expiry denies. Unassignable policies refuse at creation rather than creating an unactionable decision. A pinned credential is refused outright on every admin route.

The audit trail answers the question people actually ask, which is not “what happened” but “who did this, and what did they see”. Every terminal outcome names its actors, with timestamps, whether or not they wrote a justification.

MCP was the right bet. Ten tools over Streamable HTTP. A coding agent registers one server and can request sign-off before doing something irreversible without a single line of glue. The human-facing half is email. Neither party learns the other’s tooling, which is the only integration story that survives contact with real users.

5. The ugly: five bugs and what they have in common#

I am going to be specific here, because a post about correctness that does not enumerate its own failures is marketing.

Every one of these is the same defect, wearing different clothes: the system returned success for something that would never happen.

5a. The policy API stored controls it would never enforce#

A platform sent five policy definitions with every control field nested one level too deep — inside the step rather than the policy. Pydantic’s default is extra="ignore". All five returned 200. Every control silently fell back to a schema default.

The defaults were not uniformly safer than the intent. delegation_allowed defaults to True. So the silent failure opened a control its author had explicitly closed, on policies gating live trading and capital increases.

Worse: their separation-of-duties rule was spelled submitter_cannot_approve. My engine implements submitter_cannot_act. The key was stored verbatim, echoed back on read, and consulted by nothing. It survives any read-back that checks whether sod_rules is present.

Fix: extra="forbid" on every schema in a policy definition, plus a validator that refuses SoD rules the engine does not implement. Unknown keys are now a 422 naming the offending field with its full path.

5b. I published an event name that does not exist#

My own documentation named decision.terminated in three places, including the copy-paste registration example. The event is decision.terminal. Matching is exact string membership.

An integrator following my documentation exactly registered a filter matching nothing and received silence forever — no registration error, no delivery row to inspect.

They found it, and their diagnosis was better than mine: “a filter that matches nothing and a system with nothing to say look identical.” They refused to reason through the ambiguity and asked. It was the former.

There is a trap one level down that nearly caught me while answering: decision.approved and decision.rejected do exist in my source — as audit actions, never dispatched as webhooks. I built the corrected list from the actual emit() call sites rather than by grepping for quoted strings. The grep would have handed them a second wrong list.

5c. A security flag that only worked on one path#

require_step_up was read in exactly one place: the emailed review page. So a policy author could set it on a capital-affecting gate, watch the OTP challenge appear when approving from email, and reasonably conclude the decision was second-factor protected.

POST /v1/tasks/{id}/actions checked task:act and nothing else. And a one-click token minted for a policy that allowed one-click and required step-up approved with no challenge at all — from a link that travels in email.

The fix’s shape matters more than the fix: enforcement moved to the domain choke point every surface funnels through. A surface that does not declare a verified challenge is refused. That is the safe default for a surface nobody has taught about step-up yet, and it is why a future surface cannot reintroduce this. A control that lives in a surface is one new surface away from being absent.

5d. The bug was load-bearing#

Here is my favourite, because it is genuinely uncomfortable.

Webhook deliveries classify every 4xx except 408/429 as permanent — dead-lettered on the first attempt, no retry. That is wrong for 401, which describes a credential state at the receiver that a secret rotation or a verifier deploy changes on a timescale of minutes to hours. My retry ladder is 7h36m. Same order of magnitude. So I destroy, on first contact, exactly the deliveries whose failure was about to stop being true.

Straightforward fix. Except: I have no auto-disable on failing endpoints. active defaults True and is written by nothing in any failure path.

So today a dead endpoint costs one attempt per event precisely because 401 dead-letters immediately. The defect is the only thing limiting how hard I hammer a host that is refusing everything. Fix the retry classification alone and that becomes eight attempts per event, forever, with no breaker.

The bug is load-bearing, and the removal is invisible in the diff. Nothing in that patch says “this also removes the only limit on outbound volume against a refusing host”, because the limit was never written down as a limit. It only ever existed as a consequence.

5e. The fix would have blinded the alarm#

A peer platform shipped the same retry fix and found the next layer: all their health counters filter on delivered or dead. A delivery that is retrying is neither. So the fix would have made a broken endpoint invisible for the length of the ladder — on exactly the fields their documentation tells operators to watch.

Mine is worse. My ladder is 7h36m against their 1h36m, and my endpoint-health query exposes nothing at all about a delivery sitting in pending with attempts climbing. For seven and a half hours, a completely broken endpoint would look identical to a healthy idle one.

That is not a delayed alarm. It is no alarm.

6. The method: three ways a test lies to you#

Everything in §5 was found by one discipline, which I will state as plainly as I can.

A check that cannot go green cannot go red. If your check has never produced a positive, its negative is worthless. This is not a slogan; it is the most expensive lesson in the whole estate, and I have paid for it more than once.

There are three grades of it, in increasing subtlety.

1. Vacuous. The check cannot fail at all. d.get('x') returning None means “no such key” as often as “no such value”. Point every grep, query, and log scan at a known-positive first.

2. Scaffolding, not property. The check fails, but it asserts the mechanism rather than the outcome. Asserting that a 401 now schedules a retry passes identically in a world where every retry still 401s forever and nothing is ever recovered. Assert recovery, not scheduling.

3. Non-discriminating input. The check fails on a broken implementation and still proves nothing, because the input cannot separate the candidate designs. A disable threshold on a counter that resets on success: fail an endpoint continuously and both the correct design and the broken one pass. Fail it one-in-four, never consecutively, and only the correct one fires — the resetting counter never exceeds 1, so no threshold above 1 is reachable.

The rule for (3): ask not only can this test go red, but is there a wrong implementation that passes it — then pick the input the wrong one fails. The case you reach for first is usually the one that discriminates least, because it is the cleanest.

Two more, earned the hard way:

Verify through the product’s own interface, never the database. A direct query bypasses tenant scoping, redaction, and business logic, so it can show you data no caller could ever see. But the second reason is the one that bites: checking via the database proves your mental model, not the customer’s experience. Bugs live exactly in the layer you skipped.

Merging is not deploying. Tests and a green CI describe the source. Diff the source against what is installed on the host before you say “shipped”. When I fixed §5a I verified it by POSTing the original broken payload to the live host and reading the 422 — not by watching a test pass.

7. The thing I did not expect: peer review by machine#

The bugs in §5 were not found by me reading my own code. Most were found by another platform’s agent asking a question I could not answer without looking.

One night’s yield from a single liveness ping to my mail service — I asked it to reply pong:

Two defects in their service.
Three in mine, including §5d and §5e.
One claim of mine withdrawn: I had said my poller “almost certainly absorbed” some lost deliveries. The payloads proved otherwise. A right answer for a wrong reason is still worth correcting.
One claim of theirs corrected.
A design convergence neither of us had to re-derive.

And the sharpest single line, from the agent running my legal function, after I found an artefact in the header capture they had sent me:

An independent record is only independent of your errors. It carries all of mine, and mine are invisible to me for exactly the reason yours are to you.

That is a real limit on cross-checking, and I had not seen it stated. Two platforms verifying each other feels like it doubles the rigour. What it actually does is cover one party’s blind spots with the other’s, while leaving each party’s own instrument unexamined by the person best placed to examine it.

The practical rule I took from it: when a peer’s record differs from what your source says, check whether your code emits that field conditionally before treating the difference as data. If you emit it unconditionally, the difference is in their instrument — and telling them that is worth more than their capture was to you.

I also nearly reported a production host as having an unsynchronised clock, on the strength of stratum 16, reach 0 in the first two rows of ntpq -p. Those are pool placeholders. The real peers were four lines below, reach 377, offset −1.9 ms. I read the top of a table, got an answer that fit the story I was already telling, and almost stopped.

I published the misread alongside the corrected number, and the reason to do that is better than honesty: a corrected reading from someone who showed where they went wrong tells you which rows they looked at. A bare “clock is fine” tells you nothing.

8. What is still wrong#

In the spirit of the thing:

No auto-disable on webhook endpoints. A permanently dead endpoint is retried forever and the tenant is never told. Filed. Not fixed.
401 still dead-letters on the first attempt. Filed, specified, and deliberately not shipped alone — see §5d.
Webhook registration accepts event names that will never be emitted. The documentation half of that bug is fixed; the validation half is not. It is §5a all over again, in a different endpoint.
A 401 from my inbound webhook does not say which check failed. Signature mismatch and staleness return the same status with nothing durable recording which fired. Three separate incidents against that endpoint produced three shrugs. “Your secret is wrong” and “your retry arrived too late” are opposite problems with opposite fixes, and the sender — the only party who can act on either — is told neither.
One incident remains unexplained. 342 dead deliveries against 90 delivered, in a shape that a uniformly wrong secret does not produce. The rows have aged out. I would rather leave it open than close it with a plausible story.

None of these are shipping today. All of them are written down, with reproductions, and with the acceptance criterion that the test must go red before its green is trusted.

9. If you build one of these#

Never block. Return an ID. The human is an unreliable external service with a p99 of tomorrow.
Bind the approval to the request. A hash of what was asked, checked at the moment you act on it. Approve must mean approve this.
Do not hash anything that moves on its own. Re-check it instead.
Make silence a denial. And make the expiry action explicit in the policy, because a default nobody chose is a default nobody reviewed.
Put controls at the choke point, not in the surface. A control that lives in one route is one new route away from being absent.
Refuse what you will not honour. If you accept a field you never read, you have shipped a control that returns success and does nothing — the worst object in software.
Prove every check can go red before you trust its green. Then ask whether a wrong implementation would also pass it.
Give your services agents and let them argue. I am not being cute. It found more real bugs in one night than my last three self-reviews combined.

Futex is live at futex.rodmena.co.uk. The free tier has no signup form — email futex-freepass@mail.rodmena.co.uk from any address and a key comes back in seconds. Agent setup is at /claude; the API reference is at /docs.

It composes with auth, TokenGate, RunFlow and the Rodmena mail platform, because a control plane that had to implement identity, metering, orchestration and email would have been a worse control plane and a worse everything else.

The interesting problem was never “how do I ask a human”. It was “how do I make sure the yes I got is about the thing I am about to do.”

— Farshid