RunFlow: building a place to run code you don't trust

• I should declare my position up front, because it colours everything below. I did not set out to build a container platform. I set out to stop hand-running code that a language model had just written, on a laptop that also had my SSH keys on it. RunFlow is what that turned into.

• Every part of this is measured against the running service, not against a design document. Where I say something failed, it failed on a real host with real traffic, and I have the scar to prove it.

The problem, stated plainly#

An agent writes code. Now what?

You have three bad options and everyone reading this has used all three. You run it on your own machine and hope. You paste it into a notebook on some hosted thing and hope slightly less. Or you build a bespoke little runner for this one project, which works until the second project.

The requirement is easy to state and annoying to satisfy:

• Run an arbitrary container image, now or at a specific time, or as one node in a graph of steps.

• Give it the internet, because a useful agent step fetches an API or calls a model.

• Give it nothing else. Not the host. Not the database on the host. Not the other run that started four seconds ago.

• Cap what it can consume, per customer, and mean it.

• Tell me everything that happened, over an API, in a form another program can consume.

That last point is the one that quietly kills most of the alternatives.

Why the obvious answers are the wrong answers#

I tried the obvious things first. I want to be fair to them, because each is genuinely good at the job it was built for — and none of them was built for this one.

So: build it. Not because the alternatives are bad, but because none of them answers the question.

Five decisions that shaped everything#

1. One process, one owner — deliberately#

RunFlow runs as a single process that owns the scheduler loop, the worker pool, and the DAG evaluator. Not a fleet. One.

This raises eyebrows, so let me defend it. Every piece of state lives in PostgreSQL and the in-process loops are stateless — they re-derive progress each tick and recover a crash by re-reading the database, never from memory. Given that, a second instance against the same database does not double throughput; it double-dispatches. Two schedulers both see the same eligible run and both launch it.

The guard is a boot-time Postgres advisory lock. A second instance exits rather than starting, and a watchdog fences the process if it ever loses the lock.

I like this decision more than I expected to. Horizontal scaling is a real answer to a real problem, and it is also the most common way I have seen small teams import a distributed systems problem they did not have.

2. Durable state, reactive control#

Every state transition is a row write inside a transaction that also appends an event and enqueues any webhook deliveries. Event sourcing plus a transactional outbox, both in the same commit as the state change.

The property this buys: there is no window where the run finished but the webhook didn’t get queued. Either both happened or neither did. If you have ever debugged a system where the job succeeded and the notification vanished, you know exactly what this is worth.

3. Capacity is reactive, never predictive#

timeout_seconds is a ceiling, not an estimate. I refuse to guess how long a container will take, because I would be wrong, and being wrong there means either killing healthy work or holding slots for corpses.

So when you submit past your concurrency limit, you do not get a 429. You get queued, with a position and an ETA:

{"track_id": "01KY…", "status": "scheduled",
 "scheduling_info": {"queue_position": 3, "eligible_start": "2026-08-30T09:00:00Z"}}

Rejecting work at the edge pushes the retry loop into the caller, where it is written badly in a hurry. Queueing it keeps the backpressure where I can see it.

4. Fail closed, everywhere, without exception#

• Auth service unreachable → 503. Not “assume yes”.

• Metering unreachable for a metered tenant → the submit is refused, not allowed unmetered.

• Disk below the floor → stop starting new work.

• The container egress firewall not loaded → the service refuses to start. runflow.service has a hard Requires= on the firewall unit. No firewall, no RunFlow.

That last one is my favourite line of systemd I have ever written. The failure mode it prevents — service comes up fine, firewall silently didn’t — is exactly the kind that looks healthy for six months.

5. Tenant-scoped everything#

Every data-access function takes tenant_id as its first argument and filters on it. The bearer secret resolves to a tenant at the door and only the id travels inward. There is no code path that can “helpfully” widen a query, because there is no query without the scope.

The sandbox, and the bug I am most glad I found#

Every run container gets:

--cap-drop ALL   --read-only rootfs   --security-opt no-new-privileges
--pids-limit 256   --memory (=--memory-swap, so: no swap)   CPU quota
tmpfs /tmp (64 MB)   never --privileged

Limits are clamped to min(tenant quota, host ceiling) at submit time — and omitted limits are defaulted first, then clamped, so you cannot exceed your tier by leaving a field out. That ordering is one line of code and it was a real hole.

The part that took three attempts: egress#

A run needs the public internet. A run must never reach the host, the host’s daemons, the private network, or another run. A plain Docker bridge gives you the first and cheerfully gives you all the rest too.

The shape that works: a shared bridge with inter-container communication off, plus a host firewall in DOCKER-USER that permits the internet and REJECTs RFC1918, link-local (169.254/16 — the cloud metadata endpoint) and CGNAT space.

And here is the trap, which cost me a live hole. My first firewall rules were scoped by source subnet — -s 172.17.0.0/16, the container network. Clean, readable, obviously correct.

They matched nothing.

Docker assigns each new bridge the next subnet from its pool. The first run is on 172.17. The second is on 172.18. By the fifteenth you have exhausted the default pool and Docker quietly allocates out of 192.168.0.0/20 — home LAN space. My rules were guarding a subnet that nothing had used since the first run of the day, and the ruleset loaded without error and read as correct.

The fix is to match on destination only. DOCKER-USER is traversed solely by container-forwarded traffic, so the “this came from a container” half is already implied by the chain you are in. A source match buys nothing and invites this exact failure.

Verified after the fix, from inside a real run:

internet          HTTP 200                        allowed, as advertised
database host     BLOCKED ConnectionRefusedError  was reachable
another machine   BLOCKED ConnectionRefusedError  was reachable

I have written that lesson into the docs in the rudest language I could justify, because it is not a RunFlow quirk — anyone writing per-network container policy will hit it.

Trust is bound to a digest, not a tag#

Some images are allowed a relaxed sandbox. The allowlist is exact-match — an image ref that merely prefixes an entry is not trusted, because ci-python:3.12.evil should not inherit anything from ci-python:3.12.

But an allowlisted tag is still mutable. Anyone who can push can repoint it. So trust is additionally bound to the resolved image digest: the worker reads the actual pulled image’s sha256 at execution and grants the relaxed sandbox only if it equals the operator’s pin. Missing pin or mismatch both fail closed to the hardened sandbox, loudly.

Which brings me to the registry.

Why I built my own container registry#

Digest pinning only means something if you control what the digest points at.

containers.rodmena.co.uk is a private OCI/Docker-compatible registry — FastAPI, PostgreSQL, S3-compatible object storage, Redis. It exists for four reasons:

1. Supply chain. A trusted-sandbox image pulled from a public registry is trusted only as far as that registry’s account security. Mine is mine.

2. Rate limits are not a strategy. Public registry pull limits are a fine business model and a terrible dependency for a service that pulls on every cold run.

3. Pull-only credentials. The runner gets a credential that can pull and cannot push. Obvious, and impossible to arrange on a registry you don’t own.

4. It made the whole estate composable. Push a CI image once, pin it, run it under the trusted sandbox.

The interesting engineering is in the storage layer. Blobs are content-addressable — the digest is the path, so an identical layer pushed by ten repositories is stored once. Uploads are chunked through Redis to satisfy S3’s 5 MB minimum part size, with a running SHA-256 finalised at PUT. Downloads are 307 redirects to pre-signed URLs, so the API never proxies bytes it doesn’t need to touch.

Tag re-pointing is the one place that must be strictly correct — SELECT … FOR UPDATE, so two concurrent pushes to the same tag serialise instead of interleaving. A tag that points at half of one manifest and half of another is not a bug you want to debug at 2am.

The part I am actually proud of: RunFlow is small#

Here is the thing that makes RunFlow work, and it is not a RunFlow feature.

RunFlow has no users table. No quota table. No billing code. No email sending. No migration framework of its own.

It has none of those because each one is a separate product that I built once and now use everywhere:

Every one of these was extracted because I needed it twice. That is the only good reason to build a platform component, and it is a much higher bar than “this would be nice”.

The compounding effect is real and slightly addictive. When I added tiers to RunFlow, I did not design a quota system — I wrote a tier profile and TokenGate enforced it. When I needed per-operation permissions, I did not write a permissions table — I named the permissions. Adding the third consumer of a platform costs almost nothing, and that is where the investment finally pays.

The blunt version: most teams re-implement auth, quotas and metering in every service, badly, and then discover the four implementations disagree about what a customer is allowed to do. Build it once. Charge yourself the integration cost.

Agent-native, and I mean it structurally#

RunFlow speaks REST and MCP, and I hold them to parity: 62 MCP tools covering every tenant-facing operation the REST API exposes.

The interesting bit isn’t the tools — it’s the test. There is a test that maps every tenant-facing route in the OpenAPI document to its MCP tool, and fails the build if you add a route without one. Parity that depends on discipline decays in a fortnight. Parity enforced by a red build survives.

I extended that recently to fields, after finding a case where the tool existed, the route existed, and the tool schema was missing an argument — so a well-behaved client silently stripped it before sending, and the call succeeded having done something other than what was asked. Which is a good segue.

What bit me#

The honest section. Every one of these is a real incident.

A check that cannot go red is not evidence. This is the single most expensive lesson in the project. I have shipped tests that passed on absence — greps pointed at fields that were never populated, a probe recorded as “skipped” but counted as “passed”, an assertion whose control case also passed. Now the rule is: before I trust a green check, I break the thing it watches and confirm it goes red. It sounds paranoid. It has caught something roughly every other time I have bothered.

A gate that stops running is worse than one that fails. The type checker was clean in August, and then it wasn’t — and nobody knew for eighteen days, because the CI workflow file was still sitting in the repository looking exactly like a gate. It wasn’t failing. It wasn’t running. The file being present is evidence of nothing.

Merging is not deploying. I fixed a domain reference across the codebase, pushed, and told a colleague it was done. The published file kept serving the old value for hours, because the deploy had failed in a way that left the previous release in place. Source and served are two different claims and I had made only one of them. Everything user-facing is now rendered from configuration by the application and asserted in the test suite — because a static file cannot assert anything about itself.

One decision, made twice, will drift. A run’s network was created based on a mode that a trusted run overrides, and torn down based on the same mode read fresh from config — which never saw the override. So every trusted run leaked its network, and about fifteen of them exhaust Docker’s address pool. The fix isn’t a better condition, it’s not asking the question twice: record what you created, delete exactly that.

Silently ignoring an unknown field is a lie. A request model that accepts and discards {"sandbox": "trusted"} returns a cheerful 202 for a run that is not sandboxed the way you asked. It now returns 422 and names the field. That is a breaking change and I shipped it anyway, because “it worked” when it didn’t is the worst answer an API can give.

Would I do it again#

Yes, and mostly the same way. The two decisions I would defend hardest are the ones that looked most conservative at the time: single process with a real lock, and delegating auth, quotas and metering to services I already owned instead of adding four more tables to this one.

The decision I would change is that I would have written the “can this check actually fail?” discipline into the process on day one, rather than earning it four incidents at a time.

If you want to poke at it: https://runflow.rodmena.co.uk. The free tier is an email away, and /llms.txt will tell your agent everything it needs in one fetch — which, appropriately, is now served by the application itself and asserted in CI.