I Built My Own CI/CD, and the Interesting Part Wasn't the CI

On replacing GitHub Actions with something that owns almost nothing — and why that turned out to be the whole point.

I want to start with the objection, because it is the right objection and I have heard it from everyone I have described this to.

Why on earth would you build a CI system in 2026?

CI is solved. GitHub Actions is free for public repositories and cheap for private ones. It has a marketplace with ten thousand actions in it. It is maintained by people whose full-time job is maintaining it, and it is not going away. Writing your own is the software equivalent of churning your own butter.

I agree with all of that. I still did it, and I would do it again, and this article is my attempt to explain why in terms that are technical rather than romantic. The short version is that I did not set out to build a CI system. I set out to build one component — and the reason the project was worth doing is that the component turned out to be about seven thousand lines of decision-making sitting on top of infrastructure I already had.

The long version needs a detour through a Docker socket.

Part I — The reason#

1. The socket that changed my mind#

Here is the thing that actually started this.

I evaluated the self-hosted options properly. Woodpecker, Drone, and their relatives. They are good software. I want to be clear that nothing below is a criticism of them, because the problem I hit is not a bug in any of them — it is a consequence of the architecture that all conventional CI shares.

Self-hosted CI agents run your pipeline steps in containers. To do that, the agent needs to talk to a container runtime. The conventional way to arrange that is to mount the Docker socket into the agent:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

Now look at what else lives on that host. In my case: the identity provider, the mail API, the message bus, the ledger, and the public website. Mounting that socket grants root on all of them to anything a pipeline can be persuaded to execute. Not root in the container — root on the host, because the Docker socket is a root-equivalent API by design. docker run -v /:/host and the isolation is decoration.

And “anything a pipeline can be persuaded to execute” is a much larger set than “code I wrote”. It is every dependency of every project, every transitive dependency of those, and every pull request that a contributor opens. One typosquatted package in a requirements.txt and the attacker is not in your CI — they are on the box holding your OIDC signing keys.

You can mitigate this. Rootless Docker, a separate runner host, careful network segmentation. These are real mitigations and people run them successfully every day. But I kept coming back to a simpler observation: I already had a service whose entire job is running untrusted containers safely. I had built it. It was in production. It had a hardened sandbox, per-run CPU and memory limits, timeouts, log capture, and an authenticated HTTP API.

So the question stopped being “which CI server should I self-host” and became “why would my CI system touch a container runtime at all?”

That reframing is the whole project.

2. What was actually wrong with the hosted option#

The Docker socket is the reason I did not self-host something conventional. It is not, by itself, the reason I left GitHub Actions. That is a different argument and I want to make it honestly, because the case against Actions is weaker than CI-independence advocates usually claim.

What Actions genuinely does better than what I built: the marketplace, the enormous body of documentation and Stack Overflow answers, matrix builds across operating systems I do not own, and the fact that it costs me no operational attention whatsoever. If you are one person shipping one product, use Actions. I mean that sincerely.

Here is what pushed me off it anyway.

The credential surface. In a hosted CI system, your deployment credentials live in the CI system. That is the whole model — you put a secret in the settings page, and a workflow step reads it. Which means every step of every job of every workflow, on every branch, is a place that credential can be read from. ${{ secrets.DEPLOY_KEY }} is one line in a YAML file that anyone with write access can add, and pull requests from branches in the same repository can often reach it too.

The industry’s answer is OIDC federation, and it is a genuinely good answer. But it moves the trust rather than removing it: you are now trusting GitHub’s token issuance and your cloud provider’s trust policy, and misconfigured trust policies are their own well-populated category of incident.

The absence of a spine. This is the objection I feel most strongly and can defend least rigorously, so take it as opinion. In Actions, the unit of composition is a third-party action from a marketplace. uses: some-org/some-action@v3 is a supply-chain dependency that executes in your build with access to your secrets, pinned by a mutable tag. The ecosystem is a genuine strength and I understand why it exists. But every one of those is code I did not write, running where my credentials are.

And the honest one: I wanted the platform. I have spent a couple of years building a set of services that do specific things well — a container executor, a workflow engine, a quota service, an authorization service, a storage layer. Each was built for its own reason. But a platform is only a platform if the pieces compose, and nothing tests composition like making them carry something real. CI is an excellent forcing function, because it exercises every one of them under load, on a schedule, with immediate feedback when a seam is wrong.

If the pieces could not carry my own CI, I would rather find that out from my own builds than from a customer.

Part II — The design#

3. The component is called the conductor, and it owns almost nothing#

The central design decision, and the one I am most pleased with, is a subtraction.

The thing I built is called the conductor. Here is its complete job:

Receive a webhook. Decide what should run. Ask other services to run it. Report the result back.

That is it. Read the list of things it explicitly does not own:

The house rule I wrote into the repository’s standing orders is deliberately blunt:

If you find yourself writing a scheduler, a container supervisor, a retry loop or an S3 client, stop — you are rebuilding something that exists.

Whole categories of hard problems are simply absent from the codebase. There is no scheduler in it. There is no container supervision, no S3 client, no retry loop, no backoff implementation. The result is about 7,000 lines of Python across 41 modules, and every one of those lines is about deciding something rather than doing something.

That is the actual answer to “why build a CI system in 2026”. I did not build a CI system. I built the ~7,000 lines that were missing from a CI system I already mostly had.

4. How the pieces fit#

  GitHub
    │  POST /hooks/github   (signature verified over RAW bytes)
    ▼
┌─────────────────────────────────────────────────────────────┐
│                        CONDUCTOR                            │
│                                                             │
│  persist delivery, dedupe on X-GitHub-Delivery              │
│  fetch .ci/pipeline.yml at the BUILD sha                     │
│  validate, expand matrix, persist definition + plan          │
│  admission control (L1): concurrency groups, per-repo cap    │
│  stabilize (embedded): one stage per job, edges from needs   │
│  per job (L2): reserve cpu/mem → mint scoped token → submit  │
│  check-run OUTBOX drains to GitHub, ordered, at-least-once   │
└──────────────┬──────────────────────────────────────────────┘
               │ submit run (runner script as the command)
               ▼
        ┌─────────────┐
        │   RunFlow   │  container, limits, timeout, isolation
        └──────┬──────┘
               │  webhook = signed HINT → schedules a poll
               │  polling  = AUTHORITY for all state
               ▼
        ┌─────────────┐
        │  datashard  │  logs + artifacts → S3
        └─────────────┘

A few decisions in that picture are worth pulling out, because each one was a fork in the road.

One job is one container, and steps live inside it. The rejected alternative was one container per step, which sounds more granular and composable. It is a trap: every step boundary becomes a filesystem-persistence problem, and you spend your life shipping the workspace between containers. Steps execute inside a single generated runner script, which emits nonce-prefixed markers so per-step timing and exit codes can be parsed out of the log afterwards.

Polling is the authority; the webhook is a hint. RunFlow signs its webhooks and I verify them, but I never apply the payload as state. A webhook arrival schedules an immediate poll, and the poll’s answer is what gets recorded. This is one of those decisions that costs a little and repays constantly: a lost webhook delays a result, it never loses one. Reconciliation is not a special error path bolted on later — it is the normal path, running all the time, which means it is continuously tested rather than being the code that runs for the first time during an incident.

Two levels of admission, both in PostgreSQL. Level one admits pipelines (concurrency groups, per-repo caps). Level two reserves CPU, memory and a run slot for each job. Both are transactional, with the check and the state transition in the same transaction, so two instances racing for the last slot cannot both win. Crucially, admission is not an in-memory semaphore — it is rows in a database, because in-memory counters do not survive a restart and a CI system that forgets its budget after a deploy will happily overcommit the host.

The check-run outbox. GitHub goes down. Rate limits happen. If reporting is inline with execution, an API outage loses build results. So results are written to an outbox table and drained by a separate loop, ordered per check run, at-least-once, coalescing superseded updates. The build result survives GitHub being unavailable, which is exactly what you want at 3am.

5. The one that makes the whole thing safe#

Here is my favourite mechanism in the system, and the clearest example of what a purpose-built CI can do that a general one cannot.

When a job needs to check out your repository, it needs a credential. The conventional answer is a deploy key or a PAT sitting in the CI system’s secret store, with access to everything it was granted.

Instead: at the moment a job is submitted, the conductor mints a fresh GitHub App installation token, scoped to exactly one repository, with contents: read only, and injects it as an environment variable for that run alone. It is registered in the redaction set so it is scrubbed from every log surface. It expires in an hour, and the job usually finishes in minutes.

So a leaked build token reads one repository, read-only, for less than an hour. Compare that to a leaked deploy key.

That is a small piece of engineering — a few dozen lines — and it is only possible because the CI system is mine and can make a decision at submit time. It is the kind of thing you cannot retrofit into a general-purpose product, because the general-purpose product does not know that “one repository, one hour, read-only” is the right shape for your threat model.

6. Deployment does not live in CI#

This is the part where my design differs most from convention, and it started as a security argument and ended up being a correctness argument too.

CI does not deploy anything. There is no deploy job. The conductor holds no deployment credentials, no SSH keys, no cloud tokens.

Instead, the deploy host runs a small poller. It asks the conductor’s API: is there a succeeded pipeline, on main, for this repository, whose commit is the current tip of origin/main, and is it different from what I last deployed? If all of that holds, it deploys — itself, locally, with credentials that never leave the box.

Three gates, and each one is there because of a specific failure:

1. Succeeded, on main. The obvious one.
2. Equals the current tip. Without this, a green-but-superseded commit deploys and rolls back a newer push that is still in CI.
3. Different from the last-deployed marker. Idempotency. One green pipeline deploys exactly once.

The security argument is that a credential that is not in CI cannot leak from CI. Every step of every job on every branch is a place a secret can be read; keeping deployment credentials out of that blast radius entirely is worth an architectural change.

The correctness argument is subtler and I did not anticipate it. Because the deploy runs on the target, the checks that run alongside it run on the target too — against the real daemon, the real filesystem, the real service manager, immediately before traffic. A colleague migrating another service onto this platform put it better than I would: those checks currently run on a hosted runner that resembles production, and under this split they run on production itself. If they fail, the poller does not promote.

That is not a concession to the architecture. It is better than what it replaced.

Part III — The platform underneath#

The conductor is the newest piece, but it is the least interesting one. What makes it small is what it stands on.

7. RunFlow — container execution as a service#

RunFlow runs containers in a hardened sandbox: read-only root filesystem, an unprivileged user, per-run CPU and memory limits, timeouts, kill and pause, log capture, and an authenticated API. It was built to run untrusted, agent-generated code, which is a strictly harder problem than running CI jobs.

The conductor submits a run and polls it. That is the entire integration. No Docker socket, no runner agents to install, no container lifecycle code in my CI system at all.

The hardening is real enough to be inconvenient, which is how you know it is real. The read-only root filesystem means PostgreSQL cannot initdb, so a job that needs a database service cannot run under the default sandbox. The resolution was a narrow one: a specific allowlist of maintained images, verified by digest, that get a writable root filesystem — and RunFlow re-verifies the image against its own allowlist rather than trusting the caller’s claim. A wrong prefix from the conductor gets a 403, not a silently weakened sandbox.

I like that shape a lot. The caller requests a relaxation; the executor decides.

8. stabilize — the DAG engine#

Jobs form a graph via needs. Something has to own ordering, fan-out, fan-in, crash resume, and per-stage retries with backoff. That is a genuinely hard piece of software and I had already written it.

The integration detail I am proudest of: waiting for a container is a suspend, not a blocked thread. When a job is submitted, its stage suspends and the worker is released. When the poller sees the run go terminal, it signals the stage awake. No worker is ever held for the duration of a build. A 20-minute test suite occupies zero workers for 20 minutes.

That single property is why the whole thing runs comfortably on modest hardware.

9. datashard — logs and artifacts#

Container logs die with the container. Artifacts need to outlive the run. datashard is an Iceberg-like storage layer over S3 with partitioning and time travel, so retention is a cheap partition delete rather than a bespoke lifecycle policy.

Logs are scrubbed at copy time, not just at display time. A secret must be masked before it lands in durable storage, because a redaction that only runs on the way out is one forgotten code path away from being no redaction at all. I learned that one the way everybody learns it.

10. TokenGate, auth, and Futex — the self-service story#

These are the pieces that make the difference between “CI for my team” and “CI as a platform”, and I am going to be careful here to distinguish what is running from what is specified, because I think overstating that is the most common way technical writing becomes dishonest.

Today, the access-control model is network position. The internal API binds where only the team can reach it. That is a real model with real virtues — no tokens to leak, no sessions to fix, no permission bugs — and for a single team it is defensible. Its cost is that it can express exactly one rule: the team can do everything, everyone else can do nothing.

Self-service breaks it, and the design for that is written and not yet built:

• auth carries platform roles — ci-admin, ci-auditor — coarse RBAC by name. It is deliberately not the home for per-repository tenancy, which is row-level and would mean a permission name per repository kept in sync with GitHub forever.
• GitHub is the trust anchor for repository authority. GitHub already knows, authoritatively, who may administer which repository. Mirroring that into a local table gives you a cache with no invalidation: when someone leaves the org, CI keeps honouring the stale row until a sync runs. So authority is derived per session and cached for minutes, and a revocation on GitHub is a revocation here.
• TokenGate carries quotas — concurrent pipelines, monthly build minutes, storage — because the moment anyone with a repository can start work, the estate budget needs an explicit per-tenant cap or one repository consumes everything. Never a hand-rolled Redis counter. Quotas must be tested in both directions: that they block at the cap, and that work resumes when it should. A cap tested only for blocking is how a queue deadlocks forever.
• Futex carries human approvals, for the pipelines that should pause for a person.

Writing that spec produced the single most valuable finding of the project, and it came from pointing a probe at production from outside rather than reasoning about the design. The UI needed data; the internal API was the only source; so nginx had been made to proxy a curated subset. Which meant the read API was on the internet without an identity model to gate it — private repository names, commit SHAs, and complete build logs, publicly readable.

Nobody had made a bad decision. Each step was locally reasonable. The system had taken half a step toward self-service without the half that makes it safe.

The measurement nearly went the other way, too, and this is the part worth keeping. nginx’s try_files $uri /index.html returns 200 for every path — so /admin/secrets also answers 200, with the SPA shell. Status code alone cannot distinguish an exposed API from a fallback page. Every row of that finding had to be discriminated by content-type. A probe that checked status codes would have reported the system safe, confidently, while it was leaking build logs.

11. The container registry#

Images have to come from somewhere, and where they come from is a trust decision rather than a convenience one.

The maintained images — a Python one and a Node one — are the only images that get the relaxed sandbox, and RunFlow pins them by digest, not by tag. That distinction is the entire security property. A tag is a mutable pointer; python:3.12-slim today and python:3.12-slim next month are different bytes. Trusting a tag means trusting whoever can move it. Pinning a digest means trusting a specific artifact you have actually looked at.

The cost is a rotation discipline that is easy to get wrong: rebuilding an image produces a new digest, and the new digest has to be registered with the executor before jobs can use it. Get the order wrong and every build fails with a 403. That is the correct failure — refusing an unrecognised image is exactly what the allowlist is for — but it means image rebuilds are a small coordinated operation rather than a docker push.

I will say plainly that this is the least finished part of the story. There is a house registry effort; the estate today pulls maintained images from a public registry namespace; and moving the digest pins from one to the other is a migration I have not done. When a colleague asked me for the registry hostname so they could re-pin their trusted-image allowlist, the correct answer was that I did not have it and would not guess — because a hostname I invented would have ended up written into a security control on a box that runs untrusted workloads. Being slow there is much cheaper than being confidently wrong.

Part IV — What it cost, and what I learned#

12. Testing a thing whose job is testing things#

CI is the system that enforces quality for everything else, so it cannot be the sloppiest component in the estate. That produced a house rule I now apply everywhere:

A check that cannot go green cannot go red.

Every check that asserts something is absent, healthy or correct must first be demonstrated to fail on a known-bad case. A test that passes because it queried the wrong field, matched an empty string, or ran against an empty database is worse than no test, because it reports safety with total confidence.

In practice this means: before trusting “no secret leaked into the logs”, write a log line containing a secret and prove the check catches it. Before trusting “the circuit breaker is Postgres-backed”, write a row and read it back from Postgres — because the breaker library silently falls back to in-memory storage on any connection problem and only logs a warning, which is a healthy-looking service whose breaker state evaporates on restart.

The corollary took longer to learn: verify through the product’s own interface, never the database. To prove a pipeline worked, fetch the check run from GitHub and read its conclusion. Not the local pipelines table. A feature “verified” against the database while the public path is broken is worse than untested, because it comes with false confidence. Bugs live exactly in the layer you skipped.

13. Four failures worth writing down#

I would rather this article be useful than flattering, so here are the ones that hurt.

The connection pool that was fine until it wasn’t. The DAG engine executes task handlers in a thread pool. The database pool is opened on the main event loop. Each handler was calling asyncio.run(...) — a brand-new event loop per call, in an arbitrary thread — against that shared pool.

This looks completely fine when idle, because the pool hands out a free connection without awaiting anything. The hazard is the wait queue: when no connection is free, the borrower parks on a future created on its own throwaway loop, and whoever returns a connection resolves that future from a different loop. Nobody wakes up.

Same load, same pool, only the loop changed:

It surfaced as builds failing for reasons that had nothing to do with the user’s code — the flaky-CI generator, and it gets worse exactly when the system is busy. The fix marshals every database call onto the pool-owning loop. The lesson generalises: an async connection pool is loop-bound, not merely thread-unsafe, and you will never find it by counting successful queries. You find it by forcing more borrowers than the pool has connections.

The outbox that wasn’t atomic. The docstring said the outbox row was written “in the same transaction as the state change it reports”. Atomicity is the entire reason an outbox exists. It was not — each call opened its own connection, and I proved it by asking PostgreSQL for the transaction id of each statement. Three different ids. A crash in between leaves a job terminal with no result ever queued: a pull request that can never merge.

A week of silence. The service was killed by the kernel during a host-wide memory squeeze. It stayed dead for seven days. The dashboard still served — it is static — while every API call behind it returned 502, so it looked alive to a casual glance. The cause was one missing flag: the service supervisor was told to launch the process but not to restart it, and the host had not rebooted in seventeen days, so nothing ever brought it back.

The fix is one flag. The lesson is that I had verified the service starts, and never verified it restarts. Those are different properties and only one of them was tested. Now: kill it with SIGKILL and watch it come back in eight seconds, and — the direction people forget — stop it deliberately and confirm it stays stopped, because a supervisor that restarts a deliberate stop is its own outage.

The log that lied. This is the worst one, because it is a failure of honesty rather than of correctness.

A colleague onboarded a new repository and its first build failed. They got: exit code 1, no failure reason, an empty step list, and a log that ended mid-sentence — while the API reported eof: true, meaning “this is the whole log”.

It was not the whole log. The executor still held over a hundred lines, including the only line that explained the failure:

ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

Out of disk, not memory — and it was initially read as an out-of-memory kill, which would have led to raising the memory ceiling and fixing nothing. The tell was timing: the run lasted fifteen seconds, far too fast for memory exhaustion during dependency resolution.

The root cause was small. The runner script sets a spacious workspace, but never set TMPDIR — so pip unpacked wheels into a 64 MB temporary filesystem while the roomy workspace sat unused. It had been latent forever because the only previous heavy consumer used a different package manager that builds inside the project tree.

But the bug I actually care about is eof: true. A truncated log that announces truncation is recoverable. One that claims completeness it has not established is worse than no log at all, because the reader stops looking. It is the same disease as a test that cannot fail: reporting a property you did not verify. Someone had to ask another engineer to read their build for them, which is precisely the moment a CI system stops being trustworthy.

14. Was it worth it#

Let me try to answer the opening objection with the benefit of everything above.

If you are one person shipping one product: no. Use GitHub Actions. The operational attention this requires is real, and I have described four incidents that a hosted product would have absorbed on your behalf without you ever knowing.

It was worth it for me, for reasons that are mostly not about CI:

• The Docker socket problem is real, and I did not have to solve it, because the container executor already had.
• Build credentials are now per-job, per-repository, read-only, and expire in an hour. Deployment credentials are not in CI at all.
• The platform got tested by something that runs on every push. Four defects surfaced that would have found a customer eventually — and they surfaced in my builds first.
• It is about 7,000 lines, because everything hard is somebody else’s problem. That number is the whole argument. If it had been 70,000, this would have been a mistake.

And the thing I did not expect: the discipline transferred. “A check that cannot go green cannot go red” came out of testing this system and now applies to everything I write. So did “verify through the product’s own interface, never the database”, and “test every limit in both directions”, and “a truncated log must announce its truncation”.

Those did not come from reading about good practice. They came from four specific times this system reported something reassuring that was not true — and being able to fix the reporting rather than work around it is the actual, non-romantic argument for owning the thing that watches your code.

The estate this runs on — the container executor, the DAG engine, the storage layer, the quota and authorization services — is a set of components I have been building for a while. This article is about the ~7,000 lines that made them into a CI system. If a piece of it is useful to you, take it.