Highway Workflow Engine

The Bank-Grade, Postgres-Native Workflow Engine.

Highway is a durable workflow orchestration engine designed for systems where data integrity is non-negotiable. It enables you to build complex, long-running business processes—like payment flows, approval chains, or multi-step verifications—that are guaranteed to execute exactly once, never lose data, and transparently survive server crashes.

It turns your PostgreSQL database into a state-aware operating system for your code.

Why Highway? (The "Killer Features")#

If you are building critical backend systems, you likely already trust PostgreSQL with your data. Highway leverages that trust to give you capabilities that usually require massive, complex distributed systems infrastructure.

1. Highway vs. Airflow (The "Correctness" Gap)#

Apache Airflow is the industry standard for data pipelines (moving data from A to B on a schedule). However, it struggles with transactional business logic. * The Problem: Airflow manages tasks, not transactions. If a task charges a user's credit card and then the worker crashes before saving the "Paid" status to your database, you enter a "zombie state"—money is gone, but the order is still pending. * The Highway Solution (Atomicity): Highway runs your workflow steps inside the same database transaction as your business data. If the step crashes, the transaction rolls back completely. It is mathematically impossible to be in an inconsistent state.

2. Highway vs. Temporal (The "Complexity" Gap)#

Temporal is the gold standard for durable execution, but it demands a heavy "operational tax": running a separate, complex cluster (Cassandra/MySQL + ElasticSearch + Temporal Server). * The Problem: You now have two sources of truth: your App DB and the Temporal DB. Keeping them in sync is the "Dual Write Problem"—a notorious distributed systems challenge. * The Highway Solution (Simplicity): Highway IS Postgres. There is no second cluster to manage. Your task queue, your workflow state, and your business users live in the same database. It reduces architectural complexity by an order of magnitude.

3. Highway vs. DBOS (The "Flexibility" Gap)#

DBOS pioneered "database as operating system," but it often locks you into specific languages or deep framework integration. * The Problem: Tying your business logic deeply to a specific runtime environment can limit flexibility. * The Highway Solution (DSL): Highway uses a language-agnostic JSON/Python DSL (Domain Specific Language). You can orchestrate Python scripts, HTTP calls, LLMs, and Shell commands fluidly. It integrates with your stack, it doesn't become your stack.

Concepts for the "Not-Quite-Expert"#

We use specific distributed systems terminology. Here is what those terms actually mean for your business:

🔒 "Bank-Grade Atomicity"#

Imagine transferring money between two accounts. You subtract $50 from Alice and add $50 to Bob. * Without Atomicity: The server crashes after subtracting from Alice but before adding to Bob. $50 vanishes into thin air. * With Highway: The entire operation is wrapped in a "transactional bubble." If anything goes wrong (crash, network cut, error), the bubble pops and time rewinds. Alice keeps her $50. Partial failure is impossible.

😴 "Async Deferral" (The "Wait for Human" Pattern)#

Some tasks take a long time—like waiting for a manager to approve an expense report, or waiting for a 4K video to transcode (hours). * The Bad Way: A script sits in a loop while not approved: sleep(5). This wastes CPU, memory, and database connections for days. * The Highway Way: The workflow says "I'm waiting for 'Approval'" and stops. It saves its state to disk and completely frees up the worker. 3 days later, when the manager clicks "Approve," the API sends a signal, and the workflow wakes up instantly, remembers exactly where it was, and continues execution.

🧟 "Durable Execution" (Crash Proofing)#

The Scenario: Your server is running a critical 10-step onboarding process. It finishes step 5, and then the data center loses power.

Normal Script: The script dies. When the server reboots, you have no memory of progress. You might run step 1 again (sending a duplicate welcome email or charging the user twice).

Highway: When the server restarts, Highway checks the database, sees "Step 5 completed," and immediately resumes from Step 6. It operates like a "save game" for your code.

Get Started#

I am evaluating still what to do with Highway. Stay Tuned.