Update: I have abandoned using Absurd. It's buggy and the maintainers are not very welcoming to contributions. I am building my own.

The Integration Model

This document outlines a strategy for merging highway_dsl and Absurd. These two systems are not competitors; they are complementary components operating at different layers of abstraction.

  • highway_dsl is a Declarative Workflow Definition Layer. It provides a Python-native (and YAML-serializable) Domain Specific Language (DSL) to define complex workflow graphs. It specifies the "what": the tasks, their dependencies, conditional branches, parallel execution paths, and retry policies. It is the blueprint of the workflow.

  • Absurd is an Imperative Durable Execution Runtime. It provides a minimal, Postgres-based SDK to execute long-running functions reliably. It specifies the "how": durable steps, checkpointing of state, crash resilience, and asynchronous primitives (sleeps, events). It is the durable factory worker.

The integration strategy is to use Absurd as the durable execution engine for a workflow interpreter that runs graphs defined by highway_dsl.

The Core Pattern:

  1. A workflow is defined using highway_dsl's Python API or loaded from a YAML file.
  2. This workflow definition (the graph object or its serialized YAML/JSON representation) is passed as the initial parameter to a single Absurd task.
  3. This Absurd task is a generic workflow interpreter. It walks the highway_dsl graph node by node.
  4. Each node in the highway_dsl graph (e.g., Task, Conditional, Parallel) is mapped to a corresponding set of Absurd SDK calls (e.g., ctx.step, ctx.waitForEvent, absurd.spawn).

This model provides a clean separation of concerns: highway_dsl handles the business logic and flow definition, while Absurd handles the state persistence, checkpointing, and resilience.

2. WHAT

highway_dsl (The "Blueprint")

highway_dsl excels at defining workflow structure.

  • Graph-Based: Workflows are Directed Acyclic Graphs (DAGs) composed of nodes like Task, Conditional, Parallel, and Start/End.
  • Declarative: The developer declares the flow, rather than writing the imperative logic to glue it together. This is evident in its fluent Python API and YAML serialization (tests/data/car_factory_workflow.yaml).
  • Rich Flow Control: It natively supports complex patterns:
    • Dependencies: Tasks run based on the completion of upstream tasks.
    • Conditionals: if-then-else logic for branching.
    • Parallelism: A Parallel node to execute multiple branches concurrently.
    • Retries: A RetryPolicy can be attached to tasks, defining max_attempts, delay, and backoff (tests/test_workflow_dsl.py).
  • State Management: The DSL manages the passing of state (outputs from one task to inputs of another) within its in-memory execution model.

Absurd (The "Runtime, aka: HOW")

Based on its provided summary and SDK, Absurd excels at executing code durably.

  • Imperative & Step-Based: Execution is a standard function that calls await ctx.step(...) for any operation that needs to be durable and idempotent.
  • Durable Checkpoints: The return value of every ctx.step() is serialized to Postgres. Upon a restart, the function re-runs, but ctx.step() calls for completed steps simply return the checkpointed value without re-executing the code.
  • Resilience (Runs): A task failure (or worker crash) results in a new "Run." This new run replays events and checkpoints, resuming execution at the last failed step.
  • Asynchronous Primitives: It provides built-in durable primitives that an in-memory executor lacks:
    • ctx.sleep(duration): Suspends the task durably.
    • ctx.waitForEvent(name, timeout): Suspends the task until an external event is emitted via absurd.emitEvent().
  • Dynamic Workflows: Natively supports iterative, self-defining workflows (like AI agents) by versioning step names (e.g., iteration, iteration#2, iteration#3).

3. Feature Mapping and Integration Strategy

This section details how each highway_dsl feature is implemented using the Absurd runtime.

Basic Task Execution & State

  • highway_dsl: A Task node with an associated function.
  • Absurd: A ctx.step(name, func) call.
  • Integration: The interpreter, upon reaching a Task node, executes it within a ctx.step. The task's unique name from the highway_dsl graph is used as the Absurd step name.

Example (Pseudo-code for the interpreter):

// Inside Absurd task: run_highway_workflow(params, ctx)
// params.workflowGraph = the highway_dsl YAML/JSON
// params.taskFunctions = a map of task_name -> actual code

async function executeTask(node) {
  const taskFunction = params.taskFunctions[node.name];

  // The 'highway_dsl' task is executed as a durable 'Absurd' step
  const result = await ctx.step(node.name, async () => {
    // Load inputs from predecessor tasks
    const inputs = loadInputsForNode(node, ctx.checkpoints); 
    return await taskFunction(inputs);
  });

  // The result is now in ctx.checkpoints for future steps
  return result;
}

Retry Policy

This is a key area of feature overlap with two distinct approaches.

  • highway_dsl: Defines granular, application-level retry policies (e.g., max_attempts=3, backoff=2.0) on a per-task basis. This is for transient errors (e.g., API rate limit).
  • Absurd: Defines a task-level maxAttempts on absurd.spawn(). This retries the entire workflow from the last failed step. This is for catastrophic errors (e.g., worker crash, database connection loss).

Integration Recommendation: Use both for their intended purpose.

  1. highway_dsl Retries (Inside ctx.step): The application-level retry logic defined in highway_dsl should be executed inside the ctx.step call. This handles transient failures without failing the Absurd step.
  2. Absurd Retries (Outside ctx.step): The Absurd maxAttempts on spawn is the "outer loop" of resilience, protecting against infrastructure failure.

Example (Pseudo-code for the interpreter):

async function executeTaskWithHighwayRetry(node) {
  const taskFunction = params.taskFunctions[node.name];
  const retryPolicy = node.retryPolicy; // { max_attempts: 3, ... }

  const result = await ctx.step(node.name, async () => {
    // Implement highway_dsl's retry policy *inside* the durable step
    let attempts = 0;
    while (true) {
      try {
        attempts++;
        const inputs = loadInputsForNode(node, ctx.checkpoints);
        return await taskFunction(inputs); // Run the actual business logic
      } catch (e) {
        if (attempts >= retryPolicy.max_attempts) {
          throw e; // Final failure, this fails the ctx.step
        }
        // Wait using highway_dsl's backoff strategy
        await nonDurableSleep(retryPolicy.delay * (retryPolicy.backoff ** attempts));
      }
    }
  });
  return result;
}

If ctx.step finally fails after highway_dsl's retries are exhausted, Absurd's mechanism takes over, potentially re-running the entire workflow in a new Run.

Conditional Branching (Conditional)

  • highway_dsl: A Conditional node that points to a "condition task," a "then branch," and an "else branch."
  • Absurd: A standard imperative if/else statement.
  • Integration: The interpreter executes the condition task as a ctx.step, retrieves its result, and then imperatively follows the correct branch of the highway_dsl graph.

Example (Pseudo-code for the interpreter):

async function executeConditional(node) {
  // 1. Execute the condition task durably
  const conditionResult = await ctx.step(node.conditionTask.name, ...);

  // 2. Use the result in a standard 'if'
  if (conditionResult.value === true) {
    await executeNode(node.thenBranch); // Recursively call interpreter
  } else {
    await executeNode(node.elseBranch); // Recursively call interpreter
  }
}

Parallel Execution (Parallel)

This is the most complex integration, as Absurd's ctx.step model is linear. A single task cannot await multiple ctx.step calls concurrently.

  • highway_dsl: A Parallel node with a list of branches to execute concurrently.
  • Absurd: No ctx.parallel. Parallelism is achieved by spawning new, independent tasks (absurd.spawn) and coordinating via events (absurd.emitEvent / ctx.waitForEvent).
  • Integration: The Parallel node must be mapped to a "fork/join" pattern using Absurd's primitives.

Integration Strategy (Fork/Join):

  1. Fork: The interpreter, upon hitting a Parallel node, executes a single ctx.step (e.g., "fork_parallel_tasks"). Inside this step, it calls absurd.spawn for each branch of the highway_dsl Parallel node. These new tasks run a different Absurd task (e.g., run_highway_sub_workflow). It collects the taskIds and a unique joinEventName.
  2. Join: The main interpreter task then immediately calls await ctx.waitForEvent(joinEventName, ...) to durably wait for all forked tasks to complete.
  3. Sub-Tasks: The run_highway_sub_workflow tasks execute their branch. Upon completion, they await absurd.emitEvent(joinEventName, { ...result }) to signal the main interpreter.
  4. A separate aggregation mechanism is needed to gather all event results before the joinEventName is considered fully "met."

Agentic/Dynamic Loops

  • highway_dsl: Can define an agentic workflow (e.g., tests/data/complex_agentic_workflow.yaml), but the graph structure is likely static (e.g., a loop from "critique" back to "research").
  • Absurd: Natively supports dynamic loops via step name versioning (iteration -> iteration#2).
  • Integration: This provides a powerful "nested" durability model. A single highway_dsl Task (e.g., "run_agent_loop") can be implemented as an Absurd task that itself uses Absurd's dynamic ctx.step("iteration", ...) loop. This allows a structured highway_dsl graph to orchestrate highly dynamic, stateful sub-processes.

Asynchronous Primitives (Sleep, Events)

highway_dsl lacks these, but Absurd provides them. This integration adds new, durable capabilities to highway_dsl.

  • Integration: Create new highway_dsl node types, such as SleepTask and WaitForEventTask.
  • SleepTask: The interpreter maps this node to await ctx.sleep(node.duration). A workflow can now declaratively "wait 7 days."
  • WaitForEventTask: The interpreter maps this to await ctx.waitForEvent(node.eventName). A workflow can now declaratively "wait for user_confirmation event."

4. Conclusion: Benefits of the Merged Architecture

This "Declarative Graph + Durable Interpreter" model provides the best of both worlds:

  1. Separation of Concerns: Business logic (the flow) is defined in highway_dsl, making it easy to read, test, and modify without touching the execution runtime. Execution resilience is handled entirely by Absurd.
  2. Full Durability: The entire workflow state, including the current position in the graph (implied by the last ctx.step checkpoint), is durably stored in Postgres.
  3. Enhanced Capabilities: highway_dsl gains powerful, real-world asynchronous capabilities (durable sleeps, external events) by leveraging the Absurd runtime, features that an in-memory graph executor cannot provide.
  4. Structured Complexity: Absurd gains a high-level, structured, and serializable framework for defining complex processes, moving beyond simple, linear imperative scripts.