Building Datashard: Solving Data Corruption in High-Concurrency ML Workflows
I've been working on Highway, a Strict Atomic Workflow Management System that handles massive amounts of log data from multiple concurrent processes.
Notes on software engineering
Archive / page 2
21–40 of 550 posts, newest first.
- 2025
-
2 min read
-
A new Article about what I am doing
I just published this article medium.com/@erlang/bank-grade-workflow-engine
1 min read -
A comparison of advanced workflow engines
Feature Highway Workflow Engine DBOS (from Demos) Apache Airflow Tork — — — — — Core Philosophy Decoupled Library . A resilient engine you integrate into your app. Integrated Framework .
2 min read -
A complete workflow domain-specific language - Highway DSL
rodmena-limited.github.io/highway dsl/ link to specs are here
1 min read -
How to run processes in with limited resources in Linux (no Docker needed)
It's super powerful and simple:
1 min read -
Running fully isolated commands in Linux
To use to run a Python command that cannot see or touch files outside its own folder, you must combine with .
1 min read -
Thundering herd problem
The thundering herd problem occurs in computing when a large number of processes, threads, or client requests, all waiting for a specific event or resource, are awakened or triggered simultaneously…
2 min read -
TOCTOU (Time-of-Check to Time-of-Use) vulnerability
I encountered a bug during development of my distributed workflow engine, and I learned that: A TOCTOU (Time-of-Check to Time-of-Use) vulnerability is a software security issue that occurs when an…
2 min read -
Durable Workflow Engines - Ridiculously Jumper!
Update: I have abandoned using Jumper. It's buggy and the maintainers are not very welcoming to contributions. I am building my own.
5 min read -
Integrating `highway_dsl` and `Jumper` for a Durable Workflow Engine
The Integration Model This document outlines a strategy for merging and . These two systems are not competitors; they are complementary components operating at different layers of abstraction.
5 min read -
Highway-DSL: A Pydantic-Native DSL for Defining Complex Workflows
As systems grow, so does the complexity of their processes. Simple then scripts quickly evolve into brittle, unmanageable tangles of statements, blocks, and implicit dependencies.
4 min read -
reTunnel is Amazing
There is not easier way to expose your local network, ai agent or test webhooks. install and expose your app with . retunnel.com
1 min read -
what is derivative of A in respect to B?
The Core Concept: It's a Rate of Change At its heart, the derivative of A with respect to B measures how much A changes when B changes by a very small amount.
3 min read -
What is derivative of A with respect to B?
The Core Concept: It's a Rate of Change At its heart, the derivative of A with respect to B measures how much A changes when B changes by a very small amount.
3 min read -
Learn Neural Networks
Flash Cards ========== Flash Cards (basics) Videos ========= Neural Network From Scratch In Python The spelled-out intro to neural networks and backpropagation: building micrograd
1 min read -
PySpark Style Guide
PySpark Style Guide PySpark is a wrapper language that allows users to interface with an Apache Spark backend to quickly process data.
12 min read -
ySpark Style Guide
PySpark Style Guide PySpark is a wrapper language that allows users to interface with an Apache Spark backend to quickly process data.
12 min read -
Math cheatsheets
1. General Math Cheat Sheet (iPaper and other formats) 2. Elementary Algebra Cheat Sheet (PDF) 3. Trigonometry Cheat Sheet (PDF) 4. Calculus Cheat Sheet (PDF) 5.
1 min read -
The Bill of Materials
Alright, let's talk about something that's rapidly becoming a non-negotiable for anyone building software: the Bill of Materials, or SBOM. I'll show you the way Lockheed Martin does it.
6 min read -
Flattening nested JSON objects with jq
I wanted to take a nested set of JSON objects and import them into a SQLite database using - but I wanted to "flatten" some of the nested rows.
1 min read