bash

A shell script for running Go one-liners

bitfield/script is a really neat Go project: it tries to emulate shell scripting using Go chaining primitives, so you can run code like this: To achieve the same thing as: A comment from on Hacker…

Finding CSV files that start with a BOM using ripgrep

For sqlite-utils issue 250 I needed to locate some test CSV files that start with a UTF-8 BOM. Here's how I did that using ripgrep: The option means the search spans multiple lines - I only want to…

Converting ORF raw files to JPEG on macOS

One of our cameras takes raw photos in ORF format, which I believe stands for "Olympus Raw Format". Here's a recipe I found for converting them to JPEG on macOS: The result is a directory containing a…

Ignoring errors in a section of a Bash script

For simonw/museums 32 I wanted to have certain lines in my Bash script ignore any errors: lines that used to add columns and configure FTS, but that might fail with an error if the column already…

Convert git log output to JSON using jq

I just spent way too long messing around with ChatGPT (transcript here) trying to figure this out. After much iteration, here's a recipe that works (mostly written by me at this point): The output…

Combined release notes from GitHub with jq and paginate-json

Matt Holt asked: Is there a tool that uses the GitHub API to generate a doc with all release notes from a repo?

Calculating the size of a SQLite database file using SQL

I learned this trick today while browsing the code of Blacklite, a neat Java library for writing diagnostic logs to a SQLite database.

Attaching a bash shell to a running Docker container

Use to find the container ID: $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 81b2ad3194cb alexdebrie/livegrep-base:1 "/livegrep-github-re…" 2 minutes ago Up 2 minutes compassionate…

Running multiple servers in a single Bash script

I spotted this script that starts the opendream appication running both a Python server and a script and it intrigued me - was it this easy to have a single Bash script run two servers?

Looping over comma-separated values in Bash

Given a file (or a process) that produces comma separated values, here's how to split those into separate variables and use them in a bash script.

Calculating the size of all LFS files in a repo

I wanted to know how large the deepseek-ai/DeepSeek-V3-Base repo on Hugging Face was without actually downloading all of the files. With some help from Claude, here's the recipe that worked.

Using awk to add a prefix

I wanted to dynamically run the following command against all files in a directory: I can't use here because I need each file to be processed using the option. I found a solution using .

hexdump and hexdump -C

While exploring null bytes in this issue I learned that the command on macOS (and presumably other Unix systems) has a confusing default output.

Skipping CSV rows with odd numbers of quotes using ripgrep

I'm working with several huge CSV files - over 5 million rows total - and I ran into a problem: it turned out there were a few lines in those files that imported incorrectly because they were not…

Start, test, then stop a localhost web server in a Bash script

I wanted to write a bash script that would start a Datasette server running, run a request against it using , then stop the server again.

Enabling WAL mode for SQLite database files

I was getting occasional messages from a Datasette instance that was running against a bunch of different SQLite files that were updated by cron scripts (my personal Dogsheep).

Escaping strings in Bash using !:q

TIL this trick, via Pascal Hirsch on Twitter. Enter a line of Bash starting with a comment, then run on the next line to see what that would be with proper Bash escaping applied.

Commit a file if it changed

This recipe runs a Python script to update a README, then commits it back to the parent repo but only if it has changed: My first attempt threw an error if I tried o run and the README had not…

Copy tables between SQLite databases

I figured out a pattern for doing this today using the CLI tool - given two SQLite databases in the current folder, called and : I'm using that in this GitHub Actions workflow.

Downloading partial YouTube videos with ffmpeg

I spoke at WordCamp US 2023, and wanted to grab a copy of the video of my talk. I always try to keep my own copies of these because I've seen some conferences eventually take these offline in the…