json

Flattening nested JSON objects with jq

I wanted to take a nested set of JSON objects and import them into a SQLite database using - but I wanted to "flatten" some of the nested rows.

Streaming output of an indented JSON array

For paginate-json I wanted to implement streaming output of an indented JSON array to my terminal. The final output should look like this: But I needed this to work for an arbitrarily long sequence of…

Generated a summary of nested JSON data

I was trying to figure out the shape of the JSON object from https://github.com/simonw/coronavirus-data-gov-archive/blob/master/data latest.json?raw=true - which is 3.2MB and heavily nested, so it's…

Extracting web page content using Readability.js and shot-scraper

Readability.js is "A standalone version of the readability library used for Firefox Reader View". I figured out how to use the two of these together to extract the core content from a web page using…

Convert git log output to JSON using jq

I just spent way too long messing around with ChatGPT (transcript here) trying to figure this out. After much iteration, here's a recipe that works (mostly written by me at this point): The output…

Generating OpenAPI specifications using GPT-3

I wanted to start playing around with OpenAPI. I decided to see if I could get GPT-3 to generate the first version of a specification for me.

Exporting Amplitude events to SQLite

Amplitude offers an "Export Data" button in the project settings page. This can export up to 365 days of events (up to 4GB per export), where the export is a zip file containing gzipped…

Exploring OpenAI's deep research API model o4-mini-deep-research

I was reviewing some older PRs and landed this one by Manuel Solorzano adding pricing for o4-mini-deep-research and o3-deep-research to my llm-prices.com site.

How to read Hacker News threads with most recent comments first

Hacker News displays comments in a tree. This can be frustrating if you want to keep track of a particular conversation, as you constantly have to seek through the tree to find the latest comment.

Comparing database rows before and after with SQLite JSON functions

Here's a trick I've been using to compare the rows in a table before and after I perform an operation against it. It works well for a few hundred (and maybe a few thousand) rows.

Processing a stream of chunks of JSON with ijson

A follow-up to Using OpenAI functions and their Python library for data extraction and Using the ChatGPT streaming API from Python.

JSON Pointer

I'm looking at options for representing JSON validation errors in more JSON. The recent RFC 7807: Problem Details for HTTP APIs looks relevant here.

Export a Mastodon timeline to SQLite

I've been playing around with the Mastodon timelines API. It's pretty fun! I'm running my own instance which means I don't feel limited by politeness in terms of rate limits - if I'm just using my own…

Constructing GeoJSON in PostgreSQL

In order to efficiently generate a GeoJSON representation of a vast number of locations, I'm currently experimenting with generating the GeoJSON directly inside a PostgreSQL SQL query using and…

Extracting objects recursively with jq

The Algolia-powered Hacker News API returns nested comment threads that look like this: https://hn.algolia.com/api/v1/items/27941108 (For this story: https://news.ycombinator.com/item?id=27941108) I…

Converting Airtable JSON for use with sqlite-utils using jq

The Airtable API outputs JSON that looks like this: The items in the array each have an , a nested dictionary and a .

How I studied for my Ham radio general exam

I scraped a pass on my Ham radio general exam today, on the second attempt (you can retake on the same day for an extra $15, thankfully).

Comparing two training datasets using sqlite-utils

WizardLM is "an Instruction-following LLM Using Evol-Instruct". It's a fine-tuned model on top of Meta's LLaMA.

Converting no-decimal-point latitudes and longitudes using jq

I had some data with weird co-ordinates in it: Note the and in there. I know that this data refers to somewhere in Houston, and a few people on Twitter pointed out that it's actually really simple…