Notes on software engineering
201–220 of 546 posts, newest first.
While investigating Datasette issue 1268 I found myself with a Python process that was hanging, and I decided to try running against it based on tips in Debugging of CPython processes with gdb Here's…
Tesseract OCR has a command-line utility which is woefully under-documented. Thanks to Alexandru Nedelcu I figured out how to use it today.
I got an unexpected traffic spike to https://russian-ira-facebook-ads.datasettes.com/ - which runs on Cloud Run - and decided to use to block crawlers.
Learned about this today as a way of accessing the Kubernetes REST API. Assuming you have setup and authorized against a cluster (I'm using DigitalOcean K8S) you can start a proxy for talking to the…
Derek Willis has a Datasette instance full of political campaign emails running on Heroku. Matt Hodges pointed out that a lot of these emails include codes, which are used by ActBlue campaigns to…
I accidentally triggered a commit which added a big chunk of unwanted data to my repository. I didn't want this to stick around in the history forever, and no-one else was pulling from the repo, so I…
I helped build the tide chart visualizations for Rocky Beaches. I wanted to generate an SVG line representing 24 hours of tide levels.
Natalie has been experimenting with using a microwave as a kiln for pottery, specifically for Raku. She wanted to understand how long to run the microwave to reach the desired temperature for the Raku…
I was using the Apache Bench command to exercise some new code I'm writing in Datasette and I noticed I was getting a lot of errors: Truncated output: What's with the 953 failed requests?
I deployed https://metmusem.datasettes.com/ by creating a folder on my computer containing a Dockerfile and then shipping that folder up to Google Cloud Run.
I added some unit tests to my datasette-plugin cookiecutter template today, since the latest features involved adding a script. Here's the full test script I wrote.
I ran a survey on Twitter the other day to try and figure out what people mean when they use the term "agents" with respect to AI.
I usually use my command to import CSV data into SQLite, but for large CSV files (like a 750MB one) this can take quite a long time - over half an hour in this case.
In VIAL issue 724 a Cloud Scheduler job which triggered a Cloud Run hosted export script - by sending an HTTP POST to an endpoint - was returning an error.
I tried to install my datasette-bplist plugin today in a fresh Python 3.10 virtual environment on macOS and got this error: Thanks to StackOverflow I figured out a workaround.
I was getting this intermittent error running my Datasette test suite: OSError: Errno 24 Too many open files To figure out what was going on, I used the package and its method.
Here's how to use a lag window function to calculate new cases per day when the table just has total cases over time on different dates.
I figured out how to run a SQL query directly against a CSV file using the command-line utility: sqlite3 :memory: -cmd '.mode csv' -cmd '.import taxi.csv taxi' \ 'SELECT passenger count, COUNT( )…
TLDR: Use in and your containers will probably work! I was trying to figure out how to get a development environment for a Django project running inside Docker in a M1 Mac.
Today I learned how to use the Python trace module to output every single executed line of Python code in a program - useful for figuring out exactly when a crash or infinite loop happens.