I write a lot of my own software. I also write a lot of bugs. Since the beginning of the year, I have kept a log of every bug I found in code that I wrote. I wanted to better understand the kinds of mistakes that I tend to make and what I can do to avoid making more of them.
Stats
As of December 17, there are 67 bugs on the list (which excludes bugs at work). 58 of the bugs were in Python, 5 in Bash, and 2 each in JavaScript and Go. (I write almost all of my personal software in Python.)
6 of the bugs were in the source code for this website, 4 were in foundation, 2 were in iprecommit, and the 2 Go bugs were in fast-concordance. The remaining 53 were in my private Python monorepo, which includes the job scheduler that I have alluded to previously, as well as many other things, and about which I intend to write a blog post soon.
Static typing?
7 of the 58 Python bugs (12%) would have been type errors in a statically-typed language. A few representative examples were:
- Failing to pass a required parameter to a function.
- Calling
path.remove()when the right API waspath.unlink(). - Generating a integer too large to store in a signed 32-bit Postgres column.
I run Pyright in my IDE and on every commit, which catches most type errors before they make it out into the wild. A few bugs got through for various reasons: because I had implicitly disabled type-checking with the Any type, because I had not made the interface to an external data source sufficiently typeful (Postgres, command-line arguments, JSON, etc.), or simply because Pyright has limits.
Sample bugs
My most interesting bugs of the year:
- An exception after
forkbut beforeexeccaused clean-up handlers to be run in both the child and parent process, causing chaos. (This was the first bug in "Woes of writing your own jobserver".) - Two different deadlock bugs caused by taking a lock in a signal handler – once an explicit file lock and once a lock inside a thread-safe data structure. (These were the last two bugs in "Woes of writing your own jobserver".)
- A log rotation job inconsistently tried to delete a file that did not exist (but should have existed) and crashed. The job was scheduled at regular intervals, and sometimes (mainly on my laptop when it was asleep) a job would take longer than expected and still be running when the next job started. Both would see that the same file was ready for deletion but only one would win the race.
- I have a script to upload Markdown files to my personal site. It uses hashes to only upload files that have changed. For this to work, the server and the client have to calculate the hash in exactly the same way, but the client was stripping Markdown front-matter before uploading, so for files with front-matter the client and server always computed a different hash and the file was re-uploaded every time.
(My least interesting bugs were the 4 different ones caused by unset environment variables.)
And a few that were just amusing:
- Parsing a CSV import from Venmo failed the first time a transaction exceeded $1,000, because of the presence of the comma.
- I switched from a bespoke parser for Obsidian properties to an off-the-shelf YAML implementation, but some code couldn't handle the new parser's correct output because it was written to accept the old parser's broken output.
- I use an LLM filter to recommend articles from the front page of Hacker News. One week I noticed that my recommendations suddenly increased in quantity and decreased in quality. While refactoring, I had accidentally changed
save_bookmarks(recommended_stories)tosave_bookmarks(stories), thereby ignoring the actual recommendations.
Lessons
I don't mind shipping bugs in personal software when the stakes are low. Still, every bug that slips through is an opportunity to re-examine my process and ask myself what I could do to prevent the same mistake from happening again. Sometimes this led to durable changes (e.g., running Pyright on the whole repo instead of only on the changed files in a commit) and sometimes it did not (many of the bugs would have been caught by simple unit tests, which I nonetheless continued to fail to write). I'm optimistic that AI code review will help me catch bugs earlier, especially now that I have 67 examples of the kinds of mistakes that I tend to make. ∎