Monday morning I was unable to solve the NYT crossword. I stared at my laptop screen in disbelief and apprehension, unable to fill in a single square. On a Monday1. Dementia already? Stroke? Should I start taking Lion’s Mane extract? After finding my breath and calming my fears, I resolved to put my hopes on podman.

Podman is a container virtualization tool created and developed by my software team at Red Hat. It is not what most neurologists would suggest in a consultation; the reason I turned to it first was that my symptoms did not actually match the usual ones of cognitive decline2: my problem wasn’t that I couldn’t understand the clues, or that I could understand them but not think of the answers, it was that I couldn’t see the clues at all—nor, for that matter, the grid or anything related to the crossword program I’ve been using for the last fifteen years. It just crashed immediately upon starting.

My immediate suspicion fell on glibc. I had just run a software update on my laptop earlier that morning, and glibc was one of the upgraded packages. The crossword software I use, acrosslite, goes back a ways:

-rwxr-xr-x root/root   1269892 1999-01-12 22:11 acrosslite1.2/acrossl

Yes, 1999. (Side note: are you still running any software from the last millennium?) No source code. No updates since then.3 Dynamically linked, so as you might imagine it has been a challenge to keep running all these years. Every so often a library update causes it to fail; each time, I’ve been able to duct-tape it back into working shape. This time I was tired of that kind of effort. I wanted to fix it in a way that would keep it running for a long time. Hence podman.

My idea was to create a 32-bit container image, copy glibc and other libraries into it from a recent backup, bind-mount the X11 socket and auth file and my crosswords directory, and run it using podman. I didn’t expect this to be easy—and it wasn’t—but that’s my own damn fault, not podman’s: my laptop runs Gentoo Linux, and I build my own kernel4, so it would take some time to make it all work.

The networking dependencies helpfully advised me that I needed to enable CONFIG_NF_LOTSOFSTUFF and CONFIG_TUN; those were easy. Figuring out CONFIG_CGROUP_DEVICE took more effort and reboots. (This was all in my spare time on evenings, so it took several days in all). But eventually I had a working podman on my laptop—that meant there was a chance of getting this to work. I felt encouraged.

With podman working, it was time to play with buildah. Podman is what runs containers; I needed buildah to create one.5 I found a 32-bit alpine image as a starting point, restored my backup glibc and X11 libs, and started iterating various attempts at building a container with acrosslite and its required libraries and bind mounts.

Happy pencil indicates a solved crossword. Happy pencil indicates a solved crossword.

Today, Friday, I have a working crossword program again, and I’m seeing Mister Happy Pencil. The bad news is, today’s crossword took me 10:40—disconcertingly high for a Friday. Maybe I better see a neurologist after all.

Disclaimer: podman is not appropriate for everybody. May cause dizziness or dry mouth. Consult your neurologist before using podman. This is, to be honest, a pretty lame use case for podman: no database, no network connectivity, no actual clustered pods or failover. There are infinitely many more exciting and interesting problems that podman and its container-tool cousins are better suited for. But hey, this was my problem, and my daily crossword is important to me, and I’m grateful to the wonderful people in my team who created these amazingly flexible and powerful tools.


  1. NYT daily crosswords progress in difficulty over the course of each week, with Mondays being the culinary equivalent of boiling a pot of water, Saturdays being baking croissants from scratch. [return]
  2. Then again, if it had been a stroke, would I have had the capacity to recognize it? That’s some catch, that Catch-22. [return]
  3. to the Linux version, that is. They still provide newer versions for Windows and Mac. Most people, of course, would find it easier just to use those. [return]
  4. thank you in advance for keeping your opinion to yourself. [return]
  5. Technically speaking, not really: podman can also build container images. But I wanted something easily scriptable for the many retries I was expecting, and buildah is the better tool for that. [return]