he year ends, the North Pole tips its deepest bow to the darkness, and we see even large language models have been taking it easy for the winter¹. But for the faithful maintainer of systems, there’s still work to be done, here in the cooling embers of the year.

Now is a fine time to dust off your backup scripts and see if they’re working as they should. An untested backup is no backup at all, said the wise elders of the Scary Devil Monastery², and if we want to set the new year off to its best start, we should ensure we can pause, tear down, and re-start its march, even at its lowest points.

When examining my backup and restore process, I took the opportunity this year to test my backups and increase the size of my root partition on my server.

One last manual backup of the 256GiB SSD in my trusty server boat for the sake of the old, and then an attempted restore for the sake of the new: to a spare machine, tub, temporarily hosting a new, 1TiB SSD. If all goes well, the restored backup in tub would have new room to grow, and I could swap that drive into boat with minimal downtime.

That moment of hardware-swapping would mean that boat would have to be shut down and then restarted anew. Humans need their respite over the holiday break: but should I have granted my server the same indulgence? My plan accepted that boat would be offline for, I hoped, a small slice of time.

There are more convoluted ways to ensure that none of my websites, file syncing, and miscellaneous tools flickered, even for a moment. I could have temporarily switched my DNS settings to point at the fresh clean tub, for instance, while overwriting boat. Or perhaps just repurposed boat for gentler, less demanding tasks, giving it the end-of-year gift of a well-deserved retirement, and passing to tub a new year’s responsibility of hosting my server processes.

Within the scale of my own life, I do believe that uptime is overrated. We are surrounded by tools bent into the shapes demanded by large tech companies, for whom site reliability is their first commandment.

But we are humans, not corporations. We run at our own pace. If Homer nods, so can my home lab. Christmas has its folklore of visitors rebuffed, but hopefully, my friends have other pressing matters this time of year than hitting reload on my sites.

Yes, a personal web server can go down for a few moments – as long as it bobs back up.

Which, with restored filing systems, is ever the question. Will the backup truly come back up? A reset and reboot may also be a time for sinking exhaustion and death. Human deaths seem positively correlated with the change of pace of the holidays³, and so too, drives and computers will dutifully run for years, but expire after a moment’s rest.

I use “Relax-And-Recover” (REAR), a Linux disaster recovery system from when sysadmins wrote shell code and liked it. REAR is a sprawling set of shell scripts that runs your choice of backup code – from rsync to borg – over your entire Linux filesystem, on a regular, cron-determined, basis. Having seen to that prosaic task, REAR will also create a minimal, bootable blob. The blob, stuffed onto a USB drive, CD-ROM, or networked drive, will boot into a rescue Linux, and lead you through the reconstruction and re-partitioning of a drive that will emerge the same shape as your original machine. Then it will pull down your backups, and restore this drive to the precise state that your backups recalled it.

Readers born into our age of strong types and weak stomachs may be balking at the idea of entrusting their restoration to a bunch of stringly-typed Bash scripts. REAR’s 20K(!) lines of shellcode intimate that it has reached levels of complexity, beyond that which bashism can safely grasp.

But this is sysadmin shell code. That terrifying KLOC is defensive, modular coding of the highest order. For the casual shell user, REAR’s operation is comprehensible, its failure modes anticipated and loudly announced, and tweaks and mediations are semi-obvious. REAR’s authors offer “a relaxing recovery”, and the ghosts of sysadminning past do not lightly emphasize relaxation.

While I was never not relaxed during my holiday restoration. I did occasionally carol a high-pitched note or two of concern.

Two hefty snowbanks stood between me and a perfectly clean restore. Since I first installed it, I have had REAR create ISO files for burning onto a rescue CD-ROM. “CD” now stands for “Cretaceous Disk”: I have not used one for over a decade. Pouring a bootable ISO into a contemporary USB drive drive turns out to be surprisingly tricky, and I can never remember how to do it. In the end, I was forced, humiliatingly, to read REAR’s FAQ, where they recommended a meatier graphical utility, UnetBootin, for achieving this. In the future, I’ve set REAR to output those bootable blobs as RAWDISK, which can be burned (warmed?) onto a USB drive.

REAR’s ecumenical acceptance of external backup programs can introduce tremors into its solid scripts. My use of a distribution borg backup package broke the restore. REAR scans executables that it plans to include on its rescue bootable blob to detect what libraries they require, so that it may copy those over. Sadly, the borg executable can be either a binary executable – for which this suffices – or a Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a future REAR will fix. In the meantime, I copied over the binary borg into /usr/local/bin instead of the Pythonic borg of the Debian repos.

After those tweaks, tub was filled with the form of boat’s Christmas past. I swapped over the two drives, holding my breath, and losing the little M.2 SSD screw as always (these plastic nubbin replacements tempted me in the fallout, though apparently a pencil sharpener will work in a pinch). Boat wobbled and then, like a lucky North Sea seafarer, bobbed back up from an early visit to “Fiddler’s Green”.

Testing your recovery plans during your end-of-year downtime gives you a moment to look back at the record of what has been, and prepare for the ups and downs of the coming year. What will be the same? What will change? What parts of your life can you simply hard link to the habits of the past? And what will you have to incrementally add and integrate into your ever-evolving life?

~Integrity Mather