--- Author: Integrity Jones Title: Looking Back Up Published: 2024-01-01 --- To Old Danny, greetings: The solar nights have barely stopped closing in, and we see even large language models have been taking it easy for the winter. But for the faithful maintainer of systems, there's still work to be done, here in the cooling embers of the year. > Solar nights -- does this mean anything (No) > Link to LLM stories Now is a fine time of year to dust off your backup scripts, and see if they're working as they should. An untested backup is a no backup at all, said the wise ghosts of the Scary Devil Monastery, and if we want to set the year off to its best start, we should ensure we can pause, tear-down, and re-start it at its lowest points. > Link to Scary Devil Monastery When examining my backup and restore process, took the opportunity this year to quadruple the size of my root partition on my server. One last manual backup of the 256GiB SSD in my trusty server `boat` for the sake of the old, and then an attempted restore for the sake of the new: to a spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well, the restored backup in tub would have new room to grow, and I could swap out that drive back into `boat` with minimal downtime. > Box on hard drives? Link to goods? That moment of hardware-swapping would mean that `boat` would have to be shutdown and then restarted anew. Humans need their respite over the holiday break: but should I have permitted my server the same indulgence? My plan accepted that boat would be offline for, one hopes, a small slice of time. There are more convoluted ways to ensure that none of my websites, file syncing, and miscellaneous tools did not flicker, even for a moment. I could have switched my DNS settings to the fresh clean `tub`, for instance, while overwriting `boat`. Or perhaps just repurposed boat for gentler, less demanding tasks, giving it the end-of-year gift of a well-deserved retirement, and switched to `tub` a new year's responsibility of my hosting my main home processes. At the smallest scale, I do believe that uptime is overrated. We are surrounded by tools bent into the shapes demanded by large tech companies, for whom site reliability is their first commandment. But we are humans, not corporations. If even Homer nods, so can my home lab. Christmas has its own folklore of visitors rebuffed, but hopefully my friends have better things to do at this time than hit reload on my websites. Yes, a personal webserver can go down for a few moments -- as long as it comes back up. > Link to Home nods line Which, with restored filing systems, is ever the question. Will the backup truly come back up? A restored and rebooted machine may also be a time for failure and death, just as, morbidly, we should note that human coronary deaths seem positively correlated with the change of pace of the holidays. > Link to coronary death paper I use the more positively-framed "Relax-And-Recover" (REAR), a recover system from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash scripts that runs your choice of backup code -- from rsync to borg-- over your entire linux root, on a regular, cron-determined, basis. Having seen to that prosaic task, it will also create minimal, bootable blob. The blob, stuffed onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux, and gently lead you through reconstructing a re-partitioning of a drive that will. At its end, the partition map on the machine you have booted will be exactly the same shape as your original machine. Then it will pull down your precious backups, and restore this drive to the precise state that your backups recalled it. A perfect, royal, restoration. > Link to Relax and Recover Younger readers born into our age of strong types and weak stomachs, may be balking at the idea of entrusting restoration to a bunch of stringly-typed Bash scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the edges of complexity beyond that you might think bashism can bear. > Link to definition of stringly > Link to proof of REAR's 20K of KLOCs But this is /sysadmin/ shell code. That terrifying KLOC is defensive, modular coding of the highest order. For the casual shell user, REAR's operation is comprehensible, the failure modes anticipated, and tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing recovery", and ghosts of sysadminning past do not emphasise enhancing your calm in those moments lightly. > Link to REAR I was never /not/ relaxed during my holiday restoration, but I did occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO files for burning onto a CD-ROM that I could restore from. CD stands for "Cretaceous Disk". Even I have not used one of them in anger in over a decade. Converting a bootable ISO into a bootable USB drive drive turned out to be surprisingly tricky, and I can never remember how to do it. In the end I was forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy utility, https://unetbootin.github.io/, for achieving this. In the future, I've set REAR to output those bootable blobs as RAWDISK, which can be burned (warmed?) onto a USB. > Link to REAR's FAQ > Link to UnetBootlin REAR's ecumenical acceptance of multiple backup programs introduces additional complexity into its scripts. My use of borg backup tripped up the restore. REAR scans executables that it plans to include on its rescue bootable blob, so it can detect what libraries they need and copy those over. Sadly, the borg executable can be either a binary executable (for which this works), or just a Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a future REAR will fix, but in the meantime I just copied over the binary borg into /usr/local/bin and used that instead of the Pythonic borg of the Debian repos. > Link to binary executable > Link to executable/python script issue on github After those tweaks, tub was filled with the form of boat's Christmas past. I swapped over the two drives, holding my breath, and losing the little SSD screw as always (these plastic nubbin replacements tempted me in the fallout). Boat wobbled and then, like a plucky North Sea veteran, bobbed back up. > Link to nubbin ali express sale > Link to North Sea tiktok music Testing your recovery plans during your own end-of-year downtime and recovery gives you a moment to peer at what has been, and prepare for the ups and downs of the coming year. What will be the same? What will change? What parts of your life can you simply hard link to the habits of the past? And what will you have to incrementally add and integrate into your ever-evolving life? Until next time, I am, ~Integrity Mather