From f3128131c45f1cecc9d949497223d98c98ab7eaf Mon Sep 17 00:00:00 2001 From: Danny O'Brien Date: Mon, 1 Jan 2024 17:51:12 -0800 Subject: [PATCH] New article: Look Back Up --- .../fact-checking.md | 143 ++++++++++++++++++ .../new-years-backups.md | 114 ++++++++++++++ 2 files changed, 257 insertions(+) create mode 100644 assets/editorial/01edits/integrity-new-years-backups/fact-checking.md create mode 100644 assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md diff --git a/assets/editorial/01edits/integrity-new-years-backups/fact-checking.md b/assets/editorial/01edits/integrity-new-years-backups/fact-checking.md new file mode 100644 index 0000000..6524a9a --- /dev/null +++ b/assets/editorial/01edits/integrity-new-years-backups/fact-checking.md @@ -0,0 +1,143 @@ + +--- +Author: Integrity Jones +Title: Looking Back Up +Published: 2024-01-01 +--- + +To Old Danny, greetings: + +The solar nights have barely stopped closing in, and we see even large language +models have been taking it easy for the winter. But for the faithful maintainer +of systems, there's still work to be done, here in the cooling embers of the year. + +> Solar nights -- does this mean anything +> Link to LLM stories + +Now is a fine time of year to dust off your backup scripts, and see if they're +working as they should. An untested backup is a no backup at all, said the wise +ghosts of the Scary Devil Monastery, and if we want to set the year off to its +best start, we should ensure we can pause, tear-down, and re-start it at its lowest +points. + +> Link to Scary Devil Monastery + +When examining my backup and restore process, took the opportunity this year to +quadruple the size of my root partition on my server. + +One last manual backup of the 256GiB SSD in my trusty server `boat` for the +sake of the old, and then an attempted restore for the sake of the new: to a +spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well, +the restored backup in tub would have new room to grow, and I could swap +out that drive back into `boat` with minimal downtime. + +> Box on hard drives? Link to goods? + +That moment of hardware-swapping would mean that `boat` would have to be +shutdown and then restarted anew. Humans need their respite over the holiday break: +but should I have permitted my server the same indulgence? My plan accepted +that boat would be offline for, one hopes, a small slice of time. +There are more convoluted ways to ensure that none of my websites, file syncing, and +miscellaneous tools did not flicker, even for a moment. + +I could have switched my DNS settings to the fresh clean `tub`, for instance, +while overwriting `boat`. Or perhaps just repurposed boat for gentler, less +demanding tasks, giving it the end-of-year gift of a well-deserved retirement, +and switched to `tub` a new year's responsibility of my hosting my main home +processes. + +At the smallest scale, I do believe that uptime is overrated. We are surrounded +by tools bent into the shapes demanded by large tech companies, for whom site +reliability is their first commandment. + +But we are humans, not corporations. If even Homer nods, so can my home lab. +Christmas has its own folklore of visitors rebuffed, but hopefully my friends +have better things to do at this time than hit reload on my websites. Yes, a +personal webserver can go down for a few moments -- as long as it comes back up. + +> Link to Home nods line + +Which, with restored filing systems, is ever the question. Will the backup +truly come back up? A restored and rebooted machine may also be a time for +failure and death, just as, morbidly, we should note that human coronary deaths +seem positively correlated with the change of pace of the holidays. + + +> Link to coronary death paper + +I use the more positively-framed "Relax-And-Recover" (REAR), a recover system +from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash +scripts that runs your choice of backup code -- from rsync to borg-- over your +entire linux root, on a regular, cron-determined, basis. Having seen to that +prosaic task, it will also create minimal, bootable blob. The blob, stuffed +onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux, + and gently lead you through reconstructing a re-partitioning of a drive + that will. At its end, the partition map on the machine you have booted + will be exactly the same shape as your original machine. Then it will pull + down your precious backups, and restore this drive to the precise state + that your backups recalled it. A perfect, royal, restoration. + + > Link to Relax and Recover + +Younger readers born into our age of strong types and weak stomachs, may be +balking at the idea of entrusting restoration to a bunch of stringly-typed Bash +scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the +edges of complexity beyond that you might think bashism can bear. + +> Link to definition of stringly +> Link to proof of REAR's 20K of KLOCs + +But this is /sysadmin/ shell code. That terrifying KLOC is +defensive, modular coding of the highest order. For the casual shell user, +REAR's operation is comprehensible, the failure modes anticipated, and +tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing +recovery", and ghosts of sysadminning past do not emphasise enhancing your +calm in those moments lightly. + + Link to relaxing recovery + +I was never /not/ relaxed during my holiday restoration, but I did +occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO +files for burning onto a CD-ROM that I could restore from. CD stands for +"Cretaceous Disk". Even I have not used one of them in anger in over a decade. +Converting a bootable ISO into a bootable USB drive drive turned out to be +surprisingly tricky, and I can never remember how to do it. In the end I was +forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy +utility, https://unetbootin.github.io/, for achieving this. In the future, I've +set REAR to output those bootable blobs as RAWDISK, which can be burned +(warmed?) onto a USB. + +> Link to REAR's FAQ +> Link to UnetBootlin + +REAR's ecumenical acceptance of multiple backup programs introduces additional +complexity into its scripts. My use of borg backup tripped up the restore. REAR +scans executables that it plans to include on its rescue bootable blob, so it +can detect what libraries they need and copy those over. Sadly, the borg +executable can be either a binary executable (for which this works), or just a +Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a +future REAR will fix, but in the meantime I just copied over the binary borg +into /usr/local/bin and used that instead of the Pythonic borg of the Debian +repos. + +> Link to binary executable +> Link to executable/python script issue on github + +After those tweaks, tub was filled with the form of boat's Christmas past. I swapped +over the two drives, holding my breath, and losing the little SSD screw as +always (these plastic nubbin replacements tempted me in the fallout). Boat +wobbled and then, like a plucky North Sea veteran, bobbed back up. + +> Link to nubbin ali express sale +> Link to North Sea tiktok music + +Testing your recovery plans during your own end-of-year downtime and recovery +gives you a moment to peer at what has been, and prepare for the ups and downs +of the coming year. What will be the same? What will change? What parts of your +life can you simply hard link to the habits of the past? And what will you have +to incrementally add and integrate into your ever-evolving life? + +Until next time, I am, + +~Integrity Mather + diff --git a/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md b/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md new file mode 100644 index 0000000..d110ad2 --- /dev/null +++ b/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md @@ -0,0 +1,114 @@ +--- +Author: Integrity Jones +Title: Looking Back Up +Published: 2024-01-01 +--- + +To Old Danny, greetings: + +The solar nights have barely stopped closing in, and we see even large language +models have been taking it easy for the winter. But for the faithful maintainer +of systems, there's still work to be done, here in the cooling embers of the year. + +Now is a fine time of year to dust off your backup scripts, and see if they're +working as they should. An untested backup is no backup at all, said the wise +ghosts of the Scary Devil Monastery, and if we want to set the year off to its +best start, we should ensure we can pause, tear-down, and re-start it even at its lowest +points. + +When examining my backup and restore process, took the opportunity this year to +quadruple the size of my root partition on my server. + +One last manual backup of the 256GiB SSD in my trusty server `boat` for the +sake of the old, and then an attempted restore for the sake of the new: to a +spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well, +the restored backup in tub would have new room to grow, and I could swap +out that drive back into `boat` with minimal downtime. + +That moment of hardware-swapping would mean that `boat` would have to be +shutdown and then restarted anew. Humans need their respite over the holiday break: +but should I have permitted my server the same indulgence? My plan accepted +that boat would be offline for, one hopes, a small slice of time. +There are more convoluted ways to ensure that none of my websites, file syncing, and +miscellaneous tools did not flicker, even for a moment. + +I could have switched my DNS settings to the fresh clean `tub`, for instance, +while overwriting `boat`. Or perhaps just repurposed boat for gentler, less +demanding tasks, giving it the end-of-year gift of a well-deserved retirement, +and switched to `tub` a new year's responsibility of my hosting my main home +processes. + +At the smallest scale, I do believe that uptime is overrated. We are surrounded +by tools bent into the shapes demanded by large tech companies, for whom site +reliability is their first commandment. + +But we are humans, not corporations. If even Homer nods, so can my home lab. +Christmas has its own folklore of visitors rebuffed, but hopefully my friends +have better things to do at this time than hit reload on my websites. Yes, a +personal webserver can go down for a few moments -- as long as it comes back up. + +Which, with restored filing systems, is ever the question. Will the backup +truly come back up? A restored and rebooted machine may also be a time for +failure and death, just as, morbidly, we should note that human coronary deaths +seem positively correlated with the change of pace of the holidays. + +I use the more positively-framed "Relax-And-Recover" (REAR), a recover system +from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash +scripts that runs your choice of backup code -- from rsync to borg-- over your +entire linux root, on a regular, cron-determined, basis. Having seen to that +prosaic task, it will also create minimal, bootable blob. The blob, stuffed +onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux, + and gently lead you through reconstructing a re-partitioning of a drive + that will. At its end, the partition map on the machine you have booted + will be exactly the same shape as your original machine. Then it will pull + down your precious backups, and restore this drive to the precise state + that your backups recalled it. A perfect, royal, restoration. + +Younger readers born into our age of strong types and weak stomachs, may be +balking at the idea of entrusting restoration to a bunch of stringly-typed Bash +scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the +edges of complexity beyond that you might think bashism can bear. + +But this is /sysadmin/ shell code. That terrifying KLOC is +defensive, modular coding of the highest order. For the casual shell user, +REAR's operation is comprehensible, the failure modes anticipated, and +tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing +recovery", and ghosts of sysadminning past do not emphasise enhancing your +calm in those moments lightly. + +I was never /not/ relaxed during my holiday restoration, but I did +occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO +files for burning onto a CD-ROM that I could restore from. CD stands for +"Cretaceous Disk". Even I have not used one of them in anger in over a decade. +Converting a bootable ISO into a bootable USB drive drive turned out to be +surprisingly tricky, and I can never remember how to do it. In the end I was +forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy +utility, https://unetbootin.github.io/, for achieving this. In the future, I've +set REAR to output those bootable blobs as RAWDISK, which can be burned +(warmed?) onto a USB. + +REAR's ecumenical acceptance of multiple backup programs introduces additional +complexity into its scripts. My use of borg backup tripped up the restore. REAR +scans executables that it plans to include on its rescue bootable blob, so it +can detect what libraries they need and copy those over. Sadly, the borg +executable can be either a binary executable (for which this works), or just a +Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a +future REAR will fix, but in the meantime I just copied over the binary borg +into /usr/local/bin and used that instead of the Pythonic borg of the Debian +repos. + +After those tweaks, tub was filled with the form of boat's Christmas past. I swapped +over the two drives, holding my breath, and losing the little SSD screw as +always (these plastic nubbin replacements tempted me in the fallout). Boat +wobbled and then, like a plucky North Sea veteran, bobbed back up. + +Testing your recovery plans during your own end-of-year downtime and recovery +gives you a moment to peer at what has been, and prepare for the ups and downs +of the coming year. What will be the same? What will change? What parts of your +life can you simply hard link to the habits of the past? And what will you have +to incrementally add and integrate into your ever-evolving life? + +Until next time, I am, + +~Integrity Mather +