From 122ebef2e21762b0e6f2dba530fb491d0dbcb32b Mon Sep 17 00:00:00 2001 From: Danny O'Brien Date: Tue, 2 Jan 2024 00:13:09 -0800 Subject: [PATCH] edit: Changes to New Years Backups --- .../integrity-new-years-backups/0PASS.org | 13 ++ .../integrity-new-years-backups/FACT.md | 143 +++++++++++++++ .../new-years-backups.md | 166 +++++++++++------- 3 files changed, 255 insertions(+), 67 deletions(-) create mode 100644 assets/editorial/01edits/integrity-new-years-backups/0PASS.org create mode 100644 assets/editorial/01edits/integrity-new-years-backups/FACT.md diff --git a/assets/editorial/01edits/integrity-new-years-backups/0PASS.org b/assets/editorial/01edits/integrity-new-years-backups/0PASS.org new file mode 100644 index 0000000..c347be9 --- /dev/null +++ b/assets/editorial/01edits/integrity-new-years-backups/0PASS.org @@ -0,0 +1,13 @@ +* Editorial Passes +** DONE HIRED [/] -- Commissioning and contract +** DONE STYLE [/] -- Editor and writer work together on style and structure. +** TODO FACTS [/] -- Fact-checking + - [ ] Links. Ensure we have links to as many places as possible, and that they work. + - [ ] Check with third-parties. To the greatest extent, we should reach out to everyone mentioned or to relevant experts and confirm that factual statements are correct. + - [ ] Terminology. Check that terminology is correct. +** TODO SPELL [/] -- Grammar and spell checking +** TODO GRAPH [/] -- Graphics. Every piece needs a illustrated initial and at least one image. +** TODO FINAL [/] -- Final look through before publication. + - [ ] Add any Almnck footnote references to the canonical [[../../../../doc/footnotes.org][footnotes]] list. + - [ ] Final check of all links. +** TODO WRAPUP [/] -- Final administrivia. YOU MUST DO THIS. diff --git a/assets/editorial/01edits/integrity-new-years-backups/FACT.md b/assets/editorial/01edits/integrity-new-years-backups/FACT.md new file mode 100644 index 0000000..4cb07be --- /dev/null +++ b/assets/editorial/01edits/integrity-new-years-backups/FACT.md @@ -0,0 +1,143 @@ + +--- +Author: Integrity Jones +Title: Looking Back Up +Published: 2024-01-01 +--- + +To Old Danny, greetings: + +The solar nights have barely stopped closing in, and we see even large language +models have been taking it easy for the winter. But for the faithful maintainer +of systems, there's still work to be done, here in the cooling embers of the year. + +> Solar nights -- does this mean anything (No) +> Link to LLM stories + +Now is a fine time of year to dust off your backup scripts, and see if they're +working as they should. An untested backup is a no backup at all, said the wise +ghosts of the Scary Devil Monastery, and if we want to set the year off to its +best start, we should ensure we can pause, tear-down, and re-start it at its lowest +points. + +> Link to Scary Devil Monastery + +When examining my backup and restore process, took the opportunity this year to +quadruple the size of my root partition on my server. + +One last manual backup of the 256GiB SSD in my trusty server `boat` for the +sake of the old, and then an attempted restore for the sake of the new: to a +spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well, +the restored backup in tub would have new room to grow, and I could swap +out that drive back into `boat` with minimal downtime. + +> Box on hard drives? Link to goods? + +That moment of hardware-swapping would mean that `boat` would have to be +shutdown and then restarted anew. Humans need their respite over the holiday break: +but should I have permitted my server the same indulgence? My plan accepted +that boat would be offline for, one hopes, a small slice of time. +There are more convoluted ways to ensure that none of my websites, file syncing, and +miscellaneous tools did not flicker, even for a moment. + +I could have switched my DNS settings to the fresh clean `tub`, for instance, +while overwriting `boat`. Or perhaps just repurposed boat for gentler, less +demanding tasks, giving it the end-of-year gift of a well-deserved retirement, +and switched to `tub` a new year's responsibility of my hosting my main home +processes. + +At the smallest scale, I do believe that uptime is overrated. We are surrounded +by tools bent into the shapes demanded by large tech companies, for whom site +reliability is their first commandment. + +But we are humans, not corporations. If even Homer nods, so can my home lab. +Christmas has its own folklore of visitors rebuffed, but hopefully my friends +have better things to do at this time than hit reload on my websites. Yes, a +personal webserver can go down for a few moments -- as long as it comes back up. + +> Link to Home nods line + +Which, with restored filing systems, is ever the question. Will the backup +truly come back up? A restored and rebooted machine may also be a time for +failure and death, just as, morbidly, we should note that human coronary deaths +seem positively correlated with the change of pace of the holidays. + + +> Link to coronary death paper + +I use the more positively-framed "Relax-And-Recover" (REAR), a recover system +from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash +scripts that runs your choice of backup code -- from rsync to borg-- over your +entire linux root, on a regular, cron-determined, basis. Having seen to that +prosaic task, it will also create minimal, bootable blob. The blob, stuffed +onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux, + and gently lead you through reconstructing a re-partitioning of a drive + that will. At its end, the partition map on the machine you have booted + will be exactly the same shape as your original machine. Then it will pull + down your precious backups, and restore this drive to the precise state + that your backups recalled it. A perfect, royal, restoration. + + > Link to Relax and Recover + +Younger readers born into our age of strong types and weak stomachs, may be +balking at the idea of entrusting restoration to a bunch of stringly-typed Bash +scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the +edges of complexity beyond that you might think bashism can bear. + +> Link to definition of stringly +> Link to proof of REAR's 20K of KLOCs + +But this is /sysadmin/ shell code. That terrifying KLOC is +defensive, modular coding of the highest order. For the casual shell user, +REAR's operation is comprehensible, the failure modes anticipated, and +tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing +recovery", and ghosts of sysadminning past do not emphasise enhancing your +calm in those moments lightly. + +> Link to REAR + +I was never /not/ relaxed during my holiday restoration, but I did +occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO +files for burning onto a CD-ROM that I could restore from. CD stands for +"Cretaceous Disk". Even I have not used one of them in anger in over a decade. +Converting a bootable ISO into a bootable USB drive drive turned out to be +surprisingly tricky, and I can never remember how to do it. In the end I was +forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy +utility, https://unetbootin.github.io/, for achieving this. In the future, I've +set REAR to output those bootable blobs as RAWDISK, which can be burned +(warmed?) onto a USB. + +> Link to REAR's FAQ +> Link to UnetBootlin + +REAR's ecumenical acceptance of multiple backup programs introduces additional +complexity into its scripts. My use of borg backup tripped up the restore. REAR +scans executables that it plans to include on its rescue bootable blob, so it +can detect what libraries they need and copy those over. Sadly, the borg +executable can be either a binary executable (for which this works), or just a +Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a +future REAR will fix, but in the meantime I just copied over the binary borg +into /usr/local/bin and used that instead of the Pythonic borg of the Debian +repos. + +> Link to binary executable +> Link to executable/python script issue on github + +After those tweaks, tub was filled with the form of boat's Christmas past. I swapped +over the two drives, holding my breath, and losing the little SSD screw as +always (these plastic nubbin replacements tempted me in the fallout). Boat +wobbled and then, like a plucky North Sea veteran, bobbed back up. + +> Link to nubbin ali express sale +> Link to North Sea tiktok music + +Testing your recovery plans during your own end-of-year downtime and recovery +gives you a moment to peer at what has been, and prepare for the ups and downs +of the coming year. What will be the same? What will change? What parts of your +life can you simply hard link to the habits of the past? And what will you have +to incrementally add and integrate into your ever-evolving life? + +Until next time, I am, + +~Integrity Mather + diff --git a/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md b/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md index d110ad2..6586650 100644 --- a/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md +++ b/assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md @@ -1,114 +1,146 @@ --- -Author: Integrity Jones +Author: Integrity Mather Title: Looking Back Up -Published: 2024-01-01 --- To Old Danny, greetings: -The solar nights have barely stopped closing in, and we see even large language -models have been taking it easy for the winter. But for the faithful maintainer -of systems, there's still work to be done, here in the cooling embers of the year. +The nights have barely stopped closing in the Northern hemisphere, and we see +even large language models have been taking it easy for the +winter[^winterbreak]. But for the faithful maintainer of systems, there's still +work to be done, here in the cooling embers of the year. -Now is a fine time of year to dust off your backup scripts, and see if they're -working as they should. An untested backup is no backup at all, said the wise -ghosts of the Scary Devil Monastery, and if we want to set the year off to its -best start, we should ensure we can pause, tear-down, and re-start it even at its lowest -points. +Now is a fine time to dust off your backup scripts and see if they're working +as they should. An untested backup is no backup at all, said the wise elders of +the [Scary Devil Monastery](http://www.faqs.org/faqs/sysadmin-recovery/)[^scarydevilmonastery], and if we want to set the new year off to its best +start, we should ensure we can pause, tear-down, and re-start the marching +present even at its lowest points. -When examining my backup and restore process, took the opportunity this year to -quadruple the size of my root partition on my server. +When examining my backup and restore process, I took the opportunity this year +to test my backups while quadrupling the size of my root partition on my +server. One last manual backup of the 256GiB SSD in my trusty server `boat` for the sake of the old, and then an attempted restore for the sake of the new: to a spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well, the restored backup in tub would have new room to grow, and I could swap -out that drive back into `boat` with minimal downtime. +that drive into `boat` with minimal downtime. That moment of hardware-swapping would mean that `boat` would have to be shutdown and then restarted anew. Humans need their respite over the holiday break: -but should I have permitted my server the same indulgence? My plan accepted -that boat would be offline for, one hopes, a small slice of time. -There are more convoluted ways to ensure that none of my websites, file syncing, and -miscellaneous tools did not flicker, even for a moment. +but should I have granted my server the same indulgence? My plan accepted +that `boat` would be offline for, I hoped, a small slice of time. -I could have switched my DNS settings to the fresh clean `tub`, for instance, -while overwriting `boat`. Or perhaps just repurposed boat for gentler, less -demanding tasks, giving it the end-of-year gift of a well-deserved retirement, -and switched to `tub` a new year's responsibility of my hosting my main home +There are more convoluted ways to ensure that none of my websites, file +syncing, and miscellaneous tools did not flicker, even for a moment. I could +have switched my DNS settings to the fresh clean `tub`, for instance, while +overwriting `boat`. Or perhaps just repurposed `boat` for gentler, less demanding +tasks, giving it the end-of-year gift of a well-deserved retirement, and +switched to `tub` a new year's responsibility of my hosting my main home processes. -At the smallest scale, I do believe that uptime is overrated. We are surrounded -by tools bent into the shapes demanded by large tech companies, for whom site -reliability is their first commandment. +At the scale of my own life, I do believe that uptime is overrated. We are +surrounded by tools bent into the shapes demanded by large tech companies, for +whom site reliability is their first commandment. -But we are humans, not corporations. If even Homer nods, so can my home lab. -Christmas has its own folklore of visitors rebuffed, but hopefully my friends +But we are humans, not corporations. If Homer nods, so can my home lab. +Christmas has its own folklore of [visitors rebuffed](https://en.wikipedia.org/wiki/Befana), but hopefully my friends have better things to do at this time than hit reload on my websites. Yes, a -personal webserver can go down for a few moments -- as long as it comes back up. +personal webserver can go down for a few moments -- as long as it comes back +up. Which, with restored filing systems, is ever the question. Will the backup truly come back up? A restored and rebooted machine may also be a time for -failure and death, just as, morbidly, we should note that human coronary deaths -seem positively correlated with the change of pace of the holidays. +failure and death, just as, morbidly, we should note that human deaths +seem positively correlated with the change of pace of the holidays[^xmasdeaths]. -I use the more positively-framed "Relax-And-Recover" (REAR), a recover system +I use the more positively-framed ["Relax-And-Recover"](https://relax-and-recover.org/) (REAR), a disaster recovery system from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash -scripts that runs your choice of backup code -- from rsync to borg-- over your +scripts that runs your choice of backup code -- from `[rsync](https://jumpcloud.com/blog/how-to-backup-linux-system-rsync)` to `[borg](https://borgbackup.readthedocs.io/)` -- over your entire linux root, on a regular, cron-determined, basis. Having seen to that -prosaic task, it will also create minimal, bootable blob. The blob, stuffed -onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux, - and gently lead you through reconstructing a re-partitioning of a drive - that will. At its end, the partition map on the machine you have booted - will be exactly the same shape as your original machine. Then it will pull - down your precious backups, and restore this drive to the precise state - that your backups recalled it. A perfect, royal, restoration. +prosaic task, it will also create [minimal, bootable blob](https://relax-and-recover.org/usage/#recovery_from_usb). The blob, stuffed +onto an USB drive, CD-ROM, or networked drive, will boot into a minimal Linux, +and lead you through the reconstruction and re-partitioning of a drive +that will emerge the same shape as your original machine. Then it will pull +down your backups, and restore this drive to the precise state +that your backups recalled it. A perfect, royal, restoration. -Younger readers born into our age of strong types and weak stomachs, may be -balking at the idea of entrusting restoration to a bunch of stringly-typed Bash -scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the -edges of complexity beyond that you might think bashism can bear. +Readers born into our age of strong types and weak stomachs may be balking at +the idea of entrusting restoration to a bunch of [stringly-typed](https://samgrayson.me/essays/stop-writing-shell-scripts/) Bash scripts. +REAR's 20K(!) lines of shellcode intimate that it has reached the edges of +complexity beyond that you might think bashism can bear. -But this is /sysadmin/ shell code. That terrifying KLOC is +But this is *sysadmin* shell code. That terrifying KLOC is defensive, modular coding of the highest order. For the casual shell user, -REAR's operation is comprehensible, the failure modes anticipated, and +REAR's operation is comprehensible, failure modes anticipated and loudly-announced, and tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing -recovery", and ghosts of sysadminning past do not emphasise enhancing your -calm in those moments lightly. +recovery", and ghosts of sysadminning past do not lightly emphasise enhancing your +calm in those moments. -I was never /not/ relaxed during my holiday restoration, but I did -occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO +I was never /not/ relaxed during my holiday restoration. I did +occasionally emit a high-pitched carol or two of concern. I have had REAR create ISO files for burning onto a CD-ROM that I could restore from. CD stands for -"Cretaceous Disk". Even I have not used one of them in anger in over a decade. -Converting a bootable ISO into a bootable USB drive drive turned out to be +"Cretaceous Disk" nowadays: Even I have not used them in anger for over a decade. + +Converting a bootable ISO into a bootable USB drive drive turns out to be surprisingly tricky, and I can never remember how to do it. In the end I was -forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy -utility, https://unetbootin.github.io/, for achieving this. In the future, I've -set REAR to output those bootable blobs as RAWDISK, which can be burned +forced, humiliatingly, to read [REAR's FAQ](https://relax-and-recover.org/documentation/faq), where they recommended a heavy +utility, [UnetBootin](https://unetbootin.github.io/), for achieving this. In the future, I've +set REAR to output those bootable blobs as [RAWDISK](https://relax-and-recover.org/rear-user-guide/basics/configuration.html), which can be burned (warmed?) onto a USB. -REAR's ecumenical acceptance of multiple backup programs introduces additional +REAR's ecumenical acceptance of multiple backup programs can introduce additional complexity into its scripts. My use of borg backup tripped up the restore. REAR -scans executables that it plans to include on its rescue bootable blob, so it -can detect what libraries they need and copy those over. Sadly, the borg +[scans executables](https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/build/default/990_verify_rootfs.sh#L51) that it plans to include on its rescue bootable blobto +detect what libraries they require, so that it may copy those over. Sadly, the borg executable can be either a binary executable (for which this works), or just a Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a -future REAR will fix, but in the meantime I just copied over the binary borg -into /usr/local/bin and used that instead of the Pythonic borg of the Debian +future REAR will fix, but in the meantime I just copied over the [binary borg](https://borgbackup.readthedocs.io/en/stable/installation.html#standalone-binary) +into `/usr/local/bin` and used that instead of the Pythonic borg of the Debian repos. -After those tweaks, tub was filled with the form of boat's Christmas past. I swapped -over the two drives, holding my breath, and losing the little SSD screw as -always (these plastic nubbin replacements tempted me in the fallout). Boat -wobbled and then, like a plucky North Sea veteran, bobbed back up. +After those tweaks, `tub` was filled with the form of `boat`'s Christmas past. I swapped +over the two drives, holding my breath, and losing the little M.2 SSD screw as +always (these [plastic nubbin replacements](https://www.ebay.com/itm/275937873783) tempted me in the fallout, though apparently a [pencil sharpener](https://linustechtips.com/topic/1319971-missing-a-screw-for-your-m2-ssd-check-this-out/) also works in a pinch). `Boat` +wobbled and then, like a lucky [North Sea seafarer](https://www.youtube.com/watch?v=qlrvzLRgzdc), bobbed back up from an early visit to "[Fiddler's Green](https://en.wikipedia.org/wiki/Fiddler%27s_Green)". -Testing your recovery plans during your own end-of-year downtime and recovery -gives you a moment to peer at what has been, and prepare for the ups and downs -of the coming year. What will be the same? What will change? What parts of your -life can you simply hard link to the habits of the past? And what will you have -to incrementally add and integrate into your ever-evolving life? +Testing your recovery plans during your own end-of-year +downtime and recovery gives you a moment to peer at what has +been, and prepare for the ups and downs of the coming year. +What will be the same? What will change? What parts of your +life can you simply hard link to the habits of the past? And +what will you have to incrementally add and integrate into your +ever-evolving life? Until next time, I am, ~Integrity Mather +[^winterbreak]: Are the rumors of an AI Winter true? Do LLMs get lazier during December? [Ian Arawjo](https://ianarawjo.com/), +author of [ChainForge](https://github.com/ianarawjo/ChainForge), spotted [flaws](https://twitter.com/IanArawjo/status/1734924051242484223) in Rob Lynch's significant result that +GPT-4-Turbo produces fewer tokens when December is mentioned in its prompt, but [https://twitter.com/messages/54913-1586500784514113536](Cousin Lynch) is continuing to investigate at press time. See our earlier memo on the phenomenon, "The True Meaning of Wintermute: Northern Hemisphere Seasonability in Tessier-Ashpool AIs", Automatic Jack, Almnck. 1981. + +[^scarydevilmonastery]: Alt.sysadmin.recovery's monastic wisdom, is only dimly remembered + now that posting to Usenet and painting your nails black are +no longer professional requirements for network engineers. Nonetheless the newsgroup provided several powerful and vile proverbs on the importance of +backups, the foulest of which remain unrecorded in Heather Garvey's [extant +quotes +file](https://web.archive.org/web/20060423055444/http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html), +Garvey's document was, you may note, updated mere hours before Y2K day. +This lends some credence to the the theory that an eldritch rite committed by +the Monks on that day led to the key events in the subsequent Rupture of the +Nerds, including the abandonment of Usenet, ASR regulars Kirrily "Skud" Roberts' co-founding of the Geek +Feminism movement, and Charlie Stross being press-ganged into leaving system administraiton and forced by mysterious VC backers to become +prominent science fiction author, thereafter, following the success of Accelerando, +to be clumsily digitized into an AI corporate entity, programmed to +repeatedly deny that corporations were people until the West Lothian and Turing police backed away +from looking more any closely into his corporate structure. See, "Saint Charles of Stross: A Prohairetic Hagiography", G. Vittoria, Almnck. 2006. + +[^xmasdeaths]: Most recently -- but not *that* recently -- examined in +Phillips, D. P., Jarvinen, J. R., Abramson, I., & Phillips, R. R. (2004). Cardiac Mortality Is Higher Around Christmas and New Year’s Than at Any Other Time. *Circulation*, 110(25), 3781–3788. https://doi.org/10.1161/01.cir.0000151424.02045.f7. + +Later studies suggest that people don't get any crazier or suicidal at Christmas (See +Schneider, E., Liwinski, T., Imfeld, L., Lang, U. E., & Brühl, A. B. (2023). Who is afraid of Christmas? The effect of Christmas and Easter holidays on psychiatric hospitalizations and emergencies—Systematic review and single center experience from 2012 to 2021. *Frontiers in Psychiatry*, 13. https://doi.org/10.3389/fpsyt.2022.1049935 ), and it may just be the same effect as more people dying in the medical system during weekends, See +Castaño-Pérez, S., Medina García, J.A. & Cabrera de León, A. The dose–response effect of time between emergency admission and inpatient care on mortality. Sci Rep 13, 22244 (2023). https://doi.org/10.1038/s41598-023-49090-5 . + +For explorations of the theory that excess Winter deaths are caused by high-energy particle emissions from near-lightspeed Western gift-deliverers, see "Bremstrahlung und Blitzen!: A Comparison of Incidence Rates of Thyroid Cancer among the Naughty, Nice, and Non-Believers", Almnck. 1823.