Almanack/almnck
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

edit: Changes to New Years Backups

Browse source
This commit is contained in:
Danny O'Brien 2024-01-02 00:13:09 -08:00
parent f3128131c4
commit 122ebef2e2
3 changed files with 255 additions and 67 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
assets/editorial/01edits/integrity-new-years-backups/0PASS.org Normal file
View file

@ -0,0 +1,13 @@
* Editorial Passes
** DONE HIRED [/] -- Commissioning and contract
** DONE STYLE [/] -- Editor and writer work together on style and structure.
** TODO FACTS [/] -- Fact-checking
- [ ] Links. Ensure we have links to as many places as possible, and that they work.
- [ ] Check with third-parties. To the greatest extent, we should reach out to everyone mentioned or to relevant experts and confirm that factual statements are correct.
- [ ] Terminology. Check that terminology is correct.
** TODO SPELL [/] -- Grammar and spell checking
** TODO GRAPH [/] -- Graphics. Every piece needs a illustrated initial and at least one image.
** TODO FINAL [/] -- Final look through before publication.
- [ ] Add any Almnck footnote references to the canonical [[../../../../doc/footnotes.org][footnotes]] list.
- [ ] Final check of all links.
** TODO WRAPUP [/] -- Final administrivia. YOU MUST DO THIS.

143
assets/editorial/01edits/integrity-new-years-backups/FACT.md Normal file
View file

@ -0,0 +1,143 @@
---
Author: Integrity Jones
Title: Looking Back Up
Published: 2024-01-01
---
To Old Danny, greetings:
The solar nights have barely stopped closing in, and we see even large language
models have been taking it easy for the winter. But for the faithful maintainer
of systems, there's still work to be done, here in the cooling embers of the year.
> Solar nights -- does this mean anything (No)
> Link to LLM stories
Now is a fine time of year to dust off your backup scripts, and see if they're
working as they should. An untested backup is a no backup at all, said the wise
ghosts of the Scary Devil Monastery, and if we want to set the year off to its
best start, we should ensure we can pause, tear-down, and re-start it at its lowest
points.
> Link to Scary Devil Monastery
When examining my backup and restore process, took the opportunity this year to
quadruple the size of my root partition on my server.
One last manual backup of the 256GiB SSD in my trusty server `boat` for the
sake of the old, and then an attempted restore for the sake of the new: to a
spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well,
the restored backup in tub would have new room to grow, and I could swap
out that drive back into `boat` with minimal downtime.
> Box on hard drives? Link to goods?
That moment of hardware-swapping would mean that `boat` would have to be
shutdown and then restarted anew. Humans need their respite over the holiday break:
but should I have permitted my server the same indulgence? My plan accepted
that boat would be offline for, one hopes, a small slice of time.
There are more convoluted ways to ensure that none of my websites, file syncing, and
miscellaneous tools did not flicker, even for a moment.
I could have switched my DNS settings to the fresh clean `tub`, for instance,
while overwriting `boat`. Or perhaps just repurposed boat for gentler, less
demanding tasks, giving it the end-of-year gift of a well-deserved retirement,
and switched to `tub` a new year's responsibility of my hosting my main home
processes.
At the smallest scale, I do believe that uptime is overrated. We are surrounded
by tools bent into the shapes demanded by large tech companies, for whom site
reliability is their first commandment.
But we are humans, not corporations. If even Homer nods, so can my home lab.
Christmas has its own folklore of visitors rebuffed, but hopefully my friends
have better things to do at this time than hit reload on my websites. Yes, a
personal webserver can go down for a few moments -- as long as it comes back up.
> Link to Home nods line
Which, with restored filing systems, is ever the question. Will the backup
truly come back up? A restored and rebooted machine may also be a time for
failure and death, just as, morbidly, we should note that human coronary deaths
seem positively correlated with the change of pace of the holidays.
> Link to coronary death paper
I use the more positively-framed "Relax-And-Recover" (REAR), a recover system
from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash
scripts that runs your choice of backup code -- from rsync to borg-- over your
entire linux root, on a regular, cron-determined, basis. Having seen to that
prosaic task, it will also create minimal, bootable blob. The blob, stuffed
onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux,
and gently lead you through reconstructing a re-partitioning of a drive
that will. At its end, the partition map on the machine you have booted
will be exactly the same shape as your original machine. Then it will pull
down your precious backups, and restore this drive to the precise state
that your backups recalled it. A perfect, royal, restoration.
> Link to Relax and Recover
Younger readers born into our age of strong types and weak stomachs, may be
balking at the idea of entrusting restoration to a bunch of stringly-typed Bash
scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the
edges of complexity beyond that you might think bashism can bear.
> Link to definition of stringly
> Link to proof of REAR's 20K of KLOCs
But this is /sysadmin/ shell code. That terrifying KLOC is
defensive, modular coding of the highest order. For the casual shell user,
REAR's operation is comprehensible, the failure modes anticipated, and
tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing
recovery", and ghosts of sysadminning past do not emphasise enhancing your
calm in those moments lightly.
> Link to REAR
I was never /not/ relaxed during my holiday restoration, but I did
occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO
files for burning onto a CD-ROM that I could restore from. CD stands for
"Cretaceous Disk". Even I have not used one of them in anger in over a decade.
Converting a bootable ISO into a bootable USB drive drive turned out to be
surprisingly tricky, and I can never remember how to do it. In the end I was
forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy
utility, https://unetbootin.github.io/, for achieving this. In the future, I've
set REAR to output those bootable blobs as RAWDISK, which can be burned
(warmed?) onto a USB.
> Link to REAR's FAQ
> Link to UnetBootlin
REAR's ecumenical acceptance of multiple backup programs introduces additional
complexity into its scripts. My use of borg backup tripped up the restore. REAR
scans executables that it plans to include on its rescue bootable blob, so it
can detect what libraries they need and copy those over. Sadly, the borg
executable can be either a binary executable (for which this works), or just a
Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a
future REAR will fix, but in the meantime I just copied over the binary borg
into /usr/local/bin and used that instead of the Pythonic borg of the Debian
repos.
> Link to binary executable
> Link to executable/python script issue on github
After those tweaks, tub was filled with the form of boat's Christmas past. I swapped
over the two drives, holding my breath, and losing the little SSD screw as
always (these plastic nubbin replacements tempted me in the fallout). Boat
wobbled and then, like a plucky North Sea veteran, bobbed back up.
> Link to nubbin ali express sale
> Link to North Sea tiktok music
Testing your recovery plans during your own end-of-year downtime and recovery
gives you a moment to peer at what has been, and prepare for the ups and downs
of the coming year. What will be the same? What will change? What parts of your
life can you simply hard link to the habits of the past? And what will you have
to incrementally add and integrate into your ever-evolving life?
Until next time, I am,
~Integrity Mather

166
assets/editorial/01edits/integrity-new-years-backups/new-years-backups.md
View file

@ -1,114 +1,146 @@
---
Author: Integrity Jones
Author: Integrity Mather
Title: Looking Back Up
Published: 2024-01-01
---
To Old Danny, greetings:
The solar nights have barely stopped closing in, and we see even large language
models have been taking it easy for the winter. But for the faithful maintainer
of systems, there's still work to be done, here in the cooling embers of the year.
The nights have barely stopped closing in the Northern hemisphere, and we see
even large language models have been taking it easy for the
winter[^winterbreak]. But for the faithful maintainer of systems, there's still
work to be done, here in the cooling embers of the year.
Now is a fine time of year to dust off your backup scripts, and see if they're
working as they should. An untested backup is no backup at all, said the wise
ghosts of the Scary Devil Monastery, and if we want to set the year off to its
best start, we should ensure we can pause, tear-down, and re-start it even at its lowest
points.
Now is a fine time to dust off your backup scripts and see if they're working
as they should. An untested backup is no backup at all, said the wise elders of
the [Scary Devil Monastery](http://www.faqs.org/faqs/sysadmin-recovery/)[^scarydevilmonastery], and if we want to set the new year off to its best
start, we should ensure we can pause, tear-down, and re-start the marching
present even at its lowest points.
When examining my backup and restore process, took the opportunity this year to
quadruple the size of my root partition on my server.
When examining my backup and restore process, I took the opportunity this year
to test my backups while quadrupling the size of my root partition on my
server.
One last manual backup of the 256GiB SSD in my trusty server `boat` for the
sake of the old, and then an attempted restore for the sake of the new: to a
spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well,
the restored backup in tub would have new room to grow, and I could swap
out that drive back into `boat` with minimal downtime.
that drive into `boat` with minimal downtime.
That moment of hardware-swapping would mean that `boat` would have to be
shutdown and then restarted anew. Humans need their respite over the holiday break:
but should I have permitted my server the same indulgence? My plan accepted
that boat would be offline for, one hopes, a small slice of time.
There are more convoluted ways to ensure that none of my websites, file syncing, and
miscellaneous tools did not flicker, even for a moment.
but should I have granted my server the same indulgence? My plan accepted
that `boat` would be offline for, I hoped, a small slice of time.
I could have switched my DNS settings to the fresh clean `tub`, for instance,
while overwriting `boat`. Or perhaps just repurposed boat for gentler, less
demanding tasks, giving it the end-of-year gift of a well-deserved retirement,
and switched to `tub` a new year's responsibility of my hosting my main home
There are more convoluted ways to ensure that none of my websites, file
syncing, and miscellaneous tools did not flicker, even for a moment. I could
have switched my DNS settings to the fresh clean `tub`, for instance, while
overwriting `boat`. Or perhaps just repurposed `boat` for gentler, less demanding
tasks, giving it the end-of-year gift of a well-deserved retirement, and
switched to `tub` a new year's responsibility of my hosting my main home
processes.
At the smallest scale, I do believe that uptime is overrated. We are surrounded
by tools bent into the shapes demanded by large tech companies, for whom site
reliability is their first commandment.
At the scale of my own life, I do believe that uptime is overrated. We are
surrounded by tools bent into the shapes demanded by large tech companies, for
whom site reliability is their first commandment.
But we are humans, not corporations. If even Homer nods, so can my home lab.
Christmas has its own folklore of visitors rebuffed, but hopefully my friends
But we are humans, not corporations. If Homer nods, so can my home lab.
Christmas has its own folklore of [visitors rebuffed](https://en.wikipedia.org/wiki/Befana), but hopefully my friends
have better things to do at this time than hit reload on my websites. Yes, a
personal webserver can go down for a few moments -- as long as it comes back up.
personal webserver can go down for a few moments -- as long as it comes back
up.
Which, with restored filing systems, is ever the question. Will the backup
truly come back up? A restored and rebooted machine may also be a time for
failure and death, just as, morbidly, we should note that human coronary deaths
seem positively correlated with the change of pace of the holidays.
failure and death, just as, morbidly, we should note that human deaths
seem positively correlated with the change of pace of the holidays[^xmasdeaths].
I use the more positively-framed "Relax-And-Recover" (REAR), a recover system
I use the more positively-framed ["Relax-And-Recover"](https://relax-and-recover.org/) (REAR), a disaster recovery system
from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash
scripts that runs your choice of backup code -- from rsync to borg-- over your
scripts that runs your choice of backup code -- from `[rsync](https://jumpcloud.com/blog/how-to-backup-linux-system-rsync)` to `[borg](https://borgbackup.readthedocs.io/)` -- over your
entire linux root, on a regular, cron-determined, basis. Having seen to that
prosaic task, it will also create minimal, bootable blob. The blob, stuffed
onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux,
and gently lead you through reconstructing a re-partitioning of a drive
that will. At its end, the partition map on the machine you have booted
will be exactly the same shape as your original machine. Then it will pull
down your precious backups, and restore this drive to the precise state
that your backups recalled it. A perfect, royal, restoration.
prosaic task, it will also create [minimal, bootable blob](https://relax-and-recover.org/usage/#recovery_from_usb). The blob, stuffed
onto an USB drive, CD-ROM, or networked drive, will boot into a minimal Linux,
and lead you through the reconstruction and re-partitioning of a drive
that will emerge the same shape as your original machine. Then it will pull
down your backups, and restore this drive to the precise state
that your backups recalled it. A perfect, royal, restoration.
Younger readers born into our age of strong types and weak stomachs, may be
balking at the idea of entrusting restoration to a bunch of stringly-typed Bash
scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the
edges of complexity beyond that you might think bashism can bear.
Readers born into our age of strong types and weak stomachs may be balking at
the idea of entrusting restoration to a bunch of [stringly-typed](https://samgrayson.me/essays/stop-writing-shell-scripts/) Bash scripts.
REAR's 20K(!) lines of shellcode intimate that it has reached the edges of
complexity beyond that you might think bashism can bear.
But this is /sysadmin/ shell code. That terrifying KLOC is
But this is *sysadmin* shell code. That terrifying KLOC is
defensive, modular coding of the highest order. For the casual shell user,
REAR's operation is comprehensible, the failure modes anticipated, and
REAR's operation is comprehensible, failure modes anticipated and loudly-announced, and
tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing
recovery", and ghosts of sysadminning past do not emphasise enhancing your
calm in those moments lightly.
recovery", and ghosts of sysadminning past do not lightly emphasise enhancing your
calm in those moments.
I was never /not/ relaxed during my holiday restoration, but I did
occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO
I was never /not/ relaxed during my holiday restoration. I did
occasionally emit a high-pitched carol or two of concern. I have had REAR create ISO
files for burning onto a CD-ROM that I could restore from. CD stands for
"Cretaceous Disk". Even I have not used one of them in anger in over a decade.
Converting a bootable ISO into a bootable USB drive drive turned out to be
"Cretaceous Disk" nowadays: Even I have not used them in anger for over a decade.
Converting a bootable ISO into a bootable USB drive drive turns out to be
surprisingly tricky, and I can never remember how to do it. In the end I was
forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy
utility, https://unetbootin.github.io/, for achieving this. In the future, I've
set REAR to output those bootable blobs as RAWDISK, which can be burned
forced, humiliatingly, to read [REAR's FAQ](https://relax-and-recover.org/documentation/faq), where they recommended a heavy
utility, [UnetBootin](https://unetbootin.github.io/), for achieving this. In the future, I've
set REAR to output those bootable blobs as [RAWDISK](https://relax-and-recover.org/rear-user-guide/basics/configuration.html), which can be burned
(warmed?) onto a USB.
REAR's ecumenical acceptance of multiple backup programs introduces additional
REAR's ecumenical acceptance of multiple backup programs can introduce additional
complexity into its scripts. My use of borg backup tripped up the restore. REAR
scans executables that it plans to include on its rescue bootable blob, so it
can detect what libraries they need and copy those over. Sadly, the borg
[scans executables](https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/build/default/990_verify_rootfs.sh#L51) that it plans to include on its rescue bootable blobto
detect what libraries they require, so that it may copy those over. Sadly, the borg
executable can be either a binary executable (for which this works), or just a
Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a
future REAR will fix, but in the meantime I just copied over the binary borg
into /usr/local/bin and used that instead of the Pythonic borg of the Debian
future REAR will fix, but in the meantime I just copied over the [binary borg](https://borgbackup.readthedocs.io/en/stable/installation.html#standalone-binary)
into `/usr/local/bin` and used that instead of the Pythonic borg of the Debian
repos.
After those tweaks, tub was filled with the form of boat's Christmas past. I swapped
over the two drives, holding my breath, and losing the little SSD screw as
always (these plastic nubbin replacements tempted me in the fallout). Boat
wobbled and then, like a plucky North Sea veteran, bobbed back up.
After those tweaks, `tub` was filled with the form of `boat`'s Christmas past. I swapped
over the two drives, holding my breath, and losing the little M.2 SSD screw as
always (these [plastic nubbin replacements](https://www.ebay.com/itm/275937873783) tempted me in the fallout, though apparently a [pencil sharpener](https://linustechtips.com/topic/1319971-missing-a-screw-for-your-m2-ssd-check-this-out/) also works in a pinch). `Boat`
wobbled and then, like a lucky [North Sea seafarer](https://www.youtube.com/watch?v=qlrvzLRgzdc), bobbed back up from an early visit to "[Fiddler's Green](https://en.wikipedia.org/wiki/Fiddler%27s_Green)".
Testing your recovery plans during your own end-of-year downtime and recovery
gives you a moment to peer at what has been, and prepare for the ups and downs
of the coming year. What will be the same? What will change? What parts of your
life can you simply hard link to the habits of the past? And what will you have
to incrementally add and integrate into your ever-evolving life?
Testing your recovery plans during your own end-of-year
downtime and recovery gives you a moment to peer at what has
been, and prepare for the ups and downs of the coming year.
What will be the same? What will change? What parts of your
life can you simply hard link to the habits of the past? And
what will you have to incrementally add and integrate into your
ever-evolving life?
Until next time, I am,
~Integrity Mather
[^winterbreak]: Are the rumors of an AI Winter true? Do LLMs get lazier during December? [Ian Arawjo](https://ianarawjo.com/),
author of [ChainForge](https://github.com/ianarawjo/ChainForge), spotted [flaws](https://twitter.com/IanArawjo/status/1734924051242484223) in Rob Lynch's significant result that
GPT-4-Turbo produces fewer tokens when December is mentioned in its prompt, but [https://twitter.com/messages/54913-1586500784514113536](Cousin Lynch) is continuing to investigate at press time. See our earlier memo on the phenomenon, "The True Meaning of Wintermute: Northern Hemisphere Seasonability in Tessier-Ashpool AIs", Automatic Jack, Almnck. 1981.
[^scarydevilmonastery]: Alt.sysadmin.recovery's monastic wisdom, is only dimly remembered
now that posting to Usenet and painting your nails black are
no longer professional requirements for network engineers. Nonetheless the newsgroup provided several powerful and vile proverbs on the importance of
backups, the foulest of which remain unrecorded in Heather Garvey's [extant
quotes
file](https://web.archive.org/web/20060423055444/http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html),
Garvey's document was, you may note, updated mere hours before Y2K day.
This lends some credence to the the theory that an eldritch rite committed by
the Monks on that day led to the key events in the subsequent Rupture of the
Nerds, including the abandonment of Usenet, ASR regulars Kirrily "Skud" Roberts' co-founding of the Geek
Feminism movement, and Charlie Stross being press-ganged into leaving system administraiton and forced by mysterious VC backers to become
prominent science fiction author, thereafter, following the success of Accelerando,
to be clumsily digitized into an AI corporate entity, programmed to
repeatedly deny that corporations were people until the West Lothian and Turing police backed away
from looking more any closely into his corporate structure. See, "Saint Charles of Stross: A Prohairetic Hagiography", G. Vittoria, Almnck. 2006.
[^xmasdeaths]: Most recently -- but not *that* recently -- examined in
Phillips, D. P., Jarvinen, J. R., Abramson, I., & Phillips, R. R. (2004). Cardiac Mortality Is Higher Around Christmas and New Year’s Than at Any Other Time. *Circulation*, 110(25), 3781–3788. https://doi.org/10.1161/01.cir.0000151424.02045.f7.
Later studies suggest that people don't get any crazier or suicidal at Christmas (See
Schneider, E., Liwinski, T., Imfeld, L., Lang, U. E., & Brühl, A. B. (2023). Who is afraid of Christmas? The effect of Christmas and Easter holidays on psychiatric hospitalizations and emergencies—Systematic review and single center experience from 2012 to 2021. *Frontiers in Psychiatry*, 13. https://doi.org/10.3389/fpsyt.2022.1049935 ), and it may just be the same effect as more people dying in the medical system during weekends, See
Castaño-Pérez, S., Medina García, J.A. & Cabrera de León, A. The dose–response effect of time between emergency admission and inpatient care on mortality. Sci Rep 13, 22244 (2023). https://doi.org/10.1038/s41598-023-49090-5 .
For explorations of the theory that excess Winter deaths are caused by high-energy particle emissions from near-lightspeed Western gift-deliverers, see "Bremstrahlung und Blitzen!: A Comparison of Incidence Rates of Thyroid Cancer among the Naughty, Nice, and Non-Believers", Almnck. 1823.