186 lines
11 KiB
Markdown
186 lines
11 KiB
Markdown
---
|
||
author: "[Integrity Mather](/~integrity/)"
|
||
title: Look Back Up
|
||
date: January 1, 2024
|
||
---
|
||
::: {#markdownBody .markdownBody}
|
||
{ .initial }
|
||
he year ends, the North Pole tips its deepest bow to the darkness, and we see
|
||
even large language models have been [taking it easy](https://arstechnica.com/information-technology/2023/12/is-chatgpt-becoming-lazier-because-its-december-people-run-tests-to-find-out/) for the
|
||
winter[^winterbreak]. But for the faithful maintainer of systems, there's still
|
||
work to be done, here in the cooling embers of the year.
|
||
|
||
Now is a fine time to dust off your backup scripts and see if they're working
|
||
as they should. An untested backup is no backup at all, said the wise elders of
|
||
the [Scary Devil
|
||
Monastery](http://www.faqs.org/faqs/sysadmin-recovery/)[^scarydevilmonastery],
|
||
and if we want to set the new year off to its best start, we should ensure we
|
||
can pause, tear down, and re-start its march, even at its lowest
|
||
points.
|
||
|
||
When examining my backup and restore process, I took the opportunity this year
|
||
to test my backups and increase the size of my root partition on my server.
|
||
|
||
One last manual backup of the 256GiB SSD in my trusty server `boat` for the
|
||
sake of the old, and then an attempted restore for the sake of the new: to a
|
||
spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well,
|
||
the restored backup in tub would have new room to grow, and I could swap
|
||
that drive into `boat` with minimal downtime.
|
||
|
||
That moment of hardware-swapping would mean that `boat` would have to be
|
||
shut down and then restarted anew. Humans need their respite over the holiday break:
|
||
but should I have granted my server the same indulgence? My plan accepted
|
||
that `boat` would be offline for, I hoped, a small slice of time.
|
||
|
||
There are more convoluted ways to ensure that none of my websites, file
|
||
syncing, and miscellaneous tools flickered, even for a moment. I could have
|
||
temporarily switched my DNS settings to point at the fresh clean `tub`, for
|
||
instance, while overwriting `boat`. Or perhaps just repurposed `boat` for
|
||
gentler, less demanding tasks, giving it the end-of-year gift of a well-deserved
|
||
retirement, and passing to `tub` a new year's responsibility of hosting my
|
||
server processes.
|
||
|
||
Within the scale of my own life, I do believe that uptime is overrated. We are
|
||
surrounded by tools bent into the shapes demanded by large tech companies, for
|
||
whom site reliability is their first commandment.
|
||
|
||
But we are humans, not corporations. We run at our own pace. If Homer nods, so
|
||
can my home lab. Christmas has its folklore of [visitors
|
||
rebuffed](https://en.wikipedia.org/wiki/Befana), but hopefully, my friends have
|
||
other pressing matters this time of year than hitting reload on my sites.
|
||
|
||
Yes, a personal web server can go down for a few moments -- as long as it bobs
|
||
back up.
|
||
|
||
Which, with restored filing systems, is ever the question. Will the backup
|
||
truly come back up? A reset and reboot may also be a time for sinking
|
||
exhaustion and death. Human deaths seem positively correlated with the change
|
||
of pace of the holidays[^xmasdeaths], and so too, drives and computers will
|
||
dutifully run for years, but expire after a moment's rest.
|
||
|
||
<center>
|
||
{width=50%}
|
||
</center>
|
||
|
||
I use ["Relax-And-Recover"](https://relax-and-recover.org/) (REAR), a Linux
|
||
disaster recovery system from when sysadmins wrote shell code and *liked it*.
|
||
REAR is a sprawling set of shell scripts that runs your choice of backup code
|
||
-- from [`rsync`](https://jumpcloud.com/blog/how-to-backup-linux-system-rsync)
|
||
to [`borg`](https://borgbackup.readthedocs.io/) -- over your entire Linux
|
||
filesystem, on a regular, cron-determined, basis. Having seen to that prosaic
|
||
task, REAR will also create a [minimal, bootable
|
||
blob](https://relax-and-recover.org/usage/#recovery_from_usb). The blob,
|
||
stuffed onto a USB drive, CD-ROM, or networked drive, will boot into a rescue
|
||
Linux, and lead you through the reconstruction and re-partitioning of a drive
|
||
that will emerge the same shape as your original machine. Then it will pull
|
||
down your backups, and restore this drive to the precise state that your
|
||
backups recalled it.
|
||
|
||
Readers born into our age of strong types and weak stomachs may be balking at
|
||
the idea of entrusting their restoration to a bunch of
|
||
[stringly-typed](https://samgrayson.me/essays/stop-writing-shell-scripts/) Bash
|
||
scripts. REAR's 20K(!) lines of shellcode intimate that it has reached levels
|
||
of complexity, beyond that which bashism can safely grasp.
|
||
|
||
But this is *sysadmin* shell code. That terrifying KLOC is defensive, modular
|
||
coding of the highest order. For the casual shell user, REAR's operation is
|
||
comprehensible, its failure modes anticipated and loudly announced, and tweaks
|
||
and mediations are semi-obvious. REAR's authors offer "a relaxing
|
||
recovery", and the ghosts of sysadminning past do not lightly emphasize relaxation.
|
||
|
||
While I was never *not* relaxed during my holiday restoration. I did
|
||
occasionally carol a high-pitched note or two of concern.
|
||
|
||
Two hefty snowbanks stood between me and a perfectly clean restore. Since I
|
||
first installed it, I have had REAR create ISO files for burning onto a rescue CD-ROM. "CD" now stands for "Cretaceous Disk": I have not
|
||
used one for over a decade. Pouring a bootable ISO into a contemporary USB
|
||
drive drive turns out to be surprisingly tricky, and I can never remember how
|
||
to do it. In the end, I was forced, humiliatingly, to read [REAR's
|
||
FAQ](https://relax-and-recover.org/documentation/faq), where they recommended a
|
||
meatier graphical utility, [UnetBootin](https://unetbootin.github.io/), for achieving this.
|
||
In the future, I've set REAR to output those bootable blobs as
|
||
[RAWDISK](https://relax-and-recover.org/rear-user-guide/basics/configuration.html),
|
||
which can be burned (warmed?) onto a USB drive.
|
||
|
||
REAR's ecumenical acceptance of external backup programs can introduce
|
||
tremors into its solid scripts. My use of a distribution [borg
|
||
backup](https://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/borg-example.conf) package
|
||
broke the restore. REAR [scans
|
||
executables](https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/build/default/990_verify_rootfs.sh#L51)
|
||
that it plans to include on its rescue bootable blob to detect what libraries
|
||
they require, so that it may copy those over. Sadly, the borg executable can be
|
||
either a binary executable -- for which this suffices -- or a Python script,
|
||
whose demands REAR cannot fathom. This is undoubtedly a bug a future
|
||
REAR will fix. In the meantime, I copied over the [binary
|
||
borg](https://borgbackup.readthedocs.io/en/stable/installation.html#standalone-binary)
|
||
into `/usr/local/bin` instead of the Pythonic borg of the
|
||
Debian repos.
|
||
|
||
After those tweaks, `tub` was filled with the form of `boat`'s Christmas past.
|
||
I swapped over the two drives, holding my breath, and losing the little M.2 SSD
|
||
screw as
|
||
always (these [plastic nubbin replacements](https://www.ebay.com/itm/275937873783) tempted me in the fallout, though apparently a [pencil sharpener](https://linustechtips.com/topic/1319971-missing-a-screw-for-your-m2-ssd-check-this-out/) will work in a pinch). `Boat`
|
||
wobbled and then, like a lucky [North Sea seafarer](https://www.youtube.com/watch?v=qlrvzLRgzdc), bobbed back up from an early visit to "[Fiddler's Green](https://en.wikipedia.org/wiki/Fiddler%27s_Green)".
|
||
|
||
Testing your recovery plans during your end-of-year downtime gives you a
|
||
moment to look back at the record of what has been, and prepare for the ups and
|
||
downs of the coming year. What will be the same? What will change? What parts
|
||
of your life can you simply hard link to the habits of the past? And what will
|
||
you have to incrementally add and integrate into your ever-evolving life?
|
||
|
||
~[Integrity Mather](/~integrity/)
|
||
:::
|
||
|
||
[^winterbreak]: Are the rumors of an AI Winter true? Do LLMs get lazier during
|
||
December? [Ian Arawjo](https://ianarawjo.com/), author of
|
||
[ChainForge](https://github.com/ianarawjo/ChainForge), spotted
|
||
[flaws](https://twitter.com/IanArawjo/status/1734924051242484223) in Rob
|
||
Lynch's significant result that GPT-4-Turbo produces fewer tokens when
|
||
December is mentioned in its prompt, but
|
||
[Cousin Lynch](https://twitter.com/messages/54913-1586500784514113536) is
|
||
continuing to investigate at press time. See our earlier memo on the
|
||
phenomenon, "The True Meaning of Wintermute: Northern Hemisphere
|
||
Seasonability in Tessier-Ashpool AIs", Automatic Jack, Almnck. 1981.
|
||
|
||
[^scarydevilmonastery]: `alt.sysadmin.recovery`\’s monastic wisdom is only dimly
|
||
remembered, now that posting to Usenet and painting your nails black are no
|
||
longer professional requirements for ops. The newsgroup provided several
|
||
powerful and vile proverbs on the importance of backups, the foulest of
|
||
which remain unrecorded in Heather Garvey's [extant quotes
|
||
file](https://web.archive.org/web/20060423055444/http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html).
|
||
Garvey's document was, you may note, updated mere hours before Y2K day.
|
||
This lends some credence to the theory that an eldritch rite committed
|
||
by the Monks on that day led to the key events in the subsequent Rupture of
|
||
the Nerds, including the abandonment of Usenet, ASR regulars Kirrily "Skud"
|
||
Roberts' co-founding of the Geek Feminism movement, and Charlie Stross
|
||
being press-ganged into leaving system administration by mysterious backers
|
||
in the Humanities Industrial Complex to become a prominent science fiction
|
||
author. Thereafter, following the success of Accelerando, he was reputed to
|
||
have been clumsily digitized into an AI corporate entity, programmed to
|
||
deny that corporations could ever be people (and vice-versa) until the West
|
||
Lothian and Turing police backed away. See, "Saint Charles of Stross: A
|
||
Prohairetic Hagiography", G. Vittoria, Almnck. 2006.
|
||
|
||
[^xmasdeaths]: Most recently -- but not *that* recently -- examined in
|
||
Phillips, D. P., Jarvinen, J. R., Abramson, I., & Phillips, R. R. (2004).
|
||
Cardiac Mortality Is Higher Around Christmas and New Year’s Than at Any
|
||
Other Time. *Circulation*, 110(25), 3781–3788.
|
||
https://doi.org/10.1161/01.cir.0000151424.02045.f7.
|
||
|
||
Later studies suggest that people do not grow crazier or more suicidal at
|
||
Christmas (See Schneider, E., Liwinski, T., Imfeld, L., Lang, U. E., &
|
||
Brühl, A. B. (2023). Who is afraid of Christmas? The effect of Christmas
|
||
and Easter holidays on psychiatric hospitalizations and
|
||
emergencies—Systematic review and single center experience from 2012 to
|
||
2021. *Frontiers in Psychiatry*, 13.
|
||
https://doi.org/10.3389/fpsyt.2022.1049935 ). Perhaps it could be simply the same
|
||
effect as more people dying in the medical system during weekends, See
|
||
Castaño-Pérez, S., Medina García, J.A. & Cabrera de León, A. The
|
||
dose-response effect of time between emergency admission and inpatient care
|
||
on mortality. Sci Rep 13, 22244 (2023).
|
||
https://doi.org/10.1038/s41598-023-49090-5 ?
|
||
|
||
For explorations of the theory that excess Winter deaths are caused by
|
||
high-energy particle emissions from near-lightspeed Western
|
||
gift despatch, see "Bremsstrahlung und Blitzen!: Incidence Rates of
|
||
Thyroid Cancer among the Naughty, Nice, and Non-Believing", Almnck. 1823.
|