---
author: "[Integrity Mather](/~integrity/)"
title: Look Back Up
date: January 1, 2024
--- 
::: {.maintext}
![T](res/new-years-backups-initial.png){ .initial }
he year ends, the North Pole tips its deepest bow to the darkness, and we see
even large language models have been [taking it easy](https://arstechnica.com/information-technology/2023/12/is-chatgpt-becoming-lazier-because-its-december-people-run-tests-to-find-out/) for the
winter[^winterbreak]. But for the faithful maintainer of systems, there's still
work to be done, here in the cooling embers of the year.

Now is a fine time to dust off your backup scripts and see if they're working
as they should. An untested backup is no backup at all, said the wise elders of
the [Scary Devil
Monastery](http://www.faqs.org/faqs/sysadmin-recovery/)[^scarydevilmonastery],
and if we want to set the new year off to its best start, we should ensure we
can pause, tear down, and re-start its march, even at its lowest
points. 

When examining my backup and restore process, I took the opportunity this year
to test my backups and increase the size of my root partition on my server.

One last manual backup of the 256GiB SSD in my trusty server `boat` for the
sake of the old, and then an attempted restore for the sake of the new: to a
spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well,
the restored backup in tub would have new room to grow, and I could swap
that drive into `boat` with minimal downtime.

That moment of hardware-swapping would mean that `boat` would have to be
shut down and then restarted anew. Humans need their respite over the holiday break:
but should I have granted my server the same indulgence? My plan accepted
that `boat` would be offline for, I hoped, a small slice of time.

There are more convoluted ways to ensure that none of my websites, file
syncing, and miscellaneous tools flickered, even for a moment. I could have
temporarily switched my DNS settings to point at the fresh clean `tub`, for
instance, while overwriting `boat`. Or perhaps just repurposed `boat` for
gentler, less demanding tasks, giving it the end-of-year gift of a well-deserved
retirement, and passing to `tub` a new year's responsibility of hosting my
server processes.

Within the scale of my own life, I do believe that uptime is overrated. We are
surrounded by tools bent into the shapes demanded by large tech companies, for
whom site reliability is their first commandment. 

But we are humans, not corporations. We run at our own pace. If Homer nods, so
can my home lab. Christmas has its folklore of [visitors
rebuffed](https://en.wikipedia.org/wiki/Befana), but hopefully, my friends have
other pressing matters this time of year than hitting reload on my sites.

Yes, a personal web server can go down for a few moments -- as long as it bobs
back up.

Which, with restored filing systems, is ever the question. Will the backup
truly come back up? A reset and reboot may also be a time for sinking
exhaustion and death. Human deaths seem positively correlated with the change
of pace of the holidays[^xmasdeaths], and so too, drives and computers will
dutifully run for years, but expire after a moment's rest.

<center>
![~~~](res/new-years-backups-dingbat1.png){width=50%}
</center>

I use ["Relax-And-Recover"](https://relax-and-recover.org/) (REAR), a Linux
disaster recovery system from when sysadmins wrote shell code and *liked it*.
REAR is a sprawling set of shell scripts that runs your choice of backup code
-- from [`rsync`](https://jumpcloud.com/blog/how-to-backup-linux-system-rsync)
to [`borg`](https://borgbackup.readthedocs.io/) -- over your entire Linux
filesystem, on a regular, cron-determined, basis. Having seen to that prosaic
task, REAR will also create a [minimal, bootable
blob](https://relax-and-recover.org/usage/#recovery_from_usb). The blob,
stuffed onto a USB drive, CD-ROM, or networked drive, will boot into a rescue
Linux, and lead you through the reconstruction and re-partitioning of a drive
that will emerge the same shape as your original machine. Then it will pull
down your backups, and restore this drive to the precise state that your
backups recalled it.

Readers born into our age of strong types and weak stomachs may be balking at
the idea of entrusting their restoration to a bunch of
[stringly-typed](https://samgrayson.me/essays/stop-writing-shell-scripts/) Bash
scripts. REAR's 20K(!) lines of shellcode intimate that it has reached levels
of complexity, beyond that which bashism can safely grasp.

But this is *sysadmin* shell code. That terrifying KLOC is defensive, modular
coding of the highest order. For the casual shell user, REAR's operation is
comprehensible, its failure modes anticipated and loudly announced, and tweaks
and mediations are semi-obvious. REAR's authors offer "a relaxing
recovery", and the ghosts of sysadminning past do not lightly emphasize relaxation.

While I was never *not* relaxed during my holiday restoration. I did
occasionally carol a high-pitched note or two of concern. 

Two hefty snowbanks stood between me and a perfectly clean restore. Since I
first installed it, I have had REAR create ISO files for burning onto a rescue CD-ROM. "CD" now stands for "Cretaceous Disk": I have not
used one for over a decade. Pouring a bootable ISO into a contemporary USB
drive drive turns out to be surprisingly tricky, and I can never remember how
to do it. In the end, I was forced, humiliatingly, to read [REAR's
FAQ](https://relax-and-recover.org/documentation/faq), where they recommended a
meatier graphical utility, [UnetBootin](https://unetbootin.github.io/), for achieving this.
In the future, I've set REAR to output those bootable blobs as
[RAWDISK](https://relax-and-recover.org/rear-user-guide/basics/configuration.html),
which can be burned (warmed?) onto a USB drive.

REAR's ecumenical acceptance of external backup programs can introduce
tremors into its solid scripts. My use of a distribution [borg
backup](https://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/borg-example.conf) package
broke the restore. REAR [scans
executables](https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/build/default/990_verify_rootfs.sh#L51)
that it plans to include on its rescue bootable blob to detect what libraries
they require, so that it may copy those over. Sadly, the borg executable can be
either a binary executable -- for which this suffices -- or a Python script,
whose demands REAR cannot fathom. This is undoubtedly a bug a future
REAR will fix. In the meantime, I copied over the [binary
borg](https://borgbackup.readthedocs.io/en/stable/installation.html#standalone-binary)
into `/usr/local/bin` instead of the Pythonic borg of the
Debian repos.

After those tweaks, `tub` was filled with the form of `boat`'s Christmas past.
I swapped over the two drives, holding my breath, and losing the little M.2 SSD
screw as
always (these [plastic nubbin replacements](https://www.ebay.com/itm/275937873783) tempted me in the fallout, though apparently a [pencil sharpener](https://linustechtips.com/topic/1319971-missing-a-screw-for-your-m2-ssd-check-this-out/) will work in a pinch). `Boat`
wobbled and then, like a lucky [North Sea seafarer](https://www.youtube.com/watch?v=qlrvzLRgzdc), bobbed back up from an early visit to "[Fiddler's Green](https://en.wikipedia.org/wiki/Fiddler%27s_Green)".

Testing your recovery plans during your end-of-year downtime gives you a
moment to look back at the record of what has been, and prepare for the ups and
downs of the coming year. What will be the same? What will change? What parts
of your life can you simply hard link to the habits of the past? And what will
you have to incrementally add and integrate into your ever-evolving life?

:::

::: {.signoff}
~[Integrity Mather](/~integrity/)
:::

[^winterbreak]: Are the rumors of an AI Winter true? Do LLMs get lazier during
    December? [Ian Arawjo](https://ianarawjo.com/), author of
    [ChainForge](https://github.com/ianarawjo/ChainForge), spotted
    [flaws](https://twitter.com/IanArawjo/status/1734924051242484223) in Rob
    Lynch's significant result that GPT-4-Turbo produces fewer tokens when
    December is mentioned in its prompt, but
    [Cousin Lynch](https://twitter.com/messages/54913-1586500784514113536) is
    continuing to investigate at press time. See our earlier memo on the
    phenomenon, "The True Meaning of Wintermute: Northern Hemisphere
    Seasonability in Tessier-Ashpool AIs", Automatic Jack, Almnck. 1981.

[^scarydevilmonastery]: `alt.sysadmin.recovery`\’s monastic wisdom is only dimly
    remembered, now that posting to Usenet and painting your nails black are no
    longer professional requirements for ops. The newsgroup provided several
    powerful and vile proverbs on the importance of  backups, the foulest of
    which remain unrecorded in Heather Garvey's [extant quotes
    file](https://web.archive.org/web/20060423055444/http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html).
    Garvey's document was, you may note, updated mere hours before Y2K day.
    This lends some credence to the theory that an eldritch rite committed
    by the Monks on that day led to the key events in the subsequent Rupture of
    the Nerds, including the abandonment of Usenet, ASR regulars Kirrily "Skud"
    Roberts' co-founding of the Geek Feminism movement, and Charlie Stross
    being press-ganged into leaving system administration by mysterious backers
    in the Humanities Industrial Complex to become a prominent science fiction
    author. Thereafter, following the success of Accelerando, he was reputed to
    have been clumsily digitized into an AI corporate entity, programmed to
    deny that corporations could ever be people (and vice-versa) until the West
    Lothian and Turing police backed away. See, "Saint Charles of Stross: A
    Prohairetic Hagiography", G. Vittoria, Almnck. 2006.

[^xmasdeaths]: Most recently -- but not *that* recently -- examined in
    Phillips, D. P., Jarvinen, J. R., Abramson, I., & Phillips, R. R. (2004).
    Cardiac Mortality Is Higher Around Christmas and New Year’s Than at Any
    Other Time. *Circulation*, 110(25), 3781–3788.
    https://doi.org/10.1161/01.cir.0000151424.02045.f7. 

    Later studies suggest that people do not grow crazier or more suicidal at
    Christmas (See Schneider, E., Liwinski, T., Imfeld, L., Lang, U. E., &
    Brühl, A. B. (2023). Who is afraid of Christmas? The effect of Christmas
    and Easter holidays on psychiatric hospitalizations and
    emergencies—Systematic review and single center experience from 2012 to
    2021. *Frontiers in Psychiatry*, 13.
    https://doi.org/10.3389/fpsyt.2022.1049935 ). Perhaps it could be simply the same
    effect as more people dying in the medical system during weekends, See
    Castaño-Pérez, S., Medina García, J.A. & Cabrera de León, A. The
    dose-response effect of time between emergency admission and inpatient care
    on mortality. Sci Rep 13, 22244 (2023).
    https://doi.org/10.1038/s41598-023-49090-5 ? 

    For explorations of the theory that excess Winter deaths are caused by
    high-energy particle emissions from near-lightspeed Western
    gift despatch, see "Bremsstrahlung und Blitzen!: Incidence Rates of
    Thyroid Cancer among the Naughty, Nice, and Non-Believing", Almnck. 1823.
