almnck/assets/editorial/09final/integrity-new-years-backups/SPELL.html

<p>To Old Danny, greetings:</p>
<p>The year ends, the North pole tips its deepest bow to the darkness,
and we see even large language models have been taking it easy for the
winter<a href="#fn1" class="footnote-ref" id="fnref1"
role="doc-noteref"><sup>1</sup></a>. But for the faithful maintainer of
systems, there’s still work to be done, here in the cooling embers of
the year.</p>
<p>Now is a fine time to dust off your backup scripts and see if they’re
working as they should. An untested backup is no backup at all, said the
wise elders of the <a
href="http://www.faqs.org/faqs/sysadmin-recovery/">Scary Devil
Monastery</a><a href="#fn2" class="footnote-ref" id="fnref2"
role="doc-noteref"><sup>2</sup></a>, and if we want to set the new year
off to its best start, we should ensure we can pause, tear-down, and
re-start the marching present even at its lowest points.</p>
<p>When examining my backup and restore process, I took the opportunity
this year to test my backups and increase the size of my root partition
on my server.</p>
<p>One last manual backup of the 256GiB SSD in my trusty server
<code>boat</code> for the sake of the old, and then an attempted restore
for the sake of the new: to a spare machine, <code>tub</code>,
temporarily hosting a new, 1TiB SSD. If all goes well, the restored
backup in tub would have new room to grow, and I could swap that drive
into <code>boat</code> with minimal downtime.</p>
<p>That moment of hardware-swapping would mean that <code>boat</code>
would have to be shutdown and then restarted anew. Humans need their
respite over the holiday break: but should I have granted my server the
same indulgence? My plan accepted that <code>boat</code> would be
offline for, I hoped, a small slice of time.</p>
<p>There are more convoluted ways to ensure that none of my websites,
file syncing, and miscellaneous tools did not flicker, even for a
moment. I could have switched my DNS settings to the fresh clean
<code>tub</code>, for instance, while overwriting <code>boat</code>. Or
perhaps just repurposed <code>boat</code> for gentler, less demanding
tasks, giving it the end-of-year gift of a well-deserved retirement, and
switched to <code>tub</code> a new year’s responsibility of my hosting
my main home processes.</p>
<p>At the scale of my own life, I do believe that uptime is overrated.
We are surrounded by tools bent into the shapes demanded by large tech
companies, for whom site reliability is their first commandment.</p>
<p>But we are humans, not corporations. We run at our own pace. If Homer
nods, so can my home lab. Christmas has its own folklore of <a
href="https://en.wikipedia.org/wiki/Befana">visitors rebuffed</a>, but
hopefully my friends have other things to do at this time than hit
reload on my websites.</p>
<p>Yes, a personal webserver can go down for a few moments – as long as
it bobs back up.</p>
<p>Which, with restored filing systems, is ever the question. Will the
backup truly come back up? A reset and rebooted may also be a time for
final exhaustion and death. Morbidly, we note that human deaths seem
positively correlated with the change of pace of the holidays<a
href="#fn3" class="footnote-ref" id="fnref3"
role="doc-noteref"><sup>3</sup></a>, just as drives and computers will
dutifully run for years, but expire after a moment’s rest.</p>
<p>I use <a
href="https://relax-and-recover.org/">“Relax-And-Recover”</a> (REAR), a
Linux disaster recovery system from when sysadmins wrote shell code and
/liked it/. REAR is a sprawling set of shell scripts that runs your
choice of backup code – from
<code>[rsync](https://jumpcloud.com/blog/how-to-backup-linux-system-rsync)</code>
to <code>[borg](https://borgbackup.readthedocs.io/)</code> – over your
entire linux root, on a regular, cron-determined, basis. Having seen to
that prosaic task, it will also create <a
href="https://relax-and-recover.org/usage/#recovery_from_usb">minimal,
bootable blob</a>. The blob, stuffed onto an USB drive, CD-ROM, or
networked drive, will boot into a minimal Linux, and lead you through
the reconstruction and re-partitioning of a drive that will emerge the
same shape as your original machine. Then it will pull down your
backups, and restore this drive to the precise state that your backups
recalled it. A perfect, royal, restoration.</p>
<p>Readers born into our age of strong types and weak stomachs may be
balking at the idea of entrusting their restoration to a bunch of <a
href="https://samgrayson.me/essays/stop-writing-shell-scripts/">stringly-typed</a>
Bash scripts. REAR’s 20K(!) lines of shellcode intimate that it has
reached the edges of complexity beyond that you might think bashism can
bear.</p>
<p>But this is <em>sysadmin</em> shell code. That terrifying KLOC is
defensive, modular coding of the highest order. For the casual shell
user, REAR’s operation is comprehensible, its failure modes anticipated
and loudly-announced, and tweaks and mediations are semi-obvious. REAR’s
authors emphasise “a relaxing recovery”, and ghosts of sysadminning past
do not lightly emphasise relaxation.</p>
<p>While I was never /not/ relaxed during my holiday restoration. I did
occasionally carol a high-pitched note or two of concern.</p>
<p>Two snowbanks stood between me and a perfectly clean restore. Since I
first installed it, I have had REAR create ISO files for burning onto a
CD-ROM that I could restore from. CD must now stand for “Cretaceous
Disk”: I have not used those in anger for over a decade. Converting a
bootable ISO into a bootable USB drive drive turns out to be
surprisingly tricky, and I can never remember how to do it. In the end I
was forced, humiliatingly, to read <a
href="https://relax-and-recover.org/documentation/faq">REAR’s FAQ</a>,
where they recommended a heavy utility, <a
href="https://unetbootin.github.io/">UnetBootin</a>, for achieving this.
In the future, I’ve set REAR to output those bootable blobs as <a
href="https://relax-and-recover.org/rear-user-guide/basics/configuration.html">RAWDISK</a>,
which can be burned (warmed?) onto a USB.</p>
<p>REAR’s ecumenical acceptance of multiple backup programs can
introduce additional complexity into its scripts. My use of <a
href="https://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/borg-example.conf">borg
backup</a> tripped up the restore. REAR <a
href="https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/build/default/990_verify_rootfs.sh#L51">scans
executables</a> that it plans to include on its rescue bootable blob to
detect what libraries they require, so that it may copy those over.
Sadly, the borg executable can be either a binary executable (for which
this works), or just a Python script, whose demands REAR cannot fathom.
This is undoubtedly a bug a future REAR will fix, but in the meantime I
just copied over the <a
href="https://borgbackup.readthedocs.io/en/stable/installation.html#standalone-binary">binary
borg</a> into <code>/usr/local/bin</code> and used that instead of the
Pythonic borg of the Debian repos.</p>
<p>After those tweaks, <code>tub</code> was filled with the form of
<code>boat</code>’s Christmas past. I swapped over the two drives,
holding my breath, and losing the little M.2 SSD screw as always (these
<a href="https://www.ebay.com/itm/275937873783">plastic nubbin
replacements</a> tempted me in the fallout, though apparently a <a
href="https://linustechtips.com/topic/1319971-missing-a-screw-for-your-m2-ssd-check-this-out/">pencil
sharpener</a> will work in a pinch). <code>Boat</code> wobbled and then,
like a lucky <a href="https://www.youtube.com/watch?v=qlrvzLRgzdc">North
Sea seafarer</a>, bobbed back up from an early visit to “<a
href="https://en.wikipedia.org/wiki/Fiddler%27s_Green">Fiddler’s
Green</a>”.</p>
<p>Testing your recovery plans during your own end-of-year downtime
gives you a moment to look back at the record of what has been, and
prepare for the ups and downs of the coming year. What will be the same?
What will change? What parts of your life can you simply hard link to
the habits of the past? And what will you have to incrementally add and
integrate into your ever-evolving life?</p>
<p>Until next time, I am relaxed and restored,</p>
<p>~Integrity Mather</p>
<section id="footnotes" class="footnotes footnotes-end-of-document"
role="doc-endnotes">
<hr />
<ol>
<li id="fn1"><p>Are the rumors of an AI Winter true? Do LLMs get lazier
during December? <a href="https://ianarawjo.com/">Ian Arawjo</a>, author
of <a href="https://github.com/ianarawjo/ChainForge">ChainForge</a>,
spotted <a
href="https://twitter.com/IanArawjo/status/1734924051242484223">flaws</a>
in Rob Lynch’s significant result that GPT-4-Turbo produces fewer tokens
when December is mentioned in its prompt, but <a
href="Cousin%20Lynch">https://twitter.com/messages/54913-1586500784514113536</a>
is continuing to investigate at press time. See our earlier memo on the
phenomenon, “The True Meaning of Wintermute: Northern Hemisphere
Seasonability in Tessier-Ashpool AIs”, Automatic Jack, Almnck. 1981.<a
href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn2"><p>Alt.sysadmin.recovery’s monastic wisdom, is only dimly
remembered now that posting to Usenet and painting your nails black are
no longer professional requirements for ops. The newsgroup provided
several powerful and vile proverbs on the importance of backups, the
foulest of which remain unrecorded in Heather Garvey’s <a
href="https://web.archive.org/web/20060423055444/http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html">extant
quotes file</a>. Garvey’s document was, you may note, updated mere hours
before Y2K day. This lends some credence to the the theory that an
eldritch rite committed by the Monks on that day led to the key events
in the subsequent Rupture of the Nerds, including the abandonment of
Usenet, ASR regulars Kirrily “Skud” Roberts’ co-founding of the Geek
Feminism movement, and Charlie Stross being press-ganged into leaving
system administraiton by mysterious backers in the Humanities Industrial
Complex to become a prominent science fiction author. Thereafter,
following the success of Accelerando, he was reputed to have been
clumsily digitized into an AI corporate entity, programmed to deny that
corporations could ever be people (and vice-versa) until the West
Lothian and Turing police backed away. See, “Saint Charles of Stross: A
Prohairetic Hagiography”, G. Vittoria, Almnck. 2006.<a href="#fnref2"
class="footnote-back" role="doc-backlink">↩︎</a></p></li>
<li id="fn3"><p>Most recently – but not <em>that</em> recently –
examined in Phillips, D. P., Jarvinen, J. R., Abramson, I., &amp;
Phillips, R. R. (2004). Cardiac Mortality Is Higher Around Christmas and
New Year’s Than at Any Other Time. <em>Circulation</em>, 110(25),
3781–3788. https://doi.org/10.1161/01.cir.0000151424.02045.f7.</p>
<p>Later studies suggest that people don’t get any crazier or suicidal
at Christmas (See Schneider, E., Liwinski, T., Imfeld, L., Lang, U. E.,
&amp; Brühl, A. B. (2023). Who is afraid of Christmas? The effect of
Christmas and Easter holidays on psychiatric hospitalizations and
emergencies—Systematic review and single center experience from 2012 to
2021. <em>Frontiers in Psychiatry</em>, 13.
https://doi.org/10.3389/fpsyt.2022.1049935 ), and it may just be the
same effect as more people dying in the medical system during weekends,
See Castaño-Pérez, S., Medina García, J.A. &amp; Cabrera de León, A. The
dose–response effect of time between emergency admission and inpatient
care on mortality. Sci Rep 13, 22244 (2023).
https://doi.org/10.1038/s41598-023-49090-5 .</p>
<p>For explorations of the theory that excess Winter deaths are caused
by high-energy particle emissions from near-lightspeed Western
gift-deliverers, see “Bremstrahlung und Blitzen!: Incidence Rates of
Thyroid Cancer among Naughty, Nice, and Non-Believers”, Almnck. 1823.<a
href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>