243 lines
14 KiB
HTML
243 lines
14 KiB
HTML
<!DOCTYPE html>
|
||
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="generator" content="pandoc" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
||
<meta name="author" content="Integrity Mather" />
|
||
<meta name="dcterms.date" content="2024-01-01" />
|
||
<title>Look Back Up</title>
|
||
<style>
|
||
code{white-space: pre-wrap;}
|
||
span.smallcaps{font-variant: small-caps;}
|
||
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
||
div.column{flex: auto; overflow-x: auto;}
|
||
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
||
/* The extra [class] is a hack that increases specificity enough to
|
||
override a similar rule in reveal.js */
|
||
ul.task-list[class]{list-style: none;}
|
||
ul.task-list li input[type="checkbox"] {
|
||
font-size: inherit;
|
||
width: 0.8em;
|
||
margin: 0 0.8em 0.2em -1.6em;
|
||
vertical-align: middle;
|
||
}
|
||
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
||
</style>
|
||
<link rel="stylesheet" href="res/almanack.css" />
|
||
<!--[if lt IE 9]>
|
||
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
||
<![endif]-->
|
||
</head>
|
||
<body>
|
||
<center><h1 class="frontispiece">Almanack</h1></center>
|
||
<header id="title-block-header">
|
||
<h1 class="title">Look Back Up</h1>
|
||
<p class="author">Integrity Mather</p>
|
||
<p class="date">January 1, 2024</p>
|
||
</header>
|
||
<div class="maintext">
|
||
<p><img src="res/new-years-backups-initial.png" class="initial"
|
||
alt="T" /> he year ends, the North pole tips its deepest bow to the
|
||
darkness, and we see even large language models have been <a
|
||
href="https://arstechnica.com/information-technology/2023/12/is-chatgpt-becoming-lazier-because-its-december-people-run-tests-to-find-out/">taking
|
||
it easy</a> for the winter<a href="#fn1" class="footnote-ref"
|
||
id="fnref1" role="doc-noteref"><sup>1</sup></a>. But for the faithful
|
||
maintainer of systems, there’s still work to be done, here in the
|
||
cooling embers of the year.</p>
|
||
<p>Now is a fine time to dust off your backup scripts and see if they’re
|
||
working as they should. An untested backup is no backup at all, said the
|
||
wise elders of the <a
|
||
href="http://www.faqs.org/faqs/sysadmin-recovery/">Scary Devil
|
||
Monastery</a><a href="#fn2" class="footnote-ref" id="fnref2"
|
||
role="doc-noteref"><sup>2</sup></a> and if we want to set the new year
|
||
off to its best start, we should ensure we can pause, tear down, and
|
||
re-start its march, even at its lowest points.</p>
|
||
<p>When examining my backup and restore process, I took the opportunity
|
||
this year to test my backups and increase the size of my root partition
|
||
on my server.</p>
|
||
<p>One last manual backup of the 256GiB SSD in my trusty server
|
||
<code>boat</code> for the sake of the old, and then an attempted restore
|
||
for the sake of the new: to a spare machine, <code>tub</code>,
|
||
temporarily hosting a new, 1TiB SSD. If all goes well, the restored
|
||
backup in tub would have new room to grow, and I could swap that drive
|
||
into <code>boat</code> with minimal downtime.</p>
|
||
<p>That moment of hardware-swapping would mean that <code>boat</code>
|
||
would have to be shut down and then restarted anew. Humans need their
|
||
respite over the holiday break: but should I have granted my server the
|
||
same indulgence? My plan accepted that <code>boat</code> would be
|
||
offline for, I hoped, a small slice of time.</p>
|
||
<p>There are more convoluted ways to ensure that none of my web sites,
|
||
file syncing, and miscellaneous tools flickered, even for a moment. I
|
||
could have temporarily switched my DNS settings to point at the fresh
|
||
clean <code>tub</code>, for instance, while overwriting
|
||
<code>boat</code>. Or perhaps just repurposed <code>boat</code> for
|
||
gentler, less demanding tasks, giving it the end-of-year gift of a
|
||
well-deserved retirement, and passing to <code>tub</code> a new year’s
|
||
responsibility of hosting my server processes.</p>
|
||
<p>Within the scale of my own life, I do believe that uptime is
|
||
overrated. We are surrounded by tools bent into the shapes demanded by
|
||
large tech companies, for whom site reliability is their first
|
||
commandment.</p>
|
||
<p>But we are humans, not corporations. We run at our own pace. If Homer
|
||
nods, so can my home lab. Christmas has its folklore of <a
|
||
href="https://en.wikipedia.org/wiki/Befana">visitors rebuffed</a>, but
|
||
hopefully, my friends have other pressing matters this time of year than
|
||
hitting reload on my sites.</p>
|
||
<p>Yes, a personal web server can go down for a few moments – as long as
|
||
it bobs back up.</p>
|
||
<p>Which, with restored filing systems, is ever the question. Will the
|
||
backup truly come back up? A reset and reboot may also be a time for
|
||
sinking exhaustion and death. Human deaths seem positively correlated
|
||
with the change of pace of the holidays<a href="#fn3"
|
||
class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>,
|
||
and so too, drives and computers will dutifully run for years, but
|
||
expire after a moment’s rest.</p>
|
||
<center>
|
||
<figure>
|
||
<img src="res/new-years-backups-dingbat1.png" style="width:50.0%"
|
||
alt="~~~" />
|
||
<figcaption aria-hidden="true">~~~</figcaption>
|
||
</figure>
|
||
</center>
|
||
<p>I use <a
|
||
href="https://relax-and-recover.org/">“Relax-And-Recover”</a> (REAR), a
|
||
Linux disaster recovery system from when sysadmins wrote shell code and
|
||
<em>liked it</em>. REAR is a sprawling set of shell scripts that runs
|
||
your choice of backup code – from <a
|
||
href="https://jumpcloud.com/blog/how-to-backup-linux-system-rsync"><code>rsync</code></a>
|
||
to <a href="https://borgbackup.readthedocs.io/"><code>borg</code></a> –
|
||
over your entire Linux filesystem, on a regular, cron-determined, basis.
|
||
Having seen to that prosaic task, REAR will also create a <a
|
||
href="https://relax-and-recover.org/usage/#recovery_from_usb">minimal,
|
||
bootable blob</a>. The blob, stuffed onto a USB drive, CD-ROM, or
|
||
networked drive, will boot into a rescue Linux, and lead you through the
|
||
reconstruction and re-partitioning of a drive that will emerge the same
|
||
shape as your original machine. Then it will pull down your backups, and
|
||
restore this drive to the precise state that your backups recalled
|
||
it.</p>
|
||
<p>Readers born into our age of strong types and weak stomachs may be
|
||
balking at the idea of entrusting their restoration to a bunch of <a
|
||
href="https://samgrayson.me/essays/stop-writing-shell-scripts/">stringly-typed</a>
|
||
Bash scripts. REAR’s 20K(!) lines of shellcode intimate that it has
|
||
reached levels of complexity, beyond that which bashism can safely
|
||
grasp.</p>
|
||
<p>But this is <em>sysadmin</em> shell code. That terrifying KLOC is
|
||
defensive, modular coding of the highest order. For the casual shell
|
||
user, REAR’s operation is comprehensible, its failure modes anticipated
|
||
and loudly announced, and tweaks and mediations are semi-obvious. REAR’s
|
||
authors offer “a relaxing recovery”, and the ghosts of sysadminning past
|
||
do not lightly emphasize relaxation.</p>
|
||
<p>While I was never <em>not</em> relaxed during my holiday restoration.
|
||
I did occasionally carol a high-pitched note or two of concern.</p>
|
||
<p>Two hefty snowbanks stood between me and a perfectly clean restore.
|
||
Since I first installed it, I have had REAR create ISO files for burning
|
||
onto a CD-ROM that I could restore from. “CD’ stand now for”Cretaceous
|
||
Disk”: I have not used one for over a decade. Pouring a bootable ISO
|
||
into a contemporary USB drive drive turns out to be surprisingly tricky,
|
||
and I can never remember how to do it. In the end, I was forced,
|
||
humiliatingly, to read <a
|
||
href="https://relax-and-recover.org/documentation/faq">REAR’s FAQ</a>,
|
||
where they recommended a meatier graphical utility, <a
|
||
href="https://unetbootin.github.io/">UnetBootin</a>, for achieving this.
|
||
In the future, I’ve set REAR to output those bootable blobs as <a
|
||
href="https://relax-and-recover.org/rear-user-guide/basics/configuration.html">RAWDISK</a>,
|
||
which can be burned (warmed?) onto a USB drive.</p>
|
||
<p>REAR’s ecumenical acceptance of external backup programs can
|
||
introduce a tremors into its solid scripts. My use of a distribution <a
|
||
href="https://github.com/rear/rear/blob/master/usr/share/rear/conf/examples/borg-example.conf">borg
|
||
backup</a> package broke the restore. REAR <a
|
||
href="https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/build/default/990_verify_rootfs.sh#L51">scans
|
||
executables</a> that it plans to include on its rescue bootable blob to
|
||
detect what libraries they require, so that it may copy those over.
|
||
Sadly, the borg executable can be either a binary executable – for which
|
||
this suffices – or a Python script, whose demands REAR cannot fathom.
|
||
This is undoubtedly a bug a future REAR will fix. In the meantime I
|
||
copied over the <a
|
||
href="https://borgbackup.readthedocs.io/en/stable/installation.html#standalone-binary">binary
|
||
borg</a> into <code>/usr/local/bin</code> instead of the Pythonic borg
|
||
of the Debian repos.</p>
|
||
<p>After those tweaks, <code>tub</code> was filled with the form of
|
||
<code>boat</code>’s Christmas past. I swapped over the two drives,
|
||
holding my breath, and losing the little M.2 SSD screw as always (these
|
||
<a href="https://www.ebay.com/itm/275937873783">plastic nubbin
|
||
replacements</a> tempted me in the fallout, though apparently a <a
|
||
href="https://linustechtips.com/topic/1319971-missing-a-screw-for-your-m2-ssd-check-this-out/">pencil
|
||
sharpener</a> will work in a pinch). <code>Boat</code> wobbled and then,
|
||
like a lucky <a href="https://www.youtube.com/watch?v=qlrvzLRgzdc">North
|
||
Sea seafarer</a>, bobbed back up from an early visit to “<a
|
||
href="https://en.wikipedia.org/wiki/Fiddler%27s_Green">Fiddler’s
|
||
Green</a>”.</p>
|
||
<p>Testing your recovery plans during your own end-of-year downtime
|
||
gives you a moment to look back at the record of what has been, and
|
||
prepare for the ups and downs of the coming year. What will be the same?
|
||
What will change? What parts of your life can you simply hard link to
|
||
the habits of the past? And what will you have to incrementally add and
|
||
integrate into your ever-evolving life?</p>
|
||
</div>
|
||
<div class="signoff">
|
||
<p><a href="/~integrity/">~Integrity Mather</a></p>
|
||
</div>
|
||
<section id="footnotes" class="footnotes footnotes-end-of-document"
|
||
role="doc-endnotes">
|
||
<hr />
|
||
<ol>
|
||
<li id="fn1"><p>Are the rumors of an AI Winter true? Do LLMs get lazier
|
||
during December? <a href="https://ianarawjo.com/">Ian Arawjo</a>, author
|
||
of <a href="https://github.com/ianarawjo/ChainForge">ChainForge</a>,
|
||
spotted <a
|
||
href="https://twitter.com/IanArawjo/status/1734924051242484223">flaws</a>
|
||
in Rob Lynch’s significant result that GPT-4-Turbo produces fewer tokens
|
||
when December is mentioned in its prompt, but <a
|
||
href="https://twitter.com/messages/54913-1586500784514113536">Cousin
|
||
Lynch</a> is continuing to investigate at press time. See our earlier
|
||
memo on the phenomenon, “The True Meaning of Wintermute: Northern
|
||
Hemisphere Seasonability in Tessier-Ashpool AIs”, Automatic Jack,
|
||
Almnck. 1981.<a href="#fnref1" class="footnote-back"
|
||
role="doc-backlink">↩︎</a></p></li>
|
||
<li id="fn2"><p><code>alt.sysadmin.recovery</code>‘s monastic wisdom, is
|
||
only dimly remembered now that posting to Usenet and painting your nails
|
||
black are no longer professional requirements for ops. The newsgroup
|
||
provided several powerful and vile proverbs on the importance of
|
||
backups, the foulest of which remain unrecorded in Heather Garvey’s <a
|
||
href="https://web.archive.org/web/20060423055444/http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html">extant
|
||
quotes file</a>. Garvey’s document was, you may note, updated mere hours
|
||
before Y2K day. This lends some credence to the theory that an eldritch
|
||
rite committed by the Monks on that day led to the key events in the
|
||
subsequent Rupture of the Nerds, including the abandonment of Usenet,
|
||
ASR regulars Kirrily “Skud” Roberts’ co-founding of the Geek Feminism
|
||
movement, and Charlie Stross being press-ganged into leaving system
|
||
administration by mysterious backers in the Humanities Industrial
|
||
Complex to become a prominent science fiction author. Thereafter,
|
||
following the success of Accelerando, he was reputed to have been
|
||
clumsily digitized into an AI corporate entity, programmed to deny that
|
||
corporations could ever be people (and vice-versa) until the West
|
||
Lothian and Turing police backed away. See, “Saint Charles of Stross: A
|
||
Prohairetic Hagiography”, G. Vittoria, Almnck. 2006.<a href="#fnref2"
|
||
class="footnote-back" role="doc-backlink">↩︎</a></p></li>
|
||
<li id="fn3"><p>Most recently – but not <em>that</em> recently –
|
||
examined in Phillips, D. P., Jarvinen, J. R., Abramson, I., &
|
||
Phillips, R. R. (2004). Cardiac Mortality Is Higher Around Christmas and
|
||
New Year’s Than at Any Other Time. <em>Circulation</em>, 110(25),
|
||
3781–3788. https://doi.org/10.1161/01.cir.0000151424.02045.f7.</p>
|
||
<p>Later studies suggest that people do not grow crazier or more
|
||
suicidal at Christmas (See Schneider, E., Liwinski, T., Imfeld, L.,
|
||
Lang, U. E., & Brühl, A. B. (2023). Who is afraid of Christmas? The
|
||
effect of Christmas and Easter holidays on psychiatric hospitalizations
|
||
and emergencies—Systematic review and single center experience from 2012
|
||
to 2021. <em>Frontiers in Psychiatry</em>, 13.
|
||
https://doi.org/10.3389/fpsyt.2022.1049935 ). Perhaps it could be simply
|
||
the same effect as more people dying in the medical system during
|
||
weekends, See Castaño-Pérez, S., Medina García, J.A. & Cabrera de
|
||
León, A. The dose-response effect of time between emergency admission
|
||
and inpatient care on mortality. Sci Rep 13, 22244 (2023).
|
||
https://doi.org/10.1038/s41598-023-49090-5 ?</p>
|
||
<p>For explorations of the theory that excess Winter deaths are caused
|
||
by high-energy particle emissions from near-lightspeed Western gift
|
||
despatch, see “Bremßtrahlung und Blitzen!: Incidence Rates of Thyroid
|
||
Cancer among the Naughty, Nice, and Non-Believing”, Almnck. 1823.<a
|
||
href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
|
||
</ol>
|
||
</section>
|
||
</body>
|
||
</html>
|