New article: Look Back Up
This commit is contained in:
Danny O'Brien
parent
26473aa3b1
commit
f3128131c4
2 changed files with 257 additions and 0 deletions
|
|
@ -0,0 +1,143 @@
|
||||||
|
|
||||||
|
---
|
||||||
|
Author: Integrity Jones
|
||||||
|
Title: Looking Back Up
|
||||||
|
Published: 2024-01-01
|
||||||
|
---
|
||||||
|
|
||||||
|
To Old Danny, greetings:
|
||||||
|
|
||||||
|
The solar nights have barely stopped closing in, and we see even large language
|
||||||
|
models have been taking it easy for the winter. But for the faithful maintainer
|
||||||
|
of systems, there's still work to be done, here in the cooling embers of the year.
|
||||||
|
|
||||||
|
> Solar nights -- does this mean anything
|
||||||
|
> Link to LLM stories
|
||||||
|
|
||||||
|
Now is a fine time of year to dust off your backup scripts, and see if they're
|
||||||
|
working as they should. An untested backup is a no backup at all, said the wise
|
||||||
|
ghosts of the Scary Devil Monastery, and if we want to set the year off to its
|
||||||
|
best start, we should ensure we can pause, tear-down, and re-start it at its lowest
|
||||||
|
points.
|
||||||
|
|
||||||
|
> Link to Scary Devil Monastery
|
||||||
|
|
||||||
|
When examining my backup and restore process, took the opportunity this year to
|
||||||
|
quadruple the size of my root partition on my server.
|
||||||
|
|
||||||
|
One last manual backup of the 256GiB SSD in my trusty server `boat` for the
|
||||||
|
sake of the old, and then an attempted restore for the sake of the new: to a
|
||||||
|
spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well,
|
||||||
|
the restored backup in tub would have new room to grow, and I could swap
|
||||||
|
out that drive back into `boat` with minimal downtime.
|
||||||
|
|
||||||
|
> Box on hard drives? Link to goods?
|
||||||
|
|
||||||
|
That moment of hardware-swapping would mean that `boat` would have to be
|
||||||
|
shutdown and then restarted anew. Humans need their respite over the holiday break:
|
||||||
|
but should I have permitted my server the same indulgence? My plan accepted
|
||||||
|
that boat would be offline for, one hopes, a small slice of time.
|
||||||
|
There are more convoluted ways to ensure that none of my websites, file syncing, and
|
||||||
|
miscellaneous tools did not flicker, even for a moment.
|
||||||
|
|
||||||
|
I could have switched my DNS settings to the fresh clean `tub`, for instance,
|
||||||
|
while overwriting `boat`. Or perhaps just repurposed boat for gentler, less
|
||||||
|
demanding tasks, giving it the end-of-year gift of a well-deserved retirement,
|
||||||
|
and switched to `tub` a new year's responsibility of my hosting my main home
|
||||||
|
processes.
|
||||||
|
|
||||||
|
At the smallest scale, I do believe that uptime is overrated. We are surrounded
|
||||||
|
by tools bent into the shapes demanded by large tech companies, for whom site
|
||||||
|
reliability is their first commandment.
|
||||||
|
|
||||||
|
But we are humans, not corporations. If even Homer nods, so can my home lab.
|
||||||
|
Christmas has its own folklore of visitors rebuffed, but hopefully my friends
|
||||||
|
have better things to do at this time than hit reload on my websites. Yes, a
|
||||||
|
personal webserver can go down for a few moments -- as long as it comes back up.
|
||||||
|
|
||||||
|
> Link to Home nods line
|
||||||
|
|
||||||
|
Which, with restored filing systems, is ever the question. Will the backup
|
||||||
|
truly come back up? A restored and rebooted machine may also be a time for
|
||||||
|
failure and death, just as, morbidly, we should note that human coronary deaths
|
||||||
|
seem positively correlated with the change of pace of the holidays.
|
||||||
|
|
||||||
|
|
||||||
|
> Link to coronary death paper
|
||||||
|
|
||||||
|
I use the more positively-framed "Relax-And-Recover" (REAR), a recover system
|
||||||
|
from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash
|
||||||
|
scripts that runs your choice of backup code -- from rsync to borg-- over your
|
||||||
|
entire linux root, on a regular, cron-determined, basis. Having seen to that
|
||||||
|
prosaic task, it will also create minimal, bootable blob. The blob, stuffed
|
||||||
|
onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux,
|
||||||
|
and gently lead you through reconstructing a re-partitioning of a drive
|
||||||
|
that will. At its end, the partition map on the machine you have booted
|
||||||
|
will be exactly the same shape as your original machine. Then it will pull
|
||||||
|
down your precious backups, and restore this drive to the precise state
|
||||||
|
that your backups recalled it. A perfect, royal, restoration.
|
||||||
|
|
||||||
|
> Link to Relax and Recover
|
||||||
|
|
||||||
|
Younger readers born into our age of strong types and weak stomachs, may be
|
||||||
|
balking at the idea of entrusting restoration to a bunch of stringly-typed Bash
|
||||||
|
scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the
|
||||||
|
edges of complexity beyond that you might think bashism can bear.
|
||||||
|
|
||||||
|
> Link to definition of stringly
|
||||||
|
> Link to proof of REAR's 20K of KLOCs
|
||||||
|
|
||||||
|
But this is /sysadmin/ shell code. That terrifying KLOC is
|
||||||
|
defensive, modular coding of the highest order. For the casual shell user,
|
||||||
|
REAR's operation is comprehensible, the failure modes anticipated, and
|
||||||
|
tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing
|
||||||
|
recovery", and ghosts of sysadminning past do not emphasise enhancing your
|
||||||
|
calm in those moments lightly.
|
||||||
|
|
||||||
|
Link to relaxing recovery
|
||||||
|
|
||||||
|
I was never /not/ relaxed during my holiday restoration, but I did
|
||||||
|
occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO
|
||||||
|
files for burning onto a CD-ROM that I could restore from. CD stands for
|
||||||
|
"Cretaceous Disk". Even I have not used one of them in anger in over a decade.
|
||||||
|
Converting a bootable ISO into a bootable USB drive drive turned out to be
|
||||||
|
surprisingly tricky, and I can never remember how to do it. In the end I was
|
||||||
|
forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy
|
||||||
|
utility, https://unetbootin.github.io/, for achieving this. In the future, I've
|
||||||
|
set REAR to output those bootable blobs as RAWDISK, which can be burned
|
||||||
|
(warmed?) onto a USB.
|
||||||
|
|
||||||
|
> Link to REAR's FAQ
|
||||||
|
> Link to UnetBootlin
|
||||||
|
|
||||||
|
REAR's ecumenical acceptance of multiple backup programs introduces additional
|
||||||
|
complexity into its scripts. My use of borg backup tripped up the restore. REAR
|
||||||
|
scans executables that it plans to include on its rescue bootable blob, so it
|
||||||
|
can detect what libraries they need and copy those over. Sadly, the borg
|
||||||
|
executable can be either a binary executable (for which this works), or just a
|
||||||
|
Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a
|
||||||
|
future REAR will fix, but in the meantime I just copied over the binary borg
|
||||||
|
into /usr/local/bin and used that instead of the Pythonic borg of the Debian
|
||||||
|
repos.
|
||||||
|
|
||||||
|
> Link to binary executable
|
||||||
|
> Link to executable/python script issue on github
|
||||||
|
|
||||||
|
After those tweaks, tub was filled with the form of boat's Christmas past. I swapped
|
||||||
|
over the two drives, holding my breath, and losing the little SSD screw as
|
||||||
|
always (these plastic nubbin replacements tempted me in the fallout). Boat
|
||||||
|
wobbled and then, like a plucky North Sea veteran, bobbed back up.
|
||||||
|
|
||||||
|
> Link to nubbin ali express sale
|
||||||
|
> Link to North Sea tiktok music
|
||||||
|
|
||||||
|
Testing your recovery plans during your own end-of-year downtime and recovery
|
||||||
|
gives you a moment to peer at what has been, and prepare for the ups and downs
|
||||||
|
of the coming year. What will be the same? What will change? What parts of your
|
||||||
|
life can you simply hard link to the habits of the past? And what will you have
|
||||||
|
to incrementally add and integrate into your ever-evolving life?
|
||||||
|
|
||||||
|
Until next time, I am,
|
||||||
|
|
||||||
|
~Integrity Mather
|
||||||
|
|
||||||
|
|
@ -0,0 +1,114 @@
|
||||||
|
---
|
||||||
|
Author: Integrity Jones
|
||||||
|
Title: Looking Back Up
|
||||||
|
Published: 2024-01-01
|
||||||
|
---
|
||||||
|
|
||||||
|
To Old Danny, greetings:
|
||||||
|
|
||||||
|
The solar nights have barely stopped closing in, and we see even large language
|
||||||
|
models have been taking it easy for the winter. But for the faithful maintainer
|
||||||
|
of systems, there's still work to be done, here in the cooling embers of the year.
|
||||||
|
|
||||||
|
Now is a fine time of year to dust off your backup scripts, and see if they're
|
||||||
|
working as they should. An untested backup is no backup at all, said the wise
|
||||||
|
ghosts of the Scary Devil Monastery, and if we want to set the year off to its
|
||||||
|
best start, we should ensure we can pause, tear-down, and re-start it even at its lowest
|
||||||
|
points.
|
||||||
|
|
||||||
|
When examining my backup and restore process, took the opportunity this year to
|
||||||
|
quadruple the size of my root partition on my server.
|
||||||
|
|
||||||
|
One last manual backup of the 256GiB SSD in my trusty server `boat` for the
|
||||||
|
sake of the old, and then an attempted restore for the sake of the new: to a
|
||||||
|
spare machine, `tub`, temporarily hosting a new, 1TiB SSD. If all goes well,
|
||||||
|
the restored backup in tub would have new room to grow, and I could swap
|
||||||
|
out that drive back into `boat` with minimal downtime.
|
||||||
|
|
||||||
|
That moment of hardware-swapping would mean that `boat` would have to be
|
||||||
|
shutdown and then restarted anew. Humans need their respite over the holiday break:
|
||||||
|
but should I have permitted my server the same indulgence? My plan accepted
|
||||||
|
that boat would be offline for, one hopes, a small slice of time.
|
||||||
|
There are more convoluted ways to ensure that none of my websites, file syncing, and
|
||||||
|
miscellaneous tools did not flicker, even for a moment.
|
||||||
|
|
||||||
|
I could have switched my DNS settings to the fresh clean `tub`, for instance,
|
||||||
|
while overwriting `boat`. Or perhaps just repurposed boat for gentler, less
|
||||||
|
demanding tasks, giving it the end-of-year gift of a well-deserved retirement,
|
||||||
|
and switched to `tub` a new year's responsibility of my hosting my main home
|
||||||
|
processes.
|
||||||
|
|
||||||
|
At the smallest scale, I do believe that uptime is overrated. We are surrounded
|
||||||
|
by tools bent into the shapes demanded by large tech companies, for whom site
|
||||||
|
reliability is their first commandment.
|
||||||
|
|
||||||
|
But we are humans, not corporations. If even Homer nods, so can my home lab.
|
||||||
|
Christmas has its own folklore of visitors rebuffed, but hopefully my friends
|
||||||
|
have better things to do at this time than hit reload on my websites. Yes, a
|
||||||
|
personal webserver can go down for a few moments -- as long as it comes back up.
|
||||||
|
|
||||||
|
Which, with restored filing systems, is ever the question. Will the backup
|
||||||
|
truly come back up? A restored and rebooted machine may also be a time for
|
||||||
|
failure and death, just as, morbidly, we should note that human coronary deaths
|
||||||
|
seem positively correlated with the change of pace of the holidays.
|
||||||
|
|
||||||
|
I use the more positively-framed "Relax-And-Recover" (REAR), a recover system
|
||||||
|
from when sysadmins wrote shell code and /liked it/. REAR is a sprawling bash
|
||||||
|
scripts that runs your choice of backup code -- from rsync to borg-- over your
|
||||||
|
entire linux root, on a regular, cron-determined, basis. Having seen to that
|
||||||
|
prosaic task, it will also create minimal, bootable blob. The blob, stuffed
|
||||||
|
onto an USB drive, CD ROM, or networked drive, will boot into a minimal Linux,
|
||||||
|
and gently lead you through reconstructing a re-partitioning of a drive
|
||||||
|
that will. At its end, the partition map on the machine you have booted
|
||||||
|
will be exactly the same shape as your original machine. Then it will pull
|
||||||
|
down your precious backups, and restore this drive to the precise state
|
||||||
|
that your backups recalled it. A perfect, royal, restoration.
|
||||||
|
|
||||||
|
Younger readers born into our age of strong types and weak stomachs, may be
|
||||||
|
balking at the idea of entrusting restoration to a bunch of stringly-typed Bash
|
||||||
|
scripts. REAR's 20K(!) lines of shellcode intimate that it has reached the
|
||||||
|
edges of complexity beyond that you might think bashism can bear.
|
||||||
|
|
||||||
|
But this is /sysadmin/ shell code. That terrifying KLOC is
|
||||||
|
defensive, modular coding of the highest order. For the casual shell user,
|
||||||
|
REAR's operation is comprehensible, the failure modes anticipated, and
|
||||||
|
tweaks and errors are semi-obvious. REAR's authors emphasise "a relaxing
|
||||||
|
recovery", and ghosts of sysadminning past do not emphasise enhancing your
|
||||||
|
calm in those moments lightly.
|
||||||
|
|
||||||
|
I was never /not/ relaxed during my holiday restoration, but I did
|
||||||
|
occasionally emit a high-pitched carol or two of concern. Firstly, I had REAR create ISO
|
||||||
|
files for burning onto a CD-ROM that I could restore from. CD stands for
|
||||||
|
"Cretaceous Disk". Even I have not used one of them in anger in over a decade.
|
||||||
|
Converting a bootable ISO into a bootable USB drive drive turned out to be
|
||||||
|
surprisingly tricky, and I can never remember how to do it. In the end I was
|
||||||
|
forced, humiliatingly, to read REAR's FAQ, where they recommended a heavy
|
||||||
|
utility, https://unetbootin.github.io/, for achieving this. In the future, I've
|
||||||
|
set REAR to output those bootable blobs as RAWDISK, which can be burned
|
||||||
|
(warmed?) onto a USB.
|
||||||
|
|
||||||
|
REAR's ecumenical acceptance of multiple backup programs introduces additional
|
||||||
|
complexity into its scripts. My use of borg backup tripped up the restore. REAR
|
||||||
|
scans executables that it plans to include on its rescue bootable blob, so it
|
||||||
|
can detect what libraries they need and copy those over. Sadly, the borg
|
||||||
|
executable can be either a binary executable (for which this works), or just a
|
||||||
|
Python script, whose demands REAR cannot fathom. This is undoubtedly a bug a
|
||||||
|
future REAR will fix, but in the meantime I just copied over the binary borg
|
||||||
|
into /usr/local/bin and used that instead of the Pythonic borg of the Debian
|
||||||
|
repos.
|
||||||
|
|
||||||
|
After those tweaks, tub was filled with the form of boat's Christmas past. I swapped
|
||||||
|
over the two drives, holding my breath, and losing the little SSD screw as
|
||||||
|
always (these plastic nubbin replacements tempted me in the fallout). Boat
|
||||||
|
wobbled and then, like a plucky North Sea veteran, bobbed back up.
|
||||||
|
|
||||||
|
Testing your recovery plans during your own end-of-year downtime and recovery
|
||||||
|
gives you a moment to peer at what has been, and prepare for the ups and downs
|
||||||
|
of the coming year. What will be the same? What will change? What parts of your
|
||||||
|
life can you simply hard link to the habits of the past? And what will you have
|
||||||
|
to incrementally add and integrate into your ever-evolving life?
|
||||||
|
|
||||||
|
Until next time, I am,
|
||||||
|
|
||||||
|
~Integrity Mather
|
||||||
|
|
||||||
Loading…
Reference in a new issue