So, I've mounted my snapshot from the 11th read-only.
I've re-formatted the volume XFS and have it mounted read-write.
`rsync` is copying the files. I see there's remnants of the old "combined" PostgreSQL + Mastodon system + redis + backups directories there (as btrfs subvolumes).
Some time ago I split it out to separate RBD volumes for performance reasons. So only the back-ups remained on the disk. Not quite sure why that brought the machine down, but we might get through this unscathed.
`rsync` of course is complaining about I/O errors transferring some of the back-up files, but whatever… the other volumes are good, I can recover from this. Back-ups will be on XFS from now on.
I've done a `fsck` of the other volumes, they're okay, but I've snapshotted them just in case. I can migrate those to XFS later. We'll let this run overnight and see how things are tomorrow.
Stuart Longland (VK4MSL)
npub12h…vts7y
2026-04-13 10:45:15 UTC
in reply to nevent1q…ezty
Author Public Key
npub12h9s2ds73gxsjmewpdf0z66q0elhx4xk6tww4w98d32p7u9kp60qyvts7ySeen on
wss://relay.momostr.pinkPublished at
2026-04-13 10:45:15 UTCEvent JSON
{
"id": "d10e5b489645924ae63b5146c7ed02318a0ada95fa787180e59b6ccceda96b10",
"pubkey": "55cb05361e8a0d096f2e0b52f16b407e7f7354d6d2dceab8a76c541f70b60e9e",
"created_at": 1776077115,
"kind": 1,
"tags": [
[
"e",
"461edc6a9be2958ac0c615720fe76b3a182731a755b47606ed41aedca98bf4eb",
"",
"reply",
"55cb05361e8a0d096f2e0b52f16b407e7f7354d6d2dceab8a76c541f70b60e9e"
],
[
"e",
"763e0ca1fbe60cfeaf7dc0224eefd25013583d0fcafdbcbeba3c569e27d24b0a",
"",
"root",
"55cb05361e8a0d096f2e0b52f16b407e7f7354d6d2dceab8a76c541f70b60e9e"
],
[
"proxy",
"https://aus.social/@vk4msl/116396989813057030",
"web"
],
[
"p",
"55cb05361e8a0d096f2e0b52f16b407e7f7354d6d2dceab8a76c541f70b60e9e"
],
[
"proxy",
"https://aus.social/users/vk4msl/statuses/116396989813057030",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://aus.social/users/vk4msl/statuses/116396989813057030",
"pink.momostr"
],
[
"-"
]
],
"content": "So, I've mounted my snapshot from the 11th read-only.\nI've re-formatted the volume XFS and have it mounted read-write.\n\n`rsync` is copying the files. I see there's remnants of the old \"combined\" PostgreSQL + Mastodon system + redis + backups directories there (as btrfs subvolumes).\n\nSome time ago I split it out to separate RBD volumes for performance reasons. So only the back-ups remained on the disk. Not quite sure why that brought the machine down, but we might get through this unscathed.\n\n`rsync` of course is complaining about I/O errors transferring some of the back-up files, but whatever… the other volumes are good, I can recover from this. Back-ups will be on XFS from now on.\n\nI've done a `fsck` of the other volumes, they're okay, but I've snapshotted them just in case. I can migrate those to XFS later. We'll let this run overnight and see how things are tomorrow.",
"sig": "801c3ce6d9089824d573dd8607b1443f27c613a8e18378990fd6363e537fd1f341264fda6465d1e3a102ebe7f7ae1311afcfd658ffa6543fd7a26bcb012b1e0d"
}