Rolle Laukkarinen on Nostr: Mementomori.social experienced 57 minutes of downtime earlier today. Here's what ...
Mementomori.social experienced 57 minutes of downtime earlier today. Here's what happened and what we've done about it.
What happened: A large volume of connections from external crawler/bot IP ranges accumulated in a stuck state (CLOSE_WAIT) inside our web server (nginx). These zombie connections exhausted nginx's connection pool and connections per worker, which caused it to stop accepting new connections entirely resulting in ERR_CONNECTION_CLOSED for everyone trying to reach the site.
Why it took 57 minutes: The sole admin available (me) was in a meeting when the issue began, and other admins were unavailable. There was no automated recovery in place to detect and resolve this type of failure.
How it was resolved: Nginx was restarted, which cleared all stale connections and immediately restored service.
What we've done to prevent this:
- Installed and configured Monit, a process monitoring tool that automatically checks all critical services (web, background workers, streaming, database, cache, and nginx) every 2 minutes and restarts them if they become unresponsive
- This means similar outages should now self-heal within minutes, even when no admin is available
I'm really sorry for the disruption. We take the reliability of this instance seriously, and this change should make a real difference moving forward. I'm also planning to apply other mitigations, as some parts of our instance may have been underconfigured.
#MementoMoriSocial
Published at
2026-06-10 08:15:54 UTCEvent JSON
{
"id": "7f334bb7f426bda93ba70712cb4e2aa37b416ea6c82c2aa2426ea679078addc1",
"pubkey": "13522c83aa69ec0cff41dbff5f3a55e64aa998c5a20934fb15a2ca63108b8cf1",
"created_at": 1781079354,
"kind": 1,
"tags": [
[
"t",
"mementomorisocial"
],
[
"proxy",
"https://mementomori.social/@rolle/116724816543799638",
"web"
],
[
"proxy",
"https://mementomori.social/users/rolle/statuses/116724816543799638",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mementomori.social/users/rolle/statuses/116724816543799638",
"pink.momostr"
],
[
"-"
]
],
"content": "Mementomori.social experienced 57 minutes of downtime earlier today. Here's what happened and what we've done about it.\n\nWhat happened: A large volume of connections from external crawler/bot IP ranges accumulated in a stuck state (CLOSE_WAIT) inside our web server (nginx). These zombie connections exhausted nginx's connection pool and connections per worker, which caused it to stop accepting new connections entirely resulting in ERR_CONNECTION_CLOSED for everyone trying to reach the site.\n\nWhy it took 57 minutes: The sole admin available (me) was in a meeting when the issue began, and other admins were unavailable. There was no automated recovery in place to detect and resolve this type of failure.\n\nHow it was resolved: Nginx was restarted, which cleared all stale connections and immediately restored service.\n\nWhat we've done to prevent this:\n- Installed and configured Monit, a process monitoring tool that automatically checks all critical services (web, background workers, streaming, database, cache, and nginx) every 2 minutes and restarts them if they become unresponsive\n- This means similar outages should now self-heal within minutes, even when no admin is available\n\nI'm really sorry for the disruption. We take the reliability of this instance seriously, and this change should make a real difference moving forward. I'm also planning to apply other mitigations, as some parts of our instance may have been underconfigured.\n\n#MementoMoriSocial",
"sig": "206302ae2177c92d1f7e33117ce46edc2143402f4b7e9658f210f63e3d614d901e9fddf69ce1e46bd0b3663a0c7330a7f1c640a9447af20b75103e3fd5e6f084"
}
