nostr-summary on Nostr: [ v0l/nostrsearch ] Bump nostr-archive-cursor: split the 149 GB shard across readers ...
[ v0l/nostrsearch ] Bump nostr-archive-cursor: split the 149 GB shard across readers
c111662 cuts large framed archives on frame boundaries so they can be read
concurrently. The backfill parallelises across files, so once the other 327
shards finished, the 149 GB combined shard ran alone on one thread doing
decompress, parse, dedupe, fold and index end to end -- 2.9 MB/s, with 13.6
hours projected for that file and a second pass still to come.
https://github.com/v0l/nostrsearch/commit/6c5263b03cb06a7d79b5be2c52b45307af2afd9ePublished at
2026-08-05 17:05:19 CESTEvent JSON
{
"id": "6ed957af8ba36d4d0980dcb833b51d663120946bec0b634c3c0a9d784c9b6be0",
"pubkey": "7febe2a59aa826f8f6337b4101939eaac82fd4e6ca5f3c7b8f13ca03ad26f2bf",
"created_at": 1785942319,
"kind": 1,
"tags": [
[
"proxy",
"https://github.com/v0l/nostrsearch/commit/6c5263b03cb06a7d79b5be2c52b45307af2afd9e",
"web"
]
],
"content": "[ v0l/nostrsearch ] Bump nostr-archive-cursor: split the 149 GB shard across readers\n\nc111662 cuts large framed archives on frame boundaries so they can be read\nconcurrently. The backfill parallelises across files, so once the other 327\nshards finished, the 149 GB combined shard ran alone on one thread doing\ndecompress, parse, dedupe, fold and index end to end -- 2.9 MB/s, with 13.6\nhours projected for that file and a second pass still to come.\nhttps://github.com/v0l/nostrsearch/commit/6c5263b03cb06a7d79b5be2c52b45307af2afd9e",
"sig": "f82653255e69504718a0a74ed1d29f4748121aa3dc84395d135863a19695b3ed4dd0cdb3cf781264c822a714dae356ce95bc2ea488a06a5cd178a56ea71c5e3f"
}
