{"type":"rich","version":"1.0","author_name":"npub19helcfnqgk2jrwzjex2aflq6jwfc8zd9uzzkwlgwhve7lykv23mq5zkvn4","author_url":"https://nostr.ae/npub19helcfnqgk2jrwzjex2aflq6jwfc8zd9uzzkwlgwhve7lykv23mq5zkvn4","provider_name":"njump","provider_url":"https://nostr.ae","html":"📅 Original date posted:2018-05-21\n📝 Original message:Hi Y'all,\n\nThe script finished a few days ago with the following results:\n\nreg-filter-prev-script total size:  161236078  bytes\nreg-filter-prev-script avg:         16123.6078 bytes\nreg-filter-prev-script median:      16584      bytes\nreg-filter-prev-script max:         59480      bytes\n\nCompared to the original median size of the same block range, but with the\ncurrent filter (has both txid, prev outpoint, output scripts), we see a\nroughly 34% reduction in filter size (current median is 22258 bytes).\nCompared to the suggested modified filter (no txid, prev outpoint, output\nscripts), we see a 15% reduction in size (median of that was 19198 bytes).\nThis shows that script re-use is still pretty prevalent in the chain as of\nrecent.\n\nOne thing that occurred to me, is that on the application level, switching\nto the input prev output script can make things a bit awkward. Observe that\nwhen looking for matches in the filter, upon a match, one would need access\nto an additional (outpoint -\u003e script) map in order to locate _which_\nparticular transaction matched w/o access to an up-to-date UTOX set. In\ncontrast, as is atm, one can locate the matching transaction with no\nadditional information (as we're matching on the outpoint).\n\nAt this point, if we feel filter sizes need to drop further, then we may\nneed to consider raising the false positive rate.\n\nDoes anyone have any estimates or direct measures w.r.t how much bandwidth\ncurrent BIP 37 light clients consume? It would be nice to have a direct\ncomparison. We'd need to consider the size of their base bloom filter, the\naccumulated bandwidth as a result of repeated filterload commands (to adjust\nthe fp rate), and also the overhead of receiving the merkle branch and\ntransactions in distinct messages (both due to matches and false positives).\n\nFinally, I'd be open to removing the current \"extended\" filter from the BIP\nas is all together for now. If a compelling use case for being able to\nfilter the sigScript/witness arises, then we can examine re-adding it with a\ndistinct service bit. After all it would be harder to phase out the filter\nonce wider deployment was already reached. Similarly, if the 16% savings\nachieved by removing the txid is attractive, then we can create an\nadditional\nfilter just for the txids to allow those applications which need the\ninformation to seek out that extra filter.\n\n-- Laolu\n\n\nOn Fri, May 18, 2018 at 8:06 PM Pieter Wuille \u003cpieter.wuille at gmail.com\u003e\nwrote:\n\n\u003e On Fri, May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev \u003c\n\u003e bitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\n\u003e\u003e Greg wrote:\n\u003e\u003e \u003e What about also making input prevouts filter based on the scriptpubkey\n\u003e\u003e being\n\u003e\u003e \u003e _spent_?  Layering wise in the processing it's a bit ugly, but if you\n\u003e\u003e \u003e validated the block you have the data needed.\n\u003e\u003e\n\u003e\u003e AFAICT, this would mean that in order for a new node to catch up the\n\u003e\u003e filter\n\u003e\u003e index (index all historical blocks), they'd either need to: build up a\n\u003e\u003e utxo-set in memory during indexing, or would require a txindex in order to\n\u003e\u003e look up the prev out's script. The first option increases the memory load\n\u003e\u003e during indexing, and the second requires nodes to have a transaction index\n\u003e\u003e (and would also add considerable I/O load). When proceeding from tip, this\n\u003e\u003e doesn't add any additional load assuming that your synchronously index the\n\u003e\u003e block as you validate it, otherwise the utxo set will already have been\n\u003e\u003e updated (the spent scripts removed).\n\u003e\u003e\n\u003e\n\u003e I was wondering about that too, but it turns out that isn't necessary. At\n\u003e least in Bitcoin Core, all the data needed for such a filter is in the\n\u003e block + undo files (the latter contain the scriptPubKeys of the outputs\n\u003e being spent).\n\u003e\n\u003e I have a script running to compare the filter sizes assuming the regular\n\u003e\u003e filter switches to include the prev out's script rather than the prev\n\u003e\u003e outpoint itself. The script hasn't yet finished (due to the increased I/O\n\u003e\u003e load to look up the scripts when indexing), but I'll report back once it's\n\u003e\u003e finished.\n\u003e\u003e\n\u003e\n\u003e That's very helpful, thank you.\n\u003e\n\u003e Cheers,\n\u003e\n\u003e --\n\u003e Pieter\n\u003e\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180521/feb53119/attachment.html\u003e"}