Nostr notes by

📅 Original date posted:2014-07-31 📝 Original message:On ...

2023-06-07T15:24:39Z

In reply to nevent1q…0au2
_________________________

📅 Original date posted:2014-07-31
📝 Original message:On Thu, Jul 31, 2014 at 6:06 PM, Peter Todd <pete at petertodd.org> wrote:
> Anything that changes the semantics of nLockTime will do harm to
> existing and future applications that make use of nLockTime for things
> like refund transactions.

I think this would be compatible with most uses of nLockTime -- e.g.,
at the time a refund transaction becomes broadcastable, its
beneficiary would usually have no reason to wait for a long time
before broadcasting it; if they did so (probably because they weren't
online to redeem the refund), they'd need to use the
submit-directly-to-miner option, but they wouldn't lose their refund.

> In any case, why do transactions need finite lifespans in mempools? If
> you want to double-spend them with higher fees, then implement
> replace-by-fee.

Perpetuating transactions that have been in mempools for a long time
and are not being confirmed has been cited as a reason for nodes not
to exchange mempools (#3721, #1833, #3722); it's been implied that it
would be desirable for wallets to wait until a transaction had had a
chance to be accepted before double-spending with a higher fee
(#3722); and an unconfirmed transaction-age-based policy for
preventing mempool accumulation has been advocated (#3753, #3722) [I
hope my summarization is not misrepresenting anyone's opinions here;
please see the arguments made in the actual comments on the bugs].
These discussions are mostly fairly old, but I don't know of any
changes that have been made that provide alternative answers to these
concerns mentioned by at least three different developers.

> In any case, lifetimes will never be deterministic as not everyone runs
> the same software.

That's true, but none of the benefits of these changes require the
policy to be unanimous; most of the effect could be provided by most
of the network following these rules.

>> Transactions would stop being relayed and drop out of mempools a fixed
>> number of blocks from their creation; once that window had passed, the
>> sender's wallet could begin to expect the transaction would not be
>> confirmed. In case a reorg displaces a transaction until after its
>> expiry height, a miner can still put it back in the blockchain; the
>> expiry height is just a relay rule. Also, a user who needed to get
>> their original "expired" transaction confirmed could still do so by
>> submitting it directly to a miner with suitable policies.
>
> ...in which case someone will circumvent this IsStandard() rule by
> submitting "expired" transactions directly to miners with suitable
> policies.

Yes, that is a feature. None of the benefits of transaction expiration
rely on expiration being final, and any possible downsides of
expiration are largely mitigated by the option still being available
to get expired transactions mined.

📅 Original date posted:2014-07-31 📝 Original message:There ...

2023-06-07T15:24:38Z

In reply to nevent1q…hp38
_________________________

📅 Original date posted:2014-07-31
📝 Original message:There is currently little in place for managing transaction lifetime
in the network's mempools (see discussion in github in #3722 "mempool
transaction expiration", and it seems to be a major factor blocking
some mempool exchange, see #1833/1918, #3721). Expiry per-node a
certain amount of wall time after receipt has been proposed, but
that's a fragile mechanism -- a single node could keep all relayable
transactions alive forever by remembering transactions until most
nodes have dropped them and then releasing them back into the wild.

I have a proposal for a way to add finite and predictable lifespans to
transactions in mempools: we d̶e̶s̶t̶r̶o̶y̶ ̶t̶h̶e̶
̶r̶e̶s̶u̶r̶r̶e̶c̶t̶i̶o̶n̶ ̶h̶u̶b̶ use nLockTime and a new standardness
rule. It could be done in stages, would not necessarily require even a
soft fork, and does not cause problems with reorgs like the proposal
in #3509:
1. start setting nLockTime to the current height by default in newly
created transactions (or slightly below the current height, for
reorg-friendliness)
2. once users have had some time to upgrade to clients that set
nLockTime, start discouraging transactions without nLockTime --
possibly with a slightly higher fee required for relay
3. start rate-limiting relay of transactions without an nLockTime
(maybe this alone could be used to achieve [2])
4. add a new IsStandard rule rejecting transactions with an nLockTime
more than N blocks behind the current tip (for some fixed value N, to
be determined)

Transactions would stop being relayed and drop out of mempools a fixed
number of blocks from their creation; once that window had passed, the
sender's wallet could begin to expect the transaction would not be
confirmed. In case a reorg displaces a transaction until after its
expiry height, a miner can still put it back in the blockchain; the
expiry height is just a relay rule. Also, a user who needed to get
their original "expired" transaction confirmed could still do so by
submitting it directly to a miner with suitable policies.

📅 Original date posted:2014-07-31 📝 Original message:> ...

2023-06-07T15:24:12Z

In reply to nevent1q…kdvm
_________________________

📅 Original date posted:2014-07-31
📝 Original message:> the need to have transmitted the transaction list [..] first

32 bits per transaction is at least double the communication overhead
of the simple approach, and only offers a bound on the probability of
needing a round trip.

On Thu, Jul 31, 2014 at 2:29 PM, Gregory Maxwell <gmaxwell at gmail.com> wrote:
> On Thu, Jul 31, 2014 at 1:47 PM, Kaz Wesley <keziahw at gmail.com> wrote:
>> trip to request the missing tx; if we could somehow get the "What's
>> the Difference" approach to effectively operate on full transactions
>> instead
>
> I explain how to do this on the network block coding page.
>
> Given that you know the sizes and orders of the transactions (e.g.
> from a reconciliation step first), the sender sends non-syndromic
> forward error correcting code data somewhat larger than their estimate
> of how much data the user is missing. Then you drop the data you know
> into place and then recover the missing blocks using the fec.
>
> There is no overhead in this approach except for FEC blocks that are
> incompletely missing (and so must be completely discarded), and the
> need to have the transmitted the transaction list and sizes first.
> (note, that just more bandwidth, not an additional round trip).

📅 Original date posted:2014-07-31 📝 Original message:I ...

2023-06-07T15:24:11Z

In reply to nevent1q…up3k
_________________________

📅 Original date posted:2014-07-31
📝 Original message:I don't see how set reconciliation alone would be practical for
condensed block exchange -- if the keys are txids it'd require a round
trip to request the missing tx; if we could somehow get the "What's
the Difference" approach to effectively operate on full transactions
instead, the log(keysize) factor overhead would make any transactions
not mutually-known very expensive to exchange (at keysize=32b, data
would need to be 80% mutually-known just to break even). There's also
the complication and/or overhead of establishing an "expected block"
to reconcile with the actual block.

The approach of remembering what invs have been transmitted both
directions along each connection is less elegant; it requires
remembering a lot of communication history, introducing a major point
of statefulness to the protocol, and custom-compacting blocks for each
peer. But it is also very effective at squeezing bytes, cheap in cpu
cycles, and the implementation is fairly simple. The wealth of mutual
knowledge already available in the current protocol allows
accomplishing the goal of exchanging blocks efficiently by solving a
much easier problem than its context-less cousin. I have my doubts
that it is possible for even an optimal contextless solution to do as
well as channel memory in terms of bytes exchanged or computational
complexity -- you can't beat making use of the available information.

I have an implementation of inv-history-tracking that uses a 2b/tx
alternative to getdata for tx, and I've had that running between two
nodes for ~2 weeks now. I've been working on a better implementation
of that plus the sparseblock messages, and I'll have the sparseblock
prototype (suitable for something like Gregory's remember-last-N
approach) up and running in a couple of days or so. The prototype
handles assigning compact identifiers to transactions and using those
in block messages; there's a lot of bit-packing sort of tweaks that
can be done that I'm not including in the initial prototype. The
prototype will be able to log history-hit rates, so if we run a few
sparseblocks nodes connected to each other for a while we should get a
good idea of how much efficiency gain this provides, and how it can be
improved. This approach even without the intensive bit packing has a
total vtx transmission size of 2*nTxKnown + 1*nTxUnknown +
nBytesTxUnknown, where only a small window of very recent transactions
and any transactions that have fallen out of the history limit would
be mutually known but not known to be known.

It would be possible to nearly eliminate even that overhead for both
known and unknown transactions with compact descriptions of block tx
inclusion and ordering policies as Gavin brought up, for which
something like scripts defining priority formulas would be a possible
implementation (https://gist.github.com/kazcw/43c97d3924326beca87d#ordering-policy
-- n.b. most of the rest of the gist is currently outdated). But since
priority scripts are themselves more complicated than the rest of the
sparseblock implementation, and basic sparseblocks achieve the vast
majority of bandwidth savings, I think it's worth implementing
sparseblocks without priority scripts now and then using priority
scripts for sparseblocks2 along with all the other things they can do
later.

Set reconciliation does look like a great way to synchronize mempools.
I've been thinking, contextless low-cost mempool exchange would enable
a node to have one or more "roaming" peer slots -- connect to a node,
fill in each other's mempools, move on to another peer. It seems like
this would go a long way to mitigate potential pathological network
topologies -- it would make it very difficult to sybil attack a node
(barring an attacker in a position to spoof IP addresses), and if a
serious bug or DoS attack caused the network to start to partition
itself due to DoS bans, it only takes occasional roamers crossing the
partition to keep both sides generally in sync.
Efficient mempool synchronization would also increase the efficacy of
channel-memory sparseblocks: it picks up transactions too old to have
been exchanged via invs, and could also allow nodes to know exactly
what transactions their peers have discarded.

On Thu, Jul 31, 2014 at 8:31 AM, Gavin Andresen <gavinandresen at gmail.com> wrote:
> I've been reading up on set reconciliation algorithms, and thinking about
> constant-bandwidth propagation of new block announcements.
>
> Emin: the approach in this paper:
> What's the Difference? Efficient Set Reconciliation without Prior Context
> http://conferences.sigcomm.org/sigcomm/2011/papers/sigcomm/p218.pdf
>
> ... looks like it would be much better suited for Bitcoin's use case,
> because
>
> a) it looks much easier to implement (no polynomial math)
> b) CPU/latency versus bandwidth tradeoff looks better (somewhat higher
> bandwidth than Yaron's method, but much lower CPU/latency cost)
>
> Kaz: how much time do you have to work on this? Perhaps we can get a
> road-map for a prototype and split up work. I actually think the first step
> might be to gather a couple days worth of tx / block message broadcasts from
> a few network nodes and then create a test/benchmark harness that will tell
> us how much overlap there is in what nodes know and how much faster our
> newer algorithms are.
>
> --
> --
> Gavin Andresen

On Thu, Jul 31, 2014 at 12:10 PM, Emin Gün Sirer <el33th4x0r at gmail.com> wrote:
> Hi Gavin,
>
> Great find. I read (and sadly forgot) this paper back in 2011 and indeed,
> I'd also pick this approach over Yaron's. My reasons:
>
> a. this paper provides a more concrete implementation roadmap that has been
> explored thoroughly. While I understand Yaron's technique, there are still
> various design decisions (e.g. estimating the set difference involves a
> doubling process; representing TX ids seems to take up a lot of space) that
> are left unspec'ed for an implementation. (In general, Yaron is a
> theoretician at heart and Varghese is a very practical guy, there will be
> far fewer unforeseen issues with a Varghese tried and tested algorithm).
>
> b. on the surface, this paper seems to use a bit more bandwidth in that it
> requires O(size of set diff * log of keyspace) vs. Yaron's claim of O(size
> of set diff). But I believe that in practice Yaron's method would also have
> an O(log of keyspace) multiplier in place because the roots of his
> polynomial (the TX ids) have to be represented as numbers, and that requires
> log-of-keyspace bits. I suspect that Yaron is cleverly folding that factor
> into the constant factor of O notation, and Varghese et al are being polite
> by not pointing it out too overtly. So I suspect that the two schemes are
> actually identical in terms of space complexity.
>
> c. this technique is far more efficient in decoding than Yaron's, which
> requires Gaussian elimination, which in turn is O(d^3).
>
> If Bitcoin adopts this technique, it'll be adopting one of the best known
> techniques from the research community.
>
> BTW, don't hesitate to ping me with researchy issues in the future; I'll
> gladly point the effort in the right direction if I can.
>
> - egs
>
>
>
> On Thu, Jul 31, 2014 at 6:31 PM, Gavin Andresen <gavinandresen at gmail.com>
> wrote:
>>
>> I've been reading up on set reconciliation algorithms, and thinking about
>> constant-bandwidth propagation of new block announcements.
>>
>> Emin: the approach in this paper:
>> What's the Difference? Efficient Set Reconciliation without Prior Context
>> http://conferences.sigcomm.org/sigcomm/2011/papers/sigcomm/p218.pdf
>>
>> ... looks like it would be much better suited for Bitcoin's use case,
>> because
>>
>> a) it looks much easier to implement (no polynomial math)
>> b) CPU/latency versus bandwidth tradeoff looks better (somewhat higher
>> bandwidth than Yaron's method, but much lower CPU/latency cost)
>>
>> Kaz: how much time do you have to work on this? Perhaps we can get a
>> road-map for a prototype and split up work. I actually think the first step
>> might be to gather a couple days worth of tx / block message broadcasts from
>> a few network nodes and then create a test/benchmark harness that will tell
>> us how much overlap there is in what nodes know and how much faster our
>> newer algorithms are.
>>
>> --
>> --
>> Gavin Andresen
>
>

📅 Original date posted:2014-07-18 📝 Original ...

2023-06-07T15:24:09Z

In reply to nevent1q…d4qu
_________________________

📅 Original date posted:2014-07-18
📝 Original message:That's true, but I think it can be balanced with the usefulness of
knowing what messages a node has received. An invack would be sent if
N invs have been received without any resulting getdata; since we're
keeping track of peer inv order, one message can cover an arbitrarily
large series of invs.

On Fri, Jul 18, 2014 at 10:48 AM, Jeff Garzik <jgarzik at bitpay.com> wrote:
> On a flood-fill network, you don't want to create a storm of "I
> already have this" replies.
>
> On Fri, Jul 18, 2014 at 1:39 PM, Kaz Wesley <keziahw at gmail.com> wrote:
>> Peers exchanging mempool priority policies is great; that accomplishes
>> the flexibility in what txes to remember that I was going for with the
>> forget-filters, but much more neatly, with less overhead and some side
>> benefits.
>>
>> Here's what I'm picturing now:
>> - exchange priority policies in peer introductions
>> - assign unique sequential IDs in the order the transactions were
>> inved (per peer)
>> - receiving a getdata for a tx updates last-known-peer-received inv to
>> all invs up to the one referenced
>> - include ID-last-received, last-known-peer-received in sparse block
>> - reference txes in sparse block by index in receiver's
>> prioritiziation with peer's sent invs up to ID-last-received and
>> sender's prior invs up to last-known-peer-received
>>
>> Possible new messages:
>> - sparseblock
>> - invack message a node can send at times when it's received a bunch
>> of invs it already has, so it hasn't acked with a getdata in a while
>> - gettx: getdata, but using new sequential ID to save 28 bytes per tx
>>
>> It seems important for ordering policies to be able to be specified in
>> as much detail as possible. Parameters that should be available:
>> - total inputs
>> - total outputs
>> - bytes
>> - coin days destroyed
>> - net UTXO size change
>> - sigops
>> - is data carrier
>> - is output raw multisig
>> - age in mempool
>> - what else?
>> This parameter set should be extensible to allow for unforeseen future factors.
>>
>> Ordering policies should allow arbitrary algebraic combinations of
>> their parameters, as well as thresholds. Boolean combinations of
>> sub-policies would also be desirable. This could be implemented with a
>> tx-script-like stack-based language, in which each supported tx
>> property is pushed onto the stack by a particular opcode, and
>> +-*//min/max/boolean operators combine them to yield the sort key.
>>
>> Difficult parameters:
>> * Coin-days-destroyed: changes, peers need agreement on when (if?)
>> it's recalculated. Probably can just not recalculate, but peers still
>> need agreement on "time seen" to get CDD.
>> * Age in mempool: seems intractable in terms of time, but could be
>> done easily in terms of "how many txes old is this sequential ID"
>>
>> One potential pitfall: this allows for an environment of completely
>> heterogeneous mempool policies. I think that's a good thing, but we
>> need to avoid a situation where only least-common-denominator
>> transactions make it farther than a hop or two, and we don't want
>> nodes to have a strong preference for connecting to like-minded peers
>> since clustering reduces overall connectivity. It may be worthwhile to
>> add a parallel mechanism for relay policies, to differentiate between
>> what a node would keep in its mempool vs. what it wouldn't even relay
>> and doesn't want to see at all. Relay policies could be specified just
>> like prioritization policies, but with the final stack value evaluated
>> in a boolean context.
>>
>> An interesting additional use of policy-scripts would be a
>> standardized way for miners to include a policy script in a coinbase,
>> allowing miners a mechanism to advertise things like their relative
>> price of sigops vs bytes. Nodes may then choose to take this
>> information into account in order to optimize their mempool policies
>> for likelihood of consistency with future blocks. Since policy scripts
>> provide only relative information on prices of different transaction
>> properties rather than an absolute fee, this should not allow miners
>> to "vote fees up", although care would need to be taken they wouldn't
>> be able to drive up prices by claiming common transaction types are at
>> the high end of the fee scale.
>>
>> ------------------------------------------------------------------------------
>> Want fast and easy access to all the code in your enterprise? Index and
>> search up to 200,000 lines of code with a free copy of Black Duck
>> Code Sight - the same software that powers the world's largest code
>> search on Ohloh, the Black Duck Open Hub! Try it now.
>> http://p.sf.net/sfu/bds
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc. https://bitpay.com/

📅 Original date posted:2014-07-18 📝 Original ...

2023-06-07T15:24:09Z

In reply to nevent1q…6pfs
_________________________

📅 Original date posted:2014-07-18
📝 Original message:I've updated the gist, and added an additional proposal that I think
meshes well:
https://gist.github.com/kazcw/43c97d3924326beca87d#ultra-fast-block-validation

sparseblocks + UFBV would tighten the new-block process to this (when
txes have been received in advance):
- receive block (~2kB for 1000 tx)
- check whether block contains txes known to belong to conflict-sets,
and if so whether more than one tx from a single conflict-set has been
included (a few operations on very small sets)
- relay block (~2kB)

The benefits of these changes only occur when the transactions have
been seen in advance, but incentivizing ahead-of-block transaction
propogation is a plus, as Jeff mentioned; working on a block without
first ensuring peers have its transactions would be very expensive
from a miner's point of view.

📅 Original date posted:2014-07-18 📝 Original message:Peers ...

2023-06-07T15:24:08Z

In reply to nevent1q…rkvx
_________________________

📅 Original date posted:2014-07-18
📝 Original message:Peers exchanging mempool priority policies is great; that accomplishes
the flexibility in what txes to remember that I was going for with the
forget-filters, but much more neatly, with less overhead and some side
benefits.

Here's what I'm picturing now:
- exchange priority policies in peer introductions
- assign unique sequential IDs in the order the transactions were
inved (per peer)
- receiving a getdata for a tx updates last-known-peer-received inv to
all invs up to the one referenced
- include ID-last-received, last-known-peer-received in sparse block
- reference txes in sparse block by index in receiver's
prioritiziation with peer's sent invs up to ID-last-received and
sender's prior invs up to last-known-peer-received

Possible new messages:
- sparseblock
- invack message a node can send at times when it's received a bunch
of invs it already has, so it hasn't acked with a getdata in a while
- gettx: getdata, but using new sequential ID to save 28 bytes per tx

It seems important for ordering policies to be able to be specified in
as much detail as possible. Parameters that should be available:
- total inputs
- total outputs
- bytes
- coin days destroyed
- net UTXO size change
- sigops
- is data carrier
- is output raw multisig
- age in mempool
- what else?
This parameter set should be extensible to allow for unforeseen future factors.

Ordering policies should allow arbitrary algebraic combinations of
their parameters, as well as thresholds. Boolean combinations of
sub-policies would also be desirable. This could be implemented with a
tx-script-like stack-based language, in which each supported tx
property is pushed onto the stack by a particular opcode, and
+-*//min/max/boolean operators combine them to yield the sort key.

Difficult parameters:
* Coin-days-destroyed: changes, peers need agreement on when (if?)
it's recalculated. Probably can just not recalculate, but peers still
need agreement on "time seen" to get CDD.
* Age in mempool: seems intractable in terms of time, but could be
done easily in terms of "how many txes old is this sequential ID"

One potential pitfall: this allows for an environment of completely
heterogeneous mempool policies. I think that's a good thing, but we
need to avoid a situation where only least-common-denominator
transactions make it farther than a hop or two, and we don't want
nodes to have a strong preference for connecting to like-minded peers
since clustering reduces overall connectivity. It may be worthwhile to
add a parallel mechanism for relay policies, to differentiate between
what a node would keep in its mempool vs. what it wouldn't even relay
and doesn't want to see at all. Relay policies could be specified just
like prioritization policies, but with the final stack value evaluated
in a boolean context.

An interesting additional use of policy-scripts would be a
standardized way for miners to include a policy script in a coinbase,
allowing miners a mechanism to advertise things like their relative
price of sigops vs bytes. Nodes may then choose to take this
information into account in order to optimize their mempool policies
for likelihood of consistency with future blocks. Since policy scripts
provide only relative information on prices of different transaction
properties rather than an absolute fee, this should not allow miners
to "vote fees up", although care would need to be taken they wouldn't
be able to drive up prices by claiming common transaction types are at
the high end of the fee scale.

📅 Original date posted:2014-07-17 📝 Original ...

2023-06-07T15:24:07Z

In reply to nevent1q…pmyl
_________________________

📅 Original date posted:2014-07-17
📝 Original message:I'm moving this design document to a gist so that I can integrate
changes as they come up:
https://gist.github.com/kazcw/43c97d3924326beca87d
One thing that I think is an important improvement over my initial
idea is that the bloom filters don't need to be kept around and built
up, they can just be one-shot and clear any matching entries from the
set of known-knowns upon arrival -- provided a node is careful to
ensure the txes it wants to forget are known-known-known (which isn't
as bad as it sounds) to the peer it's telling it's forgetting them
when the forget-filter arrives.

On Thu, Jul 17, 2014 at 3:46 PM, Gavin Andresen <gavinandresen at gmail.com> wrote:
>
> A couple of half-baked thoughts:
>
> On Thu, Jul 17, 2014 at 5:35 PM, Kaz Wesley <keziahw at gmail.com> wrote:
>>
>> If there's support for this proposal, I can begin working on the specific
>> implementation details, such as the bloom filters, message format, and
>> capability advertisment, and draft a BIP once I have a concrete proposal for
>> what those would look like and a corresponding precise cost/benefit analysis.
>
>
> I'd encourage you to code up a prototype first (or at the same time), in whatever programming language / networking library you're most familiar with.
>
> Maybe not even using the existing p2p protocol; there could be a mining-only very-fast-block-propagation network separate from the existing p2p network.
>
> Combining your optimizations with "broadcast as many near-miss blocks as bandwidth will allow" on a mining backbone network should allow insanely fast propagation of most newly solved blocks.
>
> --
> --
> Gavin Andresen

Thanks Gavin, I am planning on working out the design details as I
work on a prototype. I have the beginnings of a previous shot at
implementing this in bitcoind to start from but my new design has some
important improvements to add to that.

-kaz

📅 Original date posted:2014-07-17 📝 Original ...

2023-06-07T15:24:06Z

In reply to nevent1q…xsh3
_________________________

📅 Original date posted:2014-07-17
📝 Original message:OVERVIEW

To improve block propagation, add a new block message that doesn't include
transactions the peer is known to have. The message must never require an
additional round trip due to any transactions the peer doesn't have, but
should
be compatible with peers sometimes forgetting transactions they have known.

APPROACH

For peers advertising support for squashed blocks: a node tracks what txes
it
knows each peer has seen (inv received, tx sent, tx appeared in competing
block
known to peer). Nodes push block contents as txes-not-already-known +
txids-known.

A node should be able to forget invs it has seen without invalidating what
peers
know about its known txes. To allow for this, a node assembles a bloom
filter of
a set of txes it is going to forget, and sends it to peers. The node can
erase
the txes as soon as no blocks requested before the filter was pushed are in
flight (relying on the assumption that messages can be expected to be
processed
in order).

When a node receives a forgotten-filter, it ORs it into its
forgotten-filter for
that peer. Any transactions matching the forgotten-filter are always
included in
full with a block. If the filter is getting full, the node can just clear it
along with peer.setTxKnown.

COSTS

Bloom filtering:
Since the bloom filter is likely to grow slowly and can be dropped when it
is
becoming full, a cheap set of hash functions and element size can be used to
keep overhead more restricted than the bloom filtering done for spv. It's
important for testing txes against the filter to be fast so that it doesn't
delay pushing the block more than the squashing helps.
Nodes currently forget txes rarely, so the bloom filters would only need to
be
used at all under conditions that are not currently common -- but I think
they're important to include to allow for different node behavior in this
regard
in the future.

Tracking txes known to peers:
A multimap of txid->peerId would obviate the current setCurrentlyKnown, and
would not take much more space since each additional peer adds about 1
peerId
per txid (setCurrentlyKnown keeps a uint256 per peer per txid, although it
tracks somewhat fewer txid per node).

Potential vulnerabilities:
- Since the bloom filters will have lower maximum overhead than the current
SPV
filters and can be dropped at will, this shouldn't enable any resource
exhaustion attacks that aren't already possible.
- A squashed block with bogus or missing data would be easily detected not
to
produce the correct merkle root for its BlockHeader.

BENEFITS

Assuming a fairly typical 500 tx block with transaction sizes averaging 300b
(both on the low side), for a 150kb block:

% pruned | block size reduction | relative size reduction
-------- | -------------------- | -----------------------
100 | 134 kB | 89%
50 | 67 kB | 45%
25 | 33.5 kB | 17%

I've been doing some logging, and when my node pushes a block to a peer it
seems
to typically know that a peer has seen most of the txes in the block. Even
in
the case of a small block with only 25% known-known transactions, total
network
bandwidth saved is greater than the bloom filters transmitted unless a node
is
forgetting transactions so rapidly that it pushes new maximum-size
forget-filters every block.

So this is a net gain even in total bandwidth usage, but most importantly
it's
an improvement in block propagation rate and in how block propagation rate
scales with additional transactions.

IMPLEMENTATION QUESTIONS

How should block squashing capability be advertised -- new service bit?

Bloom filters:
- How fast to test against could a suitable bloom filter be made?
- How much memory would each filter need to take, at maximum?
- Can the inputs all being 32 byte hashes be used to optimize filter hash
calculations?

ROADMAP

If there's support for this proposal, I can begin working on the specific
implementation details, such as the bloom filters, message format, and
capability advertisment, and draft a BIP once I have a concrete proposal for
what those would look like and a corresponding precise cost/benefit
analysis.

--kaz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20140717/810d2f69/attachment.html>;