Nostr notes by

📅 Original date posted:2018-05-08 📝 Original message: ...

2023-06-09T14:50:41+02:00

In reply to nevent1q…8czp
_________________________

📅 Original date posted:2018-05-08
📝 Original message:
Benjamin,

I don't agree that quantum resistance should be a blocker to deployment of
scriptless scripts on lightning because 1) it is a layer-2 solution and 2)
it already critically depends on the security of DL.

There are arguments against making certain protocol changes to the base
Bitcoin blockchain for the reason that they may not be quantum resistant.
The most notable is Confidential Transactions. There reason is that the
worst case attack is much worse: an attacker could print money freely and
without detection. In Bitcoin today, a DL break would compromise funds in
any addresses for which the pubkey has been revealed and it's not even
clear what to do about the remaining funds on chain. Compromising the
fundamental security of the blockchain is a valid cause for concern.

In the case of Lightning, the attack scenario on scriptless scripts is that
a peer is going to use a quantum computer to steal all live payments routed
through them from their senders before they get to the recipient. This
would be bad, but not catastrophic, and once it is recognized that the
attack is possible, insecure channels could be closed.

But furthermore, an attacker with a quantum computer could just steal the
multisig funding output directly instead of attacking scriptless scripts.
So additional protocol changes relying on the DL assumption don't bother me
in the least.

On Tue, May 8, 2018, 6:32 AM Benjamin Mord <ben at mord.io> wrote:

>
> Sorry, I do not wish to spam the list, but I need to correct a rather
> serious error in my last email. We must never call something
> "post-quantum", absent mathematical proof. (And good luck with that.) I
> apologise for my mistake in doing so myself.
>
> I should not even refer to lattice based cryptography as a post-quantum
> algorithm, I should at best call it a Shor's algorithm-resistant scheme. At
> least, it is not (yet) known how Shor's algorithm could be used to break
> it. Not in public circles, anyhow.
>
> A cursory glance at the history of cryptanalysis shows that primitives
> generally have finite life, which makes it odd that systems seldom use
> redundant primitives, and seldom provide for their rapid and safe swap. A
> system whose entire security rests on nothing but cryptography, ought to
> take particular care! The spectre of quantum computers may render the
> finite life of certain primitives more salient than others, but we must not
> suppose that, absent Shor's algorithm, there would be no need to plan for
> cryptographic failures. The question is not if, but when, Bitcoin and
> lightning will contend with broken primitives, whether due to classical or
> quantum cryptanalysis. Likely, both will come into play at various times,
> and one must plan accordingly.
>
> On Tue, May 8, 2018, 9:09 AM Benjamin Mord <ben at mord.io> wrote:
>
>>
>> That would be awesome. Do you have a reference?
>>
>> As pertains to the whole of asymmetric cryptography, I believe there are
>> not a variety of post quantum schemes, there is only one*: lattice-based
>> cryptography. (Which scares me, because it is not all that different from
>> the others.)
>>
>> (* Actually, in contexts where time can be used for asymmetry, as in
>> TESLA, we can then use hash functions to create something like asymmetric
>> signatures as well. But the functional context has to be compatible with
>> delayed verification.)
>>
>> (But I do not mean to focus exclusively on Schor's algorithm, the history
>> of even pre-quantum cryptanalysis shows that primitives tend to have finite
>> lifespan. Redundancy of any sort of good, even when not focused
>> specifically on quantum risks.)
>>
>> On Tue, May 8, 2018, 8:58 AM Greg Sanders <gsanders87 at gmail.com> wrote:
>>
>>> From what I understand talking to folks, the linear properties of these
>>> signature tricks are maintained under a number of post-quantum schemes.
>>>
>>> On Tue, May 8, 2018 at 8:44 AM, Benjamin Mord <ben at mord.family> wrote:
>>>
>>>>
>>>> If I'm not mistaken, the scriptless scripts concept (as currently
>>>> formulated) falls to Schor's algorithm, and at present there is no
>>>> alternative implementation of the concept to fall back on. Correct? Lest we
>>>> build a house of cards, I'd strongly urge everyone to not depend on
>>>> functional concepts whose underlying cryptographic primitives cannot be
>>>> swapped in an emergency.
>>>>
>>>> Sure, we use ecdsa for example (which is also vulnerable to Schor's
>>>> algorithm), but in contrast to scriptless scripts we have a variety of
>>>> backup primitives at our disposal that fulfill the same functional
>>>> objective.
>>>>
>>>> If scriptless scripts are found possible under lattice-based
>>>> cryptography for example, that would be something I suppose. The functional
>>>> concept of scriptless scripts is indeed very awesome - we just need to add
>>>> some cryptographic conservatism before we build on it.
>>>>
>>>>
>>>> _______________________________________________
>>>> Lightning-dev mailing list
>>>> Lightning-dev at lists.linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>>>>
>>>>
>>> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180508/f3ebdf1a/attachment.html>;

📅 Original date posted:2018-05-18 📝 Original message: Yes ...

2023-06-09T14:50:29+02:00

In reply to nevent1q…v5ru
_________________________

📅 Original date posted:2018-05-18
📝 Original message:
Yes Rusty, you are correct that an attacker still gets leverage on their
ability to destroy reputation unless the loss rates increase exponentially.
And I agree that would be a very steep increase, serving do decrease
circuit lengths dramatically. The reasoning for the linear increase in
reputation loss comes from viewing reputation loss as compensation for
time-value lost in locked HTLCs. In a sense, every node pays in reputation
for all of the time-value locked in HTLCs upstream from them in proportion
to the value assigned to it by the owners of the funds.

I'm not sure whether this would be an acceptable way of pricing payment
delays, though I do think it is important to make attackers pay for the
resources they are wasting on the network in some form.

On Fri, May 18, 2018 at 4:38 PM, ZmnSCPxj <ZmnSCPxj at protonmail.com> wrote:

> Good morning Rusty,
>
>
> > Also, you talked about reputation_loss_rate as being a private per-node
> >
> > thing, and being an explicit thing in the HTLC. I'm ignoring the
> >
> > former, and assuming the latter.
>
> Reputation (the score) is a private per-node thing, while the
> `reputation_loss_rate` is explicit in the HTLC.
>
> I am uncertain how that changes your analysis, though. In a line network
> like you showed, the reputation "bins" are Node1->Node2 and Node2->Node1
> and so on. It may be more useful to think of the reputation bins as
> assigned to half-chans than to nodes.
>
> So the initial Mallory3->Node18->Mallory2 gives high reputation to
> half-chans Node18->Mallory3 and Node18->Mallory2, then sacrifices the
> Node18->Mallory3 reputation to destroy the NodeN -> NodeN+1 and
> NodeN+1->NodeN reputations.
>
> Along that line, reputation lost is higher as N increases. Graphing
> reputation lost along that line, we form a triangle, and the area of the
> triangle is the total reputation destroyed. Only the length of one edge of
> the triangle is what is lost by the Node18->Mallory3 reputation score. So
> yes, it seems you are correct here.
>
> Regards,
> ZmnSCPxj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180518/f341c9c4/attachment.html>;

📅 Original date posted:2018-05-15 📝 Original message: > ...

2023-06-09T14:50:28+02:00

In reply to nevent1q…cern
_________________________

📅 Original date posted:2018-05-15
📝 Original message:
>
> This can still be manipulated if Rusty1 opens a direct channel to Jim.
> Then Rusty1 can route payments Rusty1->Jim->Rusty2 that succeed quickly,
> then route payments Rusty1->ZmnSCPxj->Jim->Rusty2 that stall. Thus Rusty2
> can have the Jim->Rusty2 reputation boosted, while alternating with
> reputation losses that make ZmnSCPxj->Jim reputation go down. Since local
> reputation is used, ZmnSCPxj and Jim will not talk to each other about how
> Rusty2 seems to stall when not routing a payment from Rusty1. Rusty1 can
> now manipulate the reputation view of ZmnSCPxj and Jim of each other while
> keeping Rusty2 reputation somewhat high.
>

Yes, you are correct that in scenarios like this an attacker can pay to
degrade the reputation of one of its peers (or even nodes further away).
The key point is that doing so should be costly to the attacker because
they must pay the victim node to continue making itself vulnerable to
payment delays. But if the node is getting compensated, is that really an
attack then? This system is designed with the assumption that the best way
to defend an anonymous/decentralized network that allows sybils is by
pricing resource utilization appropriately. In a similar way, the Bitcoin
blockchain is "vulnerable" to spam attacks in the sense that attackers can
pay to fill up block space.

> As mentioned, the CLTV total leaks information on how far the payee is. I
> might decide to keep track of a reputation score, not of the local peers I
> have, but on the entire network. If the CLTV total at my outgoing is high,
> then if the outgoing HTLC takes a long time to respond, I will distribute a
> small reputation loss to a large number of nodes that are accessible from
> the outgoing channel; if the CLTV total at my outgoing is low, I will
> distribute a large reputation loss to a small number of nodes that are
> accessible from the outgoing channel. I now have the incentive to make
> this estimation even more accurate in the future.
>

One could do this today. I'd even argue that they are incentivized to
already as a protection against loop attacks/payment delays. But it's
likely a pretty ineffective strategy depending on the number of channels
that the possible downstream hops have open.

Please describe the below:
>
> 1. Behavior if payment succeeds after T time.
> 2. Behavior if payment fails after T time.
>
> It seems you only described "Behavior if payment succeeds after T time".
>

Ah, sorry if I didn't make that clear. The reputation is increased in the
case of successful payments by the fee collected. The reputation is
decreased on the downstream peer proportional to time T *regardless* of
whether the payment succeeds or fails. If a payment succeeds quickly, the
increase should outweigh the decrease, but if the payment succeeds after a
long time, the change in reputation may be net negative. If the payment
fails, the upstream peer's reputation does not change and the downstream
peer's reputation always decreases proportional to time T.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180515/755f6eaf/attachment-0001.html>;

📅 Original date posted:2018-05-09 📝 Original message: One ...

2023-06-09T14:50:25+02:00

In reply to nevent1q…3flk
_________________________

📅 Original date posted:2018-05-09
📝 Original message:
One more point in terms of information leakage is that noise can be added
to the "this is the rate that you'll lose reputation at" field to help
obfuscate the number of upstream hops. I proposed setting it to "this is
the upstream rate that I'm losing reputation at" + downstream HTLC value,
but a node can decide to add noise. If they make it too low however,
there's a risk of insufficiently punishing bad nodes and if they make it
too high, there's a heightened risk that the payment fails because the
downstream reputation is insufficient along the route.

This is why I say it's kind of symmetric to the CLTV value: if the delta is
too low, there's risk of loss of funds, if the delta is too high, someone
might decide to fail the payment instead of taking the delay risk.

On Wed, May 9, 2018 at 10:23 AM, Jim Posen <jim.posen at gmail.com> wrote:

> Thanks for the thoughtful responses.
>
> > You missed the vital detail: that you must prove channel closure if you
> > can't unpeel the onion further. That *will* hit an unresponsive party
> > with a penalty.
>
> Ah, that is a good point. I still find the proposal overall worryingly
> complex in terms of communication overhead, time it takes to prove channel
> closure, all of your points in [1], [2], [3], etc. Furthermore, this
> mandates that immediate channel closure is the only allowed reaction to a
> party delaying an HTLC for a time period above a threshold -- the node
> reputation approach gives more discretion to the preceding hop.
> Deobfuscating the route may turn out to be the right option, but I think
> the reputation system has certain advantages over this.
>
> > The models we tried in Milan all created an incentive to fail payments,
> > which is a non-starter.
>
> Do you mind elaborating or summarizing the reasons? The way I'm analyzing
> it, even if there's a nominal spam fee paid to routing nodes that fail
> payments, as long as it's low enough (say 2-5% for arguments sake), the
> nodes still have more to gain by forwarding the payment and earning the
> full fee on a completed payment, and possibly the reputation boost
> associated with completing a payment if that system was in effect.
> Moreover, a node that constantly fails payments will be blacklisted by the
> sender eventually and stop receiving HTLCs from them at all. Overall, I
> don't think this is a profitable strategy. Furthermore, I think it works
> quite well in combination with the reputation system.
>
> > This seems like we'd need some serious evaluation to show that this
> > works, because the risks are very high.
>
> I agree that it needs to be evaluated. I may start working on some network
> simulations to test various DOS mitigation strategies.
>
> > I can destroy your node's reputation by routing crap through you; even
> > if it costs me marginaly more reputation than it does you, that just
> > means that the largest players can force failure upon smaller players,
> > centralizing the network. And I think trying to ensure that it costs me
> > more reputation than the sum of downstream reputation loss leaks too
> > much information
>
> I will add to ZmnSCPxj's response, which is mostly on point. The key here
> is that the only way to lose significant reputation is to delay a payment
> yourself or forward to a malicious downstream that delays -- neither of
> these can be forced by the sender alone. This amounts to a system where you
> are on the hook for any malicious behavior of your downstream peers, which
> is why you must keep a reputation score for each which they earn over time.
> This should keep all links in the network high quality and quickly
> disconnect off delaying nodes if the incentives are right.
>
> While I agree that a lot of reputation is leaked by aggregating the losses
> along the route, this serves exactly to prevent large nodes with high
> reputation from ruining links elsewhere. There are two things a node
> looking to cause reputation loss could do. 1) Identify a node (not itself)
> it thinks will delay a payment and send to them. This locks up funds on
> their behalf, but is actually good behavior because it identifies a faulty
> node and rightfully forces a loss in their reputation, eventually resulting
> in them being booted from the network. Everyone upstream loses some
> reputation for having connectivity to them, but less because of the loss
> aggregation along the route. 2) Delay a payment oneself and force upstream
> reputation loss. This is why I think it's important that the reputation
> loss aggregate so that the malicious party loses the most.
>
> As for the amount of information leaked, yes, it helps determine the
> number of upstream hops in a route. However, the CLTV values help determine
> the number of downstream hops in a route in exactly the same way. I see
> these as symmetric in a sense.
>
> To address ZmnSCPxj's point:
>
> > But it also looks more and more like a policy of "just
> `update_htlc_fail`" keeps our reputation high: indeed never accepting a
> forwarding attempt would ensure reputation.
> > However, earning via fees should help provide incentive against "Just
> `update_htlc_fail`" always. If the goal is "how do I earn money fastest"
> then there is some optimal threshhold > of risk-of-reputation-loss vs.
> fee-earnings-if-I-forward that is unlikely to be near the "Just fail it"
> spectrum, but somewhere in between. We hope.
>
> This is exactly the question that your local view of peer reputations
> helps solve: are the potential fees here worth the risk of forwarding this
> payment to this downstream? If their reputation is high, then you will want
> to forward because you think there's a low chance of you incurring
> reputation loss. If their reputation is low and the HTLC value is too high,
> you will fail it. So I disagree that "just `update_htlc_fail`" is an
> optimal strategy. Consider as well that all fees you earn on successful
> payments are profit to you as well as a reputation boost in the view of
> both of your peers. So in order to earn reputation, you have to forward
> payments. The key is not forwarding through malicious peers.
>
> -jimpo
>
> On Wed, May 9, 2018 at 12:31 AM, ZmnSCPxj <ZmnSCPxj at protonmail.com> wrote:
>
>> Good morning Rusty, Jim, and list,
>>
>> > I can destroy your node's reputation by routing crap through you; even
>> >
>> > if it costs me marginaly more reputation than it does you, that just
>> >
>> > means that the largest players can force failure upon smaller players,
>> >
>> > centralizing the network.
>>
>> My understanding of the proposal was that reputation loss would occur
>> only if the reply (`update_htlc_fail` or `update_htlc_success`) is delayed;
>> this means that for you to force me to lose reputation, you need to somehow
>> make me delay my reply. In particular if you do simple things like give me
>> an invalid onion, or make me forward to a payee who does not know the
>> preimage, I do not lose reputation by replying very quickly with an
>> `update_htlc_fail`.
>>
>> Of course, a large player could force reputation loss by delaying reply
>> when they receive, and having patsy nodes route to them. So for instance
>> if it is Jim -> ZmnSCPxj -> Rusty, and Rusty activates the
>> Blockstream-takes-over-the-world Apocalypse program, the Rusty node
>> would then delay for a long time before replying, which makes my reputation
>> suffer. But it also makes Rusty reputation suffer even more and my
>> reaction would be that, the next time Jim hands me an HTLC that forwards to
>> Rusty, I would instead quickly `update_htlc_fail` back to Jim (which does
>> not lose me significant reputation due to my quick response) than risk
>> forwarding it to you, since you have a reputation for being slow and
>> unresponsive.
>>
>> Indeed, another aspect of Jim proposal is that it is extremely local: if
>> Jim has no channel to Rusty, then Jim has no opinion about Rusty, only
>> about ZmnSCPxj. However, ZmnSCPxj does have an opinion about Rusty, as
>> ZmnSCPxj has channel with Rusty. If I suffer too much reputation loss due
>> to Rusty, my opinion of Rusty drops even faster, and I decide to
>> `update_htlc_fail` in order to prevent Jim opinion of me from dropping too
>> much that Jim decides not to forward to me (if I have other channels with
>> more reasonable nodes).
>>
>> But it also looks more and more like a policy of "just
>> `update_htlc_fail`" keeps our reputation high: indeed never accepting a
>> forwarding attempt would ensure reputation.
>>
>> However, earning via fees should help provide incentive against "Just
>> `update_htlc_fail`" always. If the goal is "how do I earn money fastest"
>> then there is some optimal threshhold of risk-of-reputation-loss vs.
>> fee-earnings-if-I-forward that is unlikely to be near the "Just fail it"
>> spectrum, but somewhere in between. We hope.
>>
>> > And I think trying to ensure that it costs me
>> >
>> > more reputation than the sum of downstream reputation loss leaks too
>> >
>> > much information
>>
>> Yes, this is a major drawback of the proposal. The rate at which the
>> sender of the HTLC threatens me with reputation loss lets me estimate my
>> distance from the ultimate sender of the funds.
>>
>> Regards,
>> ZmnSCPxj
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180509/1894d9e0/attachment-0001.html>;

📅 Original date posted:2018-05-09 📝 Original message: ...

2023-06-09T14:50:24+02:00

In reply to nevent1q…cswt
_________________________

📅 Original date posted:2018-05-09
📝 Original message:
Thanks for the thoughtful responses.

> You missed the vital detail: that you must prove channel closure if you
> can't unpeel the onion further. That *will* hit an unresponsive party
> with a penalty.

Ah, that is a good point. I still find the proposal overall worryingly
complex in terms of communication overhead, time it takes to prove channel
closure, all of your points in [1], [2], [3], etc. Furthermore, this
mandates that immediate channel closure is the only allowed reaction to a
party delaying an HTLC for a time period above a threshold -- the node
reputation approach gives more discretion to the preceding hop.
Deobfuscating the route may turn out to be the right option, but I think
the reputation system has certain advantages over this.

> The models we tried in Milan all created an incentive to fail payments,
> which is a non-starter.

Do you mind elaborating or summarizing the reasons? The way I'm analyzing
it, even if there's a nominal spam fee paid to routing nodes that fail
payments, as long as it's low enough (say 2-5% for arguments sake), the
nodes still have more to gain by forwarding the payment and earning the
full fee on a completed payment, and possibly the reputation boost
associated with completing a payment if that system was in effect.
Moreover, a node that constantly fails payments will be blacklisted by the
sender eventually and stop receiving HTLCs from them at all. Overall, I
don't think this is a profitable strategy. Furthermore, I think it works
quite well in combination with the reputation system.

> This seems like we'd need some serious evaluation to show that this
> works, because the risks are very high.

I agree that it needs to be evaluated. I may start working on some network
simulations to test various DOS mitigation strategies.

> I can destroy your node's reputation by routing crap through you; even
> if it costs me marginaly more reputation than it does you, that just
> means that the largest players can force failure upon smaller players,
> centralizing the network. And I think trying to ensure that it costs me
> more reputation than the sum of downstream reputation loss leaks too
> much information

I will add to ZmnSCPxj's response, which is mostly on point. The key here
is that the only way to lose significant reputation is to delay a payment
yourself or forward to a malicious downstream that delays -- neither of
these can be forced by the sender alone. This amounts to a system where you
are on the hook for any malicious behavior of your downstream peers, which
is why you must keep a reputation score for each which they earn over time.
This should keep all links in the network high quality and quickly
disconnect off delaying nodes if the incentives are right.

While I agree that a lot of reputation is leaked by aggregating the losses
along the route, this serves exactly to prevent large nodes with high
reputation from ruining links elsewhere. There are two things a node
looking to cause reputation loss could do. 1) Identify a node (not itself)
it thinks will delay a payment and send to them. This locks up funds on
their behalf, but is actually good behavior because it identifies a faulty
node and rightfully forces a loss in their reputation, eventually resulting
in them being booted from the network. Everyone upstream loses some
reputation for having connectivity to them, but less because of the loss
aggregation along the route. 2) Delay a payment oneself and force upstream
reputation loss. This is why I think it's important that the reputation
loss aggregate so that the malicious party loses the most.

As for the amount of information leaked, yes, it helps determine the number
of upstream hops in a route. However, the CLTV values help determine the
number of downstream hops in a route in exactly the same way. I see these
as symmetric in a sense.

To address ZmnSCPxj's point:

> But it also looks more and more like a policy of "just
`update_htlc_fail`" keeps our reputation high: indeed never accepting a
forwarding attempt would ensure reputation.
> However, earning via fees should help provide incentive against "Just
`update_htlc_fail`" always. If the goal is "how do I earn money fastest"
then there is some optimal threshhold > of risk-of-reputation-loss vs.
fee-earnings-if-I-forward that is unlikely to be near the "Just fail it"
spectrum, but somewhere in between. We hope.

This is exactly the question that your local view of peer reputations helps
solve: are the potential fees here worth the risk of forwarding this
payment to this downstream? If their reputation is high, then you will want
to forward because you think there's a low chance of you incurring
reputation loss. If their reputation is low and the HTLC value is too high,
you will fail it. So I disagree that "just `update_htlc_fail`" is an
optimal strategy. Consider as well that all fees you earn on successful
payments are profit to you as well as a reputation boost in the view of
both of your peers. So in order to earn reputation, you have to forward
payments. The key is not forwarding through malicious peers.

-jimpo

On Wed, May 9, 2018 at 12:31 AM, ZmnSCPxj <ZmnSCPxj at protonmail.com> wrote:

> Good morning Rusty, Jim, and list,
>
> > I can destroy your node's reputation by routing crap through you; even
> >
> > if it costs me marginaly more reputation than it does you, that just
> >
> > means that the largest players can force failure upon smaller players,
> >
> > centralizing the network.
>
> My understanding of the proposal was that reputation loss would occur only
> if the reply (`update_htlc_fail` or `update_htlc_success`) is delayed; this
> means that for you to force me to lose reputation, you need to somehow make
> me delay my reply. In particular if you do simple things like give me an
> invalid onion, or make me forward to a payee who does not know the
> preimage, I do not lose reputation by replying very quickly with an
> `update_htlc_fail`.
>
> Of course, a large player could force reputation loss by delaying reply
> when they receive, and having patsy nodes route to them. So for instance
> if it is Jim -> ZmnSCPxj -> Rusty, and Rusty activates the
> Blockstream-takes-over-the-world Apocalypse program, the Rusty node would
> then delay for a long time before replying, which makes my reputation
> suffer. But it also makes Rusty reputation suffer even more and my
> reaction would be that, the next time Jim hands me an HTLC that forwards to
> Rusty, I would instead quickly `update_htlc_fail` back to Jim (which does
> not lose me significant reputation due to my quick response) than risk
> forwarding it to you, since you have a reputation for being slow and
> unresponsive.
>
> Indeed, another aspect of Jim proposal is that it is extremely local: if
> Jim has no channel to Rusty, then Jim has no opinion about Rusty, only
> about ZmnSCPxj. However, ZmnSCPxj does have an opinion about Rusty, as
> ZmnSCPxj has channel with Rusty. If I suffer too much reputation loss due
> to Rusty, my opinion of Rusty drops even faster, and I decide to
> `update_htlc_fail` in order to prevent Jim opinion of me from dropping too
> much that Jim decides not to forward to me (if I have other channels with
> more reasonable nodes).
>
> But it also looks more and more like a policy of "just `update_htlc_fail`"
> keeps our reputation high: indeed never accepting a forwarding attempt
> would ensure reputation.
>
> However, earning via fees should help provide incentive against "Just
> `update_htlc_fail`" always. If the goal is "how do I earn money fastest"
> then there is some optimal threshhold of risk-of-reputation-loss vs.
> fee-earnings-if-I-forward that is unlikely to be near the "Just fail it"
> spectrum, but somewhere in between. We hope.
>
> > And I think trying to ensure that it costs me
> >
> > more reputation than the sum of downstream reputation loss leaks too
> >
> > much information
>
> Yes, this is a major drawback of the proposal. The rate at which the
> sender of the HTLC threatens me with reputation loss lets me estimate my
> distance from the ultimate sender of the funds.
>
> Regards,
> ZmnSCPxj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180509/1cec1f17/attachment-0001.html>;

📅 Original date posted:2018-05-02 📝 Original message: I ...

2023-06-09T14:50:23+02:00

In reply to nevent1q…6gdh
_________________________

📅 Original date posted:2018-05-02
📝 Original message:
I have been thinking a lot lately about attacks on lightning routing nodes,
the worst of which is the so-called Loop Attack [1]. See the mailing list
thread for more details, but the basic idea is that a sender and receiver
collude to create a long circuit and refuse to settle or fail the HTLC at
the end until the last possible moment. The attack is particularly
insidious for a few reasons. Firstly, an attack on a multi-hop route gives
the attacker leverage, so that their funds locked in one channel cause
funds to be locked up in every channel on the route. Second, the onion
routing makes it difficult to attribute blame to a single node and defend
against a repeated attack.

First, I'll note that the case where a sender and recipient collude is a
more active variant on an attack that any node in a route can perform. Even
if a node does not initiate the payment, it can just accept offered HTLCs
in the route, not forward them, and wait until the offered HTLC almost
expires before failing. As noted above, the prior node in the route cannot
determine whether the attacking node is at fault or one downstream from it.
Even the sender may not be able to determine where in the route the payment
failed because of the issue where the obfuscated error reason is not MAC'ed
on every hop.

There are two directions of solutions I have heard: 1) protocol support for
decrypting the onion route if the HTLC is kept in-flight for too long 2)
requiring fees even if the payment fails as a cost to the attacker 3) some
sort of reputation system for nodes.

Option 1 I'm afraid may be quite complex. Say this mechanism kicks in and
all nodes in the circuit deobfuscate the route and are able to see the
delays at each hop. The outcome we hope for is that there is one node
clearly to blame and the prior hop in the route fails all channels with
them. However, the attacker can of course control multiple successive hops
in the route, one that looks innocent in front of one that looks guilty,
then keep the channel alive and try again. So then all nodes need to keep a
record of the full circuits and iteratively shift blame up the chain if bad
HTLCs keep going through those channels.

Option 2 is also problematic because it only protects against the case
where the sender is colluding with the receiver, and not where a routing
node is opportunistically delaying payments. This would, however, likely be
successful against nodes being annoying and sending tons of payments with
randomly generated payment hashes in order to "ping" a circuit.

Option 3 has become my preferred solution. The idea is that that for each
node that one has channels with, it only forwards payments through them if
they have a good history, otherwise it fails the payment. Notably, we
ignore whether the downstream hop is directly responsible for delaying a
payment or whether they are simply willing to forward to another node that
is intentionally delaying -- both should be considered bad behavior. In my
opinion, this type of solution fits best into the Lightning model of
independent, linked channels where each node has private contracts with its
direct peers. It also is the simplest in the context of onion routing
because if you are offered an HTLC to route, the only decision you can make
is whether to forward it or fail it based on the amount, previous hop, and
next hop. When I refer to "reputation" hereafter, I do not mean a global
reputation that is gossiped about -- just a local view of a peer's history.

For a Sybil-resistant reputation system, we can use money as a way of
measuring reputation. Fees collected through routing payments across
channels raise the reputation of the channel peer. You lose reputation by
accepting an HTLC and not failing or fulfilling it quickly, proportional to
the value of the HTLC and time to respond. What this essentially means is
that if an attacker wants people to send them HTLCs to delay, they pay a
certain amount in fees over time. So each node will track for each peer 1)
total fees collected from forwarding on routes where that peer is the prior
hop 2) total fees collect from forwarding on routes where that peer is the
next hop 3) total value times time of money locked in offered HTLCs on
channels to the peer. To track the third, as soon as an offered HTLC is
locked in by the peer, the node starts a timer. When the HTLC is settled or
failed or handled on-chain, the timer stops and the node registers the
total elapsed time multiplied by the value of the HTLC.

A very simple strategy would be to have two factors, R_inbound and
R_outbound and calculate reputation per peer as R_inbound * total inbound
fees + R_outbound * total outbound fees - total Bitcoin-seconds locked in
an HTLC. When forwarding a payment, you calculate the worst case reputation
loss, HTLC value * (CLTV - current block height) * 10 min / block, and fail
the payment if that value is greater than their current score. Effectively,
reputation scores are the maximum number of Bitcoin-seconds on could waste
in HTLCs as the downstream node in a channel. If you think, for example,
having 1 BTC for 1 hour is worth 1 satoshi, you might set R_inbound =
R_outbound = 10^8. Honest nodes should earn reputation passively by
forwarding payments and failing/fulfilling quickly, while an attacker would
lose reputation and have to pay fees to re-earn it. Furthermore, high
reputation nodes should be able to earn reputation faster since peers will
be willing to forward higher value HTLCs through them.

We want an additional property from this system though, which is that an
attacker be limited in their ability to disrupt network connectivity and
induce throttling on other channels. The above scheme has the problem that
for every unit of reputation the attacker loses, each upstream hop also
loses approximately the same amount -- they effectively get leverage on the
ability to reduce reputation. So instead, consider a modification: each hop
loses a quantity of reputation units equal to the *total* amount of
Bitcoin-seconds lost upstream. So say there's a payment A -> B -> C -> D,
where the CLTV decrements by 12 and the amount decrements by 10 satoshis at
each hop, with a terminal CLTV of 12 and a terminal amount of 100. So A
tells B "Here's an HTLC for 120 satoshis, CLTV is in 36 blocks, and you
will lose reputation at a rate of 120 satoshi-seconds per second of delay".
B calculates the maximum he could lose by forwarding to C as 120 * 36
blocks * 10 min / block. Assuming C has a high enough reputation score, B
will offer an HTLC of 110 satoshis, CLTV of 24 blocks, reputation rate of
120 + 110 = 230 satoshi-seconds per second. And finally C requires D to
stake reputation at a rate of 230 + 100 = 330 satoshi-seconds / second. The
effect is that if any node withholds the preimage, they lose an amount of
reputation equal to all of the reputation lost upstream combined, removing
any attack leverage.

This strategy has some downsides, namely in terms of centralization and
privacy. Telling the downstream node the rate at which they must stake
reputation may allow them to determine the number of upstream hops in the
circuit. To obfuscate this, the sender may require an addition reputation
stake, but takes a higher risk of the payment failing somewhere downstream.
The more obvious issue is centralization -- if payments are getting
throttled by reputation, there is even more of an incentive to route
through a small number of channels and decrease the length of payment
circuits. While this is a concern, the degree to while throttling happens
in controlled by the R parameters (the ratio at which fees contribute to
reputation). If R is high and reputation is abundant, then there will be
little throttling. But if a node finds itself under attack, it can raise
its R parameters and start throttling bad peers.

In summary, I think a simple local reputation mechanism of this form could
be implemented with only minor changes to the BOLT spec and could provide
decent resistance against DOS/loop attacks. Sorry for the excessively long
email.

[1] https://lists.linuxfoundation.org/pipermail/lightning-dev/2015-August/
000135.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180501/35997f70/attachment.html>;

📅 Original date posted:2018-02-08 📝 Original message: If ...

2023-06-09T14:49:00+02:00

In reply to nevent1q…xk0t
_________________________

📅 Original date posted:2018-02-08
📝 Original message:
If using two hashes to deliver the payment while still getting a proof, I'm
not sure what that provides above just sending regular lightning payments
over multiple routes with one hash. Firstly, if there is a second hash, it
would presumably be the same for all routes, making them linkable again,
which AMP tries to solve. And secondly, the receiver has no incentive to
claim any of the HTLCs before all of them are locked in, because in that
case they are releasing the transaction receipt before fully being paid.

On Thu, Feb 8, 2018 at 8:41 AM, Johan Torås Halseth <johanth at gmail.com>
wrote:

> An obvious way to make this compatible with proof-of-payment would be to
> require two hashes to claim the HTLC: the presage from the invoice payment
> hash (as today) + the new hash introduced here. This would give the sender
> a receipt after only one of the HTLCs was claimed. Would require changes to
> the scripts of course.
>
> With Schnorr/EC operations this could probably be made more elegant, as
> mentioned.
>
> - Johan
> On Wed, Feb 7, 2018 at 18:21, Rusty Russell <rusty at rustcorp.com.au> wrote:
>
> Olaoluwa Osuntokun <laolu32 at gmail.com> writes:
> > Hi Y'all,
> >
> > A common question I've seen concerning Lightning is: "I have five $2
> > channels, is it possible for me to *atomically* send $6 to fulfill a
> > payment?". The answer to this question is "yes", provided that the
> receiver
>
> This is awesome! I'm kicking myself for not proposing it :)
>
> Unfortunately, your proposal defines a way to make multipath donations,
> not multipath payments :(
>
> In other words, you've lost proof of payment, which IMHO is critical.
>
> Fortunately, this can be fairly trivially fixed when we go to scriptless
> scripts or other equivalent decorrelation mechanism, when I think this
> mechanism becomes extremely powerful.
>
> > - Potential fee savings for larger payments, contingent on there being a
> > super-linear component to routed fees. It's possible that with
> > modifications to the fee schedule, it's actually *cheaper* to send
> > payments over multiple flows rather than one giant flow.
>
> This is a stretch. I'd stick with the increased reliability/privacy
> arguments which are overwhelmingly compelling IMHO.
>
> If I have any important feedback on deeper reading (and after a sccond
> coffee), I'll send a separate email.
>
> Thanks!
> Rusty.
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180208/31041639/attachment.html>;

📅 Original date posted:2018-02-07 📝 Original message: This ...

2023-06-09T14:48:59+02:00

In reply to nevent1q…x56u
_________________________

📅 Original date posted:2018-02-07
📝 Original message:
This is a really neat idea.

This is a question about non-interactive payments in general, but is there
any way to get a proof of payment? With regular invoices, knowledge of the
preimage serves as cryptographic proof that the payment was delivered.

On Feb 6, 2018 6:26 PM, "Conner Fromknecht" <conner at lightning.engineering>
wrote:

>
> Hi ZmnSCPxj and Laolu,
>
> > Indeed, the existence of per-hop fees (`fee_base_msat`) means,
> splitting the
> > payment over multiple flows will be, very likely, more expensive,
> compared to
> > using a single flow.
>
> As Laolu pointed out, we have yet to see how fees evolve on mainnet or
> what will
> emerge as a sane, default fee schedules. I agree that if the same
> proportional
> fee is used across all partial payments, then it could certainly be more
> expensive.
>
> However, it could also be the case that you were paying a needlessly high
> proportional fee to begin with, because paths of sufficient capacity to the
> destination were scarce. In an AMP world, there will be an abundance of
> channels
> that can route small, partial payments, which may itself drive down the
> competitive fee rate for smaller payments. Just a hypothesis, we shall see
> where
> supply meets demand!
>
> At the end of the day, the user can always fall back to regular payment if
> they
> expect to end up paying more fees using an AMP.
>
> > (If we want to support multiple routes converging to an intermediate
> node,
> > then continue routing to a different final node after routes have merged
> (i.e.
> > A->B->C->D, and A->E->C->D, with the payment being merged by C, who
> forwards
> > the combination to D), then we need to follow the current hop data
> format, but
> > I think supporting AMP at final payees is actually enough...
>
> I think this is an interesting idea, sounds maybe like a
> recursive/hierarchical
> AMP? The ability to merge the payments seems like it would result in a
> decent privacy
> leak, as I believe an intermediary would have enough evidence to prove
> that two
> payments were merged/correlated. Simple traffic analysis would also reveal
> a
> discrepancy in the number of incoming and outgoing packets, and possibly
> other
> observable differences in routing (some) AMPs vs regular payments.
>
> FWIW the current proposal allows the paths of partial payments to overlap,
> in such a scenario C would just forward the HTLCs independently. One could
> send
> them all along the same path if they desired! I'm assuming the intent here
> is to
> try and reduce total fees?
>
> Minor correction^2:
>
> > This should actually be (H(s_0 || s_1 || ...), s_i).
>
> This assumes the receiver knows the indexes of each share. Without this
> knowledge they would have to brute force all orderings to check the
> fingerprint.
>
> To maintain order invariance on the receiving end, I would propose sending
> (0, s_i) for the first n-1 partial payments, and then (n, s_i) on the
> final one.
> As in the description of the basic AMP scheme, the receiver maintains a
> persistent count of how many partial payments have been received for ID.
> If the
> receiver does not get the last payment last, the receiver just waits until
> all n
> have been received before deciding that its reconstructed value is BP.
>
> The receiver can verify they've received the correct BP and n by
> rederiving the
> partial preimages r_i = H(BP || i) and checking that there are n
> outstanding
> payments, one for each h_i = H(r_i). This also saves the receiving node n
> additional hash invocations.
>
> -Conner
>
> On Tue, Feb 6, 2018 at 4:04 PM Olaoluwa Osuntokun <laolu32 at gmail.com>
> wrote:
>
>> Hi ZmnSCPxj,
>>
>> > This is excellent work!
>>
>> Thanks!
>>
>> > I think, a `globalfeatures` odd bit could be used for this. As it is
>> > end-ot-end, `localfeatures` is not appropriate.
>>
>> Yep, it would need to be a global feature bit. In the case that we're
>> sending to a destination which isn't publicly advertised, then perhaps an
>> extension to BOLT-11 could be made to signal receiver support.
>>
>> > I believe, currently, fees have not this super-linear component
>>
>> Yep they don't. Arguably, we should also have a component that scales
>> according to the proposed CLTV value of the outgoing HTLC. At Scaling
>> Bitcoin Stanford, Aviv Zohar gave a talked titled "How to Charge
>> Lightning"
>> where the authors analyzed the possible evolution of fees on the network
>> (and also suggested adding this super-linear component to extend the
>> lifetime of channels). However, the talk itself focused on a very simple
>> "mega super duper hub" topology. Towards the end he alluded to a
>> forthcoming
>> paper that had more comprehensive analysis of more complex topologies. I
>> look forward to the publication of their finalized work.
>>
>> > Indeed, the existence of per-hop fees (`fee_base_msat`) means, splitting
>> > the payment over multiple flows will be, very likely, more expensive,
>> > compared to using a single flow.
>>
>> Well it's still to be seen how the fee structure on mainnet emerges once
>> the
>> network is still fully bootstrapped. AFAIK, most running on mainnet atm
>> are
>> using the default fee schedules for their respective implementations. For
>> example, the default fee_base_msat for lnd is 1000 msat (1 satoshi).
>>
>> > I believe the `realm` byte is intended for this.
>>
>> The realm byte is meant to signal "forward this to the dogecoin channel".
>> ATM, we just default to 0 as "Bitcoin". However, the byte itself only
>> really
>> need significance between the sender and the intermediate node. So there
>> isn't necessarily pressure to have a globally synchronized set of realm
>> bytes.
>>
>> > Thus, you can route over nodes that are unaware of AMP, and only provide
>> > an AMP realm byte to the destination node, who, is able to reconstruct
>> this
>> > your AMP data as per your algorithm.
>>
>> Yes, the intermediate nodes don't need to be aware of the end-to-end
>> protocol. For the final hop, there are actually 53 free bytes (before one
>> needs to signal the existence of EOBs):
>>
>> * 1 byte realm
>> * 8 bytes next addr (all zeroes to signal final dest)
>> * 32 bytes hmac (also all zeroes for the final dest)
>> * 12 bytes padding
>>
>> So any combo of these bytes can be used to signal more advanced protocols
>> to
>> the final destination.
>>
>>
>> A correction from the prior email description:
>>
>> > We can further modify our usage of the per-hop payloads to send
>> > (H(BP), s_i) to consume most of the EOB sent from sender to receiver.
>>
>> This should actually be (H(s_0 || s_1 || ...), s_i). So we still allow
>> them
>> to check this finger print to see if they have all the final shares, but
>> don't allow them to preemptively pull all the payments.
>>
>>
>> -- Laolu
>>
>>
>> On Mon, Feb 5, 2018 at 11:12 PM ZmnSCPxj <ZmnSCPxj at protonmail.com> wrote:
>>
>>> Good morning Laolu,
>>>
>>> This is excellent work!
>>>
>>> Some minor comments...
>>>
>>>
>>> (Atomic Multi-path Payments). It can be experimented with on Lightning
>>> *today* with the addition of a new feature bit to gate this new
>>> feature. The beauty of the scheme is that it requires no fundamental
>>> changes
>>> to the protocol as is now, as the negotiation is strictly *end-to-end*
>>> between sender and receiver.
>>>
>>>
>>> I think, a `globalfeatures` odd bit could be used for this. As it is
>>> end-ot-end, `localfeatures` is not appropriate.
>>>
>>> - Potential fee savings for larger payments, contingent on there being
>>> a
>>> super-linear component to routed fees. It's possible that with
>>> modifications to the fee schedule, it's actually *cheaper* to send
>>> payments over multiple flows rather than one giant flow.
>>>
>>>
>>> I believe, currently, fees have not this super-linear component.
>>> Indeed, the existence of per-hop fees (`fee_base_msat`) means, splitting
>>> the payment over multiple flows will be, very likely, more expensive,
>>> compared to using a single flow. Tiny roundoffs in computing the
>>> proportional fees (`fee_proportional_millionths`) may make smaller
>>> flows give a slight fee advantage, but I think the multiplication of
>>> per-hop fees will dominate.
>>>
>>>
>>> - Using smaller payments increases the set of possible paths a partial
>>> payment could have taken, which reduces the effectiveness of static
>>> analysis techniques involving channel capacities and the plaintext
>>> values being forwarded.
>>>
>>>
>>> Strongly agree!
>>>
>>>
>>> In order to include the three tuple within the per-hop payload for the
>>> final
>>> destination, we repurpose the _first_ byte of the un-used padding bytes
>>> in
>>> the payload to signal version 0x01 of the AMP protocol (note this is a
>>> PoC
>>> outline, we would need to standardize signalling of these 12 bytes to
>>> support other protocols).
>>>
>>>
>>> I believe the `realm` byte is intended for this. Intermediate nodes do
>>> not need to understand realm bytes that are understood by other nodes in
>>> the route, including the realm bytes understood by the final destination,
>>> as intermediate nodes cannot, indeed, read the hop data of other nodes.
>>> Thus, you can route over nodes that are unaware of AMP, and only provide an
>>> AMP realm byte to the destination node, who, is able to reconstruct this
>>> your AMP data as per your algorithm.
>>>
>>> Indeed, the `realm` byte controls the interpretation of the rest of the
>>> 65-byte packet. If you define, instead, a separate `realm` that is
>>> understood by the destination node, you can redefine the entire 64 bytes of
>>> the final hop data as you wish.
>>>
>>> If we support AMP only at final payees, we can completely redefine the
>>> 64 bytes in the final hop data for the new AMP `realm`, and not consume the
>>> next hop (which would reduce route length by 1).
>>>
>>> (If we want to support multiple routes converging to an intermediate
>>> node, then continue routing to a different final node after routes have
>>> merged (i.e. A->B->C->D, and A->E->C->D, with the payment being merged by
>>> C, who forwards the combination to D), then we need to follow the current
>>> hop data format, but I think supporting AMP at final payees is actually
>>> enough... AMP at intermediate nodes might not be used often enough by
>>> senders for it to matter, as taking advantage of that seems more complex
>>> than just asking your routing algo to provide you multiple routes to a
>>> destination, which you are probably already doing)
>>>
>>> ----
>>>
>>> Overall, good work I think.
>>>
>>> Regards,
>>> ZmnSCPxj
>>>
>> _______________________________________________
>> Lightning-dev mailing list
>> Lightning-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>>
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180207/48a507f8/attachment-0001.html>;

📅 Original date posted:2018-02-07 📝 Original message: I ...

2023-06-09T14:48:51+02:00

In reply to nevent1q…25jv
_________________________

📅 Original date posted:2018-02-07
📝 Original message:
I like Christian's proposal of adding a simple announcement cutoff
timestamp with the intention of designing something more sophisticated
given more time.

I prefer the approach of having an optional feature bit signalling that a
`set_gossip_timestamp` message must be sent immediately after `init`, as
Laolu suggested. This way it doesn't conflict with and other possible
handshake extensions.

On Feb 7, 2018 9:50 AM, "Fabrice Drouin" <fabrice.drouin at acinq.fr> wrote:

Hi,

Suppose you partition nodes into 3 generic roles:
- payers: they mostly send payments, are typically small and operated
by end users, and are offline quite a lot
- relayers: they mostly relay payments, and would be online most of
the time (if they're too unreliable other nodes will eventually close
their channels with them)
- payees: they mostly receive payments, how often they can be online
is directly link to their particular mode of operations (since you
need to be online to receive payments)

Of course most nodes would play more or less all roles. However,
mobile nodes would probably be mostly "payers", and they have specific
properties:
- if they don't relay payments they don't have to be announced. There
could be millions of mobile nodes that would have no impact on the
size of the routing table
- it does not impact the network when they're offline
- but they need an accurate routing table. This is very different from
nodes who mostly relay or accept payements
- they would be connected to a very small number of nodes
- they would typically be online for just a few hours every day, but
could be stopped/paused/restarted many times a day

Laolu wrote:
> So I think the primary distinction between y'alls proposals is that
> cdecker's proposal focuses on eventually synchronizing all the set of
> _updates_, while Fabrice's proposal cares *only* about the newly created
> channels. It only cares about new channels as the rationale is that if
once
> tries to route over a channel with a state channel update for it, then
> you'll get an error with the latest update encapsulated.

If you have one filter per day and they don't match (because your peer
has channels that you missed, or
have been closed and you were not aware of it) then you will receive
all channel announcements for
this particular day, and the associated updates

Laolu wrote:
> I think he's actually proposing just a general update horizon in which
> vertexes+edges with a lower time stamp just shouldn't be set at all. In
the
> case of an old zombie channel which was resurrected, it would eventually
be
> re-propagated as the node on either end of the channel should broadcast a
> fresh update along with the original chan ann.

Yes but it could take a long time. It may be worse on testnet since it
seems that nodes
don't change their fees very often. "Payer nodes" need a good routing
table (as opposed
to "relayers" which could work without one if they never initiate payments)

Laolu wrote:
> This seems to assume that both nodes have a strongly synchronized view of
> the network. Otherwise, they'll fall back to sending everything that went
on
> during the entire epoch regularly. It also doesn't address the zombie
churn
> issue as they may eventually send you very old channels you'll have to
deal
> with (or discard).

Yes I agree that for nodes which have connections to a lot of peers,
strongly synchronized routing tables is
harder to achieve since a small change may invalidate an entire
bucket. Real queryable filters would be much
better, but worst case scenario is we've sent an additionnal 30 Kb or
o of sync messages.
(A very naive filter would be sort + pack all short ids for example)

But we focus on nodes which are connected to a very small number of
peers, and in this particular
case it is not an unrealistic expectation.
We have built a prototype and on testnet it works fairly well. I also
found nodes which have no direct
channel betweem them but produce the same filters for 75% of the
buckets ("produce" here means
that I opened a simple gossip connection to them, got their routing
table and used it to generate filters).

Laolu wrote:
> How far back would this go? Weeks, months, years?
Since forever :)
One filter per day for all annoucements that are older than now - 1
week (modulo 144)
One filter per block for recent announcements

>
> FWIW this approach optimizes for just learning of new channels instead of
> learning of the freshest state you haven't yet seen.

I'd say it optimizes the case where you are connected to very few
peers, and are online a few times every day (?)

>
> -- Laolu
>
>
> On Mon, Feb 5, 2018 at 7:08 AM Fabrice Drouin <fabrice.drouin at acinq.fr>
> wrote:
>>
>> Hi,
>>
>> On 5 February 2018 at 14:02, Christian Decker
>> <decker.christian at gmail.com> wrote:
>> > Hi everyone
>> >
>> > The feature bit is even, meaning that it is required from the peer,
>> > since we extend the `init` message itself, and a peer that does not
>> > support this feature would be unable to parse any future extensions to
>> > the `init` message. Alternatively we could create a new
>> > `set_gossip_timestamp` message that is only sent if both endpoints
>> > support this proposal, but that could result in duplicate messages
being
>> > delivered between the `init` and the `set_gossip_timestamp` message and
>> > it'd require additional messages.
>>
>> We chose the other aproach and propose to use an optional feature
>>
>> > The reason I'm using timestamp and not the blockheight in the short
>> > channel ID is that we already use the timestamp for pruning. In the
>> > blockheight based timestamp we might ignore channels that were created,
>> > then not announced or forgotten, and then later came back and are now
>> > stable.
>>
>> Just to be clear, you propose to use the timestamp of the most recent
>> channel updates to filter
>> the associated channel announcements ?
>>
>> > I hope this rather simple proposal is sufficient to fix the short-term
>> > issues we are facing with the initial sync, while we wait for a real
>> > sync protocol. It is definitely not meant to allow perfect
>> > synchronization of the topology between peers, but then again I don't
>> > believe that is strictly necessary to make the routing successful.
>> >
>> > Please let me know what you think, and I'd love to discuss Pierre's
>> > proposal as well.
>> >
>> > Cheers,
>> > Christian
>>
>> Our idea is to group channel announcements by "buckets", create a
>> filter for each bucket, exchange and use them to filter out channel
>> announcements.
>>
>> We would add a new `use_channel_announcement_filters` optional feature
>> bit (7 for example), and a new `channel_announcement_filters` message.
>>
>> When a node that supports channel announcement filters receives an
>> `init` message with the `use_channel_announcement_filters` bit set, it
>> sends back its channel filters.
>>
>> When a node that supports channel announcement filters receives
>> a`channel_announcement_filters` message, it uses it to filter channel
>> announcements (and, implicitly ,channel updates) before sending them.
>>
>> The filters we have in mind are simple:
>> - Sort announcements by short channel id
>> - Compute a marker height, which is `144 * ((now - 7 * 144) / 144)`
>> (we round to multiples of 144 to make sync easier)
>> - Group channel announcements that were created before this marker by
>> groups of 144 blocks
>> - Group channel announcements that were created after this marker by
>> groups of 1 block
>> - For each group, sort and concatenate all channel announcements short
>> channel ids and hash the result (we could use sha256, or the first 16
>> bytes of the sha256 hash)
>>
>> The new `channel_announcement_filters` would then be a list of
>> (height, hash) pairs ordered by increasing heights.
>>
>> This implies that implementation can easily sort announcements by
>> short channel id, which should not be very difficult.
>> An additional step could be to send all short channel ids for all
>> groups for which the group hash did not match. Alternatively we could
>> use smarter filters
>>
>> The use case we have in mind is mobile nodes, or more generally nodes
>> which are often offline and need to resync very often.
>>
>> Cheers,
>> Fabrice
>> _______________________________________________
>> Lightning-dev mailing list
>> Lightning-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
_______________________________________________
Lightning-dev mailing list
Lightning-dev at lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180207/cb4ed693/attachment-0001.html>;

📅 Original date posted:2019-04-04 📝 Original message:> ...

2023-06-07T20:17:24+02:00

In reply to nevent1q…ewv0
_________________________

📅 Original date posted:2019-04-04
📝 Original message:>
> Learning C++ is something within everyone's capability. Even people who do
> not
> wish to learn it can hire someone to perform review for them.
>

Anyone with enough knowledge of C++ to audit the entire the Bitcoin Core
codebase is more than capable of running it with assumeutxo disabled and
checking the hard-coded vale themself.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190403/fa2a9a58/attachment.html>;

📅 Original date posted:2018-06-10 📝 Original message:I ...

2023-06-07T20:12:55+02:00

In reply to nevent1q…d9zp
_________________________

📅 Original date posted:2018-06-10
📝 Original message:I generally like the direction of this proposal in terms of allowing full
nodes to run with a different storage/bandwidth tradeoff. Cory, were this
implemented, would you expect Core to support both operating modes (full
UTXO set and UHS) depending on user configuration, or would UHS be
mandatory?

Also, given that Bram Cohen's TXO bitfield proposal was an inspiration for
this, could you comment on why the UHS is preferable to that approach? An
alternative that goes even further in the direction of more bandwidth, less
storage, would be for nodes to simply maintain a Merkle Mountain Range over
all TXOs in order of creation and a spentness bitfield. Blocks could be
requested with the prev outputs and a Merkle proof linking them into the
MMR root. Since the Merkle proof is deterministic, it could be computed by
archive nodes and miners and saved alongside the block data for relay.
Another benefit of this is the TXO MMR root may be independently useful if
committed into the coinbase transaction.

On Thu, Jun 7, 2018 at 7:02 AM Sjors Provoost via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> eMMC storage, which low end devices often use, come in 2x increments.
> Running a pruned full node on 8 GB is difficult if not impossible (the UTXO
> set peaked at 3.5 GB in January, but a full node stores additional stuff).
>
> However, 16 GB is only €10 more expensive and presumably standard by the
> time this would be rolled out.
>
> On AWS every GB of SSD storage avoided saves $1 per year, not end of the
> world stuff, but not negligible either. Outbound traffic costs $0.10 / GB
> (ignoring free allowance), so when uploading 200 GB per year, the 5% would
> offset $1 of storage cost savings.
>
> The above seems marginal, probably not worth it unless there’s really no
> downside.
>
> What I find attractive about this proposal is the ability to squeeze more
> out of limited RAM (typically only 1 or 2 GB on these low end devices). I’d
> have to test Cory’s branch to see if that actually matters in practice.
>
> It’s also useful to distinguish benefits during initial sync from ongoing
> operation. The former I’ve almost given up on for low end devices (can
> take weeks), in favor of doing it on a faster computer and copying the
> result. The latter needs far less RAM, so perhaps this proposal doesn’t
> help much there, but that would be useful to measure.
>
> Did you try the recent SHA256 optimizations on your branch?
>
> Sjors
>
> > Op 17 mei 2018, om 18:56 heeft Gregory Maxwell via bitcoin-dev <
> bitcoin-dev at lists.linuxfoundation.org> het volgende geschreven:
> >
> > On Wed, May 16, 2018 at 4:36 PM, Cory Fields via bitcoin-dev
> > <bitcoin-dev at lists.linuxfoundation.org> wrote:
> >> Tl;dr: Rather than storing all unspent outputs, store their hashes.
> >
> > My initial thoughts are it's not _completely_ obvious to me that a 5%
> > ongoing bandwidth increase is actually a win to get something like a
> > 40% reduction in the size of a pruned node (and less than a 1%
> > reduction in an archive node) primarily because I've not seen size of
> > a pruned node cited as a usage limiting factor basically anywhere. I
> > would assume it is a win but wouldn't be shocked to see a careful
> > analysis that concluded it wasn't.
> >
> > But perhaps more interestingly, I think the overhead is not really 5%,
> > but it's 5% measured in the context of the phenomenally inefficient tx
> > mechanisms ( https://bitcointalk.org/index.php?topic=1377345.0 ).
> > Napkin math on the size of a txn alone tells me it's more like a 25%
> > increase if you just consider size of tx vs size of
> > tx+scriptpubkeys,amounts. If I'm not missing something there, I think
> > that would get in into a very clear not-win range.
> >
> > On the positive side is that it doesn't change the blockchain
> > datastructure, so it's something implementations could do without
> > marrying the network to it forever.
> >
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180610/609b21d1/attachment.html>;

📅 Original date posted:2018-06-04 📝 Original message:> ...

2023-06-07T20:12:44+02:00

In reply to nevent1q…d7vw
_________________________

📅 Original date posted:2018-06-04
📝 Original message:>
> I was wondering why this multi-layer multi-block filter proposal isn't
> getting any comment,
> is it because not asking all filters is leaking information?
>

It's an interesting idea, but it adds more complexity to the client and
could be added later on if clients adopt BIP 157 and complain about
bandwidth. It also derives all bandwidth gains from address reuse. So I'm
hesitant to make the complexity tradeoff for bandwidth savings due to a
behavior that is actively discouraged.

On another note, I've been thinking that block TXO commitments could
resolve the issue we are facing now with deciding between the prev script
approach and outpoint. The whole argument for outpoints is that there are
compact-ish (<1 MiB) proofs of filter validity, which is not currently
possible if the filters included prev output data. Such proofs would be
feasible if blocks headers (well, actually coinbase txs) had a commitment
to the Merkle root of all newly created outputs in the block.

This idea has been tossed around before in the context of fraud proofs and
TXO bitfields, and seems to unlock a whole bunch of other P2P commitments.
For example, if we wanted to do P2P commitments (BIP 157-style) to the
distribution of tx fees in a block, one could use block TXO commitments to
prove correctness of fees for non-segwit txs. It also enables block
validity proofs (assuming parent blocks are valid), which are not as
powerful as invalidity/fraud proofs, but interesting nonetheless.

This would require a new getdata type BLOCK_WITH_PREVOUTS or something. I
assume for most coinbase-tx-committed proposals, we'll also need a new
getcoinbases/coinbases that requests the coinbase tx and Merkle branch for
a range of headers as well. But with these additions, we could start
serving more block-derived data to light clients under the BIP 157
at-least-one-honest-peer assumption.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180604/ec43b862/attachment-0001.html>;

📅 Original date posted:2018-06-01 📝 Original message:To ...

2023-06-07T20:12:38+02:00

In reply to nevent1q…4t68
_________________________

📅 Original date posted:2018-06-01
📝 Original message:To address the at-least-one-honest peer security assumption for light
clients, I think this is a rather good security model for light clients.
First it significantly reduces the chances that an attacker can eclipse a
client just by chance, and clients can implement measures like ensuring
connectivity to peers from different subnets. But even if, as you suggest,
a network attacker controls the target's local network, peers still can
have good security guarantees by requiring authenticated connections to
semi-trusted peers. A client can select a set of N servers that it believes
will not collude to attack it, and only sync filters if connected to a
threshold of them. So even if the network is malicious, the attacker cannot
forge the authenticated responses. The level of trust in these designated
parties again is quite low because only one has to be honest. This would
require something like BIP 150.

Even if clients are uncomfortable with whitelisting required peers, it
could have a policy of requiring a certain number of connections to peers
that have honestly served it filters in the past. This is sort of like
trust-on-first-use. This type of scheme, however, would require nodes to
advertise a pubkey per address, which BIP 150/151 does not support at
present.

All in all, I think this is an acceptable security model for light clients.
Without the ability to verify filter validity, a client would have to stop
syncing altogether in the presence of just one malicious peer, which is
unacceptable.

The other concern you raise, Greg, is using a filter for P2P communications
that we expect may be replaced in the future. You also raise the point that
full node wallets can use the smaller filters for rescans because the
filter validity is not in question. I'd perfectly fine with the idea of
defining two filter types in the BIP, one that is output script + outpoint
and the other output script + prev script. But I imagine some people would
object to the idea of full nodes storing two different filters that overlap
in contents. If we had to pick just one though, I'm strongly in support of
output script + outpoint so that BIP 157 can be deployed ASAP without a
consensus change. It's entirely possible we will learn even more about
optimal filter design through deployment and adoption.

On Fri, Jun 1, 2018 at 5:22 PM Gregory Maxwell <greg at xiph.org> wrote:

> On Sat, Jun 2, 2018 at 12:01 AM, Olaoluwa Osuntokun <laolu32 at gmail.com>
> wrote:
> >> A typical network attacker (e.g. someone on your lan or wifi segmet, or
> >> someone who has compromised or operates an upstream router) can be all
> of
> >> your peers.
> >
> > This is true, but it cannot make us accept any invalid filters unless the
> > attacker is also creating invalid blocks w/ valid PoW.
>
> I wish that were the true, but absent commitments that wouldn't be the
> case unless you were always downloading all the blocks-- since you
> wouldn't have any sign that there was something wrong with the
> filter-- and downloading all the blocks would moot using the filters
> in the first place. :)
>
> Or have I misunderstood you massively here?
>
> For segwit originally I had proposed adding additional commitments
> that would make it possible to efficiently prove invalidity of a
> block; but that got stripped because many people were of the view that
> the "assume you have at least one honest peer who saw that block and
> rejected it to tell you that the block was invalid" security
> assumption was of dubious value. Maybe it's more justifiable to make
> use of a dubious assumption for a P2P feature than for a consensus
> feature? Perhaps, I'd rather have both filter types from day one so
> that things not implementing the comparison techniques don't get the
> efficiency loss or the extra work to change filter types for a
> consensus one.
>
> [I think now that we're much closer to a design that would be worth
> making a consensus committed version of than we were a few months ago
> now, since we are effectively already on a second generation of the
> design with the various improvements lately]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180601/b99c927d/attachment-0001.html>;

📅 Original date posted:2018-05-23 📝 Original message:I ...

2023-06-07T20:12:28+02:00

In reply to nevent1q…pw22
_________________________

📅 Original date posted:2018-05-23
📝 Original message:I decided to look into the metrics around compression ratios of TXO
bitfields, as proposed by Bram Cohen [1]. I'm specifically interested in
the feasibility of committing to them with block headers. In combination
with block commitments to TXOs themselves, this would enable UTXO
inclusion/exclusion proofs for light clients.

First, looking just at proofs of inclusion in the UTXO set, each block
needs what Bram calls a "proof of position." Concretely, one such
construction is a Merkle root over all of the block's newly created coins,
including their output data (scriptPubKey + amount), the outpoint (txid +
index), and an absolute index of the output in the entire blockchain. A
Merkle branch in this tree constitutes a proof of position. Alternatively,
the "position", rather than being an absolute index in the chain, could be
a block hash plus an output index within the block.

Let's say we use the absolute index in the chain as position. A TXO
spentness bitfield can be constructed for the entire chain, which is added
to when new coins are created and modified when they are spent. In order to
compactly prove spentness in this bitfield to a client, one could chunk up
the bitfield and construct a Merkle Mountain Range [2] over the chunks.
Instead of building an MMR over outputs themselves, as proposed by Peter
Todd [3], an MMR constructed over bitfield chunks grows far slower, by a
large constant factor. Slower growth means faster updates.

So there's the question of how much these bitfields can be compressed. We
expect some decent level because patterns of spending coins are very
non-random.

The top graph in the attached figure shows the compression ratios possible
on a TXO bitfield split into 4 KiB chunks, using gzip (level=9) and lz4.
Data was collected at block height 523,303. You can see that the
compression ratio is much lower for older chunks and is worse for more
recent blocks. Over the entire history, gzip achieves 34.4%, lz4 54.8%, and
bz2 37.6%. I'm kind of surprised that the ratios are not lower with
off-the-shelf algorithms. And that gzip performs better than bz2 (it seems
to be a factor of the chunk size?).

Alternatively, we can look at bitfields stored separately by block, which
is more compatible with constructions where an output's position is its
block hash plus relative index. The per-block bitfield sizes are shown in
the bottom graph. The compression ratios overall are 50% for gzip, 70% for
lz4, and 61.5% for bz2.

[1]
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-March/013928.html
[2]
https://github.com/opentimestamps/opentimestamps-server/blob/master/doc/merkle-mountain-range.md
[3] https://petertodd.org/2016/delayed-txo-commitments
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180523/8e1272fd/attachment-0001.html>;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bitfield_sizes.svg
Type: image/svg+xml
Size: 1788184 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180523/8e1272fd/attachment-0001.svg>;

📅 Original date posted:2018-05-28 📝 Original message:> ...

2023-06-07T20:12:17+02:00

In reply to nevent1q…47j8
_________________________

📅 Original date posted:2018-05-28
📝 Original message:> Is there an application that requires watching for output scripts that
> doesn't also require watching for input scrips (or, less efficiently,
> input outpoints)?
>

Certain wallets may be able to use only the output script filter by using
output scripts to watch for confirmations on sent transactions, assuming
that application is the only one with access to the private keys. The
additional benefit of the input script/outpoint filter is to watch for
unexpected spends (coins getting stolen or spent from another wallet) or
transactions without a unique change or output address. I think this is a
reasonable implementation, and it would be nice to be able to download that
filter without any input elements.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180528/8fdc2ff6/attachment.html>;

📅 Original date posted:2018-05-24 📝 Original message:Greg, ...

2023-06-07T20:12:16+02:00

In reply to nevent1q…v27g
_________________________

📅 Original date posted:2018-05-24
📝 Original message:Greg, I've attached a graph including the input scripts.

In the top graph, we can see how the input script filter compares to the
input outpoint filter. It is definitely smaller as a result of address
reuse. The bottom graph shows the ratio over time of combining the input
prev script and output script filters vs keeping them separate. In more
recent blocks, it appears that there are decreasing savings.

On Wed, May 23, 2018 at 6:04 PM Conner Fromknecht
<conner at lightning.engineering> wrote:

> Hi all,
>
> Jimpo, thanks for looking into those stats! I had always imagined that
> there
> would be a more significant savings in having all filters in one bundle, as
> opposed to separate. These results are interesting, to say the least, and
> definitely offer us some flexibility in options for filter sharding.
>
> So far, the bulk of this discussion has centered around bandwidth. I am
> concerned, however, that splitting up the filters is at odds with the
> other
> goal of the proposal in offering improved privacy.
>
> Allowing clients to choose individual filter sets trivially exposes the
> type of
> data that client is interested in. This alone might be enough to
> fingerprint the
> function of a peer and reduce anonymity set justifying their potential
> behavior.
>
> Furthermore, if a match is encountered, and block requested, full nodes
> have
> more targeted insight into what caused a particular match. They could
> infer that
> the client received funds in a particular block, e.g., if they are only
> requesting
> output scripts.
>
> This is above and beyond the additional complexity of now syncing,
> validating,
> and managing five or six distinct header/filter-header/filter/block chains.
>
> I agree that saving on bandwidth is an important goal, but bandwidth and
> privacy
> are always seemingly at odds. Strictly comparing the bandwidth
> requirements of
> a system that heavily weighs privacy to existing ones, e.g. BIP39, that
> don't is a
> losing battle IMO.
>
> I'm not fundamentally opposed to splitting the filters, I certainly see the
> arguments for flexibility. However, I also want to ensure we are
> considering the
> second order effects that fall out of optimizing for one metric when
> others exist.
>
> Cheers,
> Conner
> On Wed, May 23, 2018 at 10:29 Gregory Maxwell via bitcoin-dev <
> bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>> Any chance you could add a graph of input-scripts (instead of input
>> outpoints)?
>>
>> On Wed, May 23, 2018 at 7:38 AM, Jim Posen via bitcoin-dev
>> <bitcoin-dev at lists.linuxfoundation.org> wrote:
>> > So I checked filter sizes (as a proportion of block size) for each of
>> the
>> > sub-filters. The graph is attached.
>> >
>> > As interpretation, the first ~120,000 blocks are so small that the
>> > Golomb-Rice coding can't compress the filters that well, which is why
>> the
>> > filter sizes are so high proportional to the block size. Except for the
>> > input filter, because the coinbase input is skipped, so many of them
>> have 0
>> > elements. But after block 120,000 or so, the filter compression
>> converges
>> > pretty quickly to near the optimal value. The encouraging thing here is
>> that
>> > if you look at the ratio of the combined size of the separated filters
>> vs
>> > the size of a filter containing all of them (currently known as the
>> basic
>> > filter), they are pretty much the same size. The mean of the ratio
>> between
>> > them after block 150,000 is 99.4%. So basically, not much compression
>> > efficiently is lost by separating the basic filter into sub-filters.
>> >
>> > On Tue, May 22, 2018 at 5:42 PM, Jim Posen <jim.posen at gmail.com> wrote:
>> >>>
>> >>> My suggestion was to advertise a bitfield for each filter type the
>> node
>> >>> serves,
>> >>> where the bitfield indicates what elements are part of the filters.
>> This
>> >>> essentially
>> >>> removes the notion of decided filter types and instead leaves the
>> >>> decision to
>> >>> full-nodes.
>> >>
>> >>
>> >> I think it makes more sense to construct entirely separate filters for
>> the
>> >> different types of elements and allow clients to download only the
>> ones they
>> >> care about. If there are enough elements per filter, the compression
>> ratio
>> >> shouldn't be much worse by splitting them up. This prevents the
>> exponential
>> >> blowup in the number of filters that you mention, Johan, and it works
>> nicely
>> >> with service bits for advertising different filter types independently.
>> >>
>> >> So if we created three separate filter types, one for output scripts,
>> one
>> >> for input outpoints, and one for TXIDs, each signaled with a separate
>> >> service bit, are people good with that? Or do you think there
>> shouldn't be a
>> >> TXID filter at all, Matt? I didn't include the option of a prev output
>> >> script filter or rolling that into the block output script filter
>> because it
>> >> changes the security model (cannot be proven to be correct/incorrect
>> >> succinctly).
>> >>
>> >> Then there's the question of whether to separate or combine the
>> headers.
>> >> I'd lean towards keeping them separate because it's simpler that way.
>> >
>> >
>> >
>> > _______________________________________________
>> > bitcoin-dev mailing list
>> > bitcoin-dev at lists.linuxfoundation.org
>> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>> >
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180523/fe5608eb/attachment-0001.html>;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: filter_sizes.svg
Type: image/svg+xml
Size: 2833873 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180523/fe5608eb/attachment-0001.svg>;

📅 Original date posted:2018-05-23 📝 Original message:So I ...

2023-06-07T20:12:14+02:00

In reply to nevent1q…afrj
_________________________

📅 Original date posted:2018-05-23
📝 Original message:So I checked filter sizes (as a proportion of block size) for each of the
sub-filters. The graph is attached.

As interpretation, the first ~120,000 blocks are so small that the
Golomb-Rice coding can't compress the filters that well, which is why the
filter sizes are so high proportional to the block size. Except for the
input filter, because the coinbase input is skipped, so many of them have 0
elements. But after block 120,000 or so, the filter compression converges
pretty quickly to near the optimal value. The encouraging thing here is
that if you look at the ratio of the combined size of the separated filters
vs the size of a filter containing all of them (currently known as the
basic filter), they are pretty much the same size. The mean of the ratio
between them after block 150,000 is 99.4%. So basically, not much
compression efficiently is lost by separating the basic filter into
sub-filters.

On Tue, May 22, 2018 at 5:42 PM, Jim Posen <jim.posen at gmail.com> wrote:

> My suggestion was to advertise a bitfield for each filter type the node
>> serves,
>> where the bitfield indicates what elements are part of the filters. This
>> essentially
>> removes the notion of decided filter types and instead leaves the
>> decision to
>> full-nodes.
>>
>
> I think it makes more sense to construct entirely separate filters for the
> different types of elements and allow clients to download only the ones
> they care about. If there are enough elements per filter, the compression
> ratio shouldn't be much worse by splitting them up. This prevents the
> exponential blowup in the number of filters that you mention, Johan, and it
> works nicely with service bits for advertising different filter types
> independently.
>
> So if we created three separate filter types, one for output scripts, one
> for input outpoints, and one for TXIDs, each signaled with a separate
> service bit, are people good with that? Or do you think there shouldn't be
> a TXID filter at all, Matt? I didn't include the option of a prev output
> script filter or rolling that into the block output script filter because
> it changes the security model (cannot be proven to be correct/incorrect
> succinctly).
>
> Then there's the question of whether to separate or combine the headers.
> I'd lean towards keeping them separate because it's simpler that way.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180523/5a74bcf7/attachment-0001.html>;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: filter_sizes.svg
Type: image/svg+xml
Size: 2066101 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180523/5a74bcf7/attachment-0001.svg>;

📅 Original date posted:2018-05-17 📝 Original message:> ...

2023-06-07T20:12:12+02:00

In reply to nevent1q…dfk7
_________________________

📅 Original date posted:2018-05-17
📝 Original message:>
> It isn't a question of 'some lite clients' -- I am aware of no
> implementation of these kinds of measures in any cryptocurrency ever.
>

Doesn't mean there can't or shouldn't be a first. :-)

> The same kind of comparison to the block could have been done with
> BIP37 filtering, but no one has implemented that. (similarly, the
> whitepaper suggests doing that for all network rules when a
> disagreement has been seen, though that isn't practical for all
> network rules it could be done for many of them-- but again no
> implementation or AFAIK any interest in implementing that)

Correct me if I'm wrong, but I don't think it's true that the same could be
done for BIP 37. With BIP 37, one would have to download every partial
block from every peer to determine if there is a difference between them.
With BIP 157, you only download a 32 byte filter header from every peer
(because filters are deterministic), and using that commitment can
determine whether there's a conflict requiring further interrogation. The
difference in overhead makes checking for conflicts with BIP 157 practical,
whereas it's not as practical with BIP 37.

> Sure, but at what cost? And "additional" while nice doesn't
> necessarily translate into a meaningful increase in delivered security
> for any particular application.
>
> I think we might be speaking too generally here.
>

Sure. The security model that BIP 157 now allows is that a light client with*
at least one honest peer serving filters* can get the correct information
about the chain. No, this does not prevent against total eclipse attacks,
but I think it's a much stronger security guarantee than requiring all
peers or even a majority of peers to be honest. In a decentralized network
that stores money, I think there's a big difference between those security
models.

> But in exchange the filters for a given FP rate would be probably
> about half the current size (actual measurements would be needed
> because the figure depends on much scriptpubkey reuse there is, it
> probably could be anywhere between 1/3 and 2/3rd).
>

This does not seem right. Let's assume txids are removed because they are
not relevant to this particular point. The difference as I understand it is
whether to include in the filter serialized outpoints for inputs or
serialized prev scriptPubkeys for inputs. When hashed these are the same
size, and there's an equal number of them (one per input in a block). So
the only savings comes from deduping the prev scriptPubkeys with each other
and with the scriptPubkeys in the block's outputs. So it comes down
entirely to how much address reuse there is on the chain.

> Monitoring inputs by scriptPubkey vs input-txid also has a massive
> advantage for parallel filtering: You can usually known your pubkeys
> well in advance, but if you have to change what you're watching block
> N+1 for based on the txids that paid you in N you can't filter them
> in parallel.
>

Yes, I'll grant that this is a benefit of your suggestion.

> I think Peter missed Matt's point that you can monitor for a specific
> transaction's confirmation by monitoring for any of the outpoints that
> transaction contains. Because the txid commits to the outpoints there
> shouldn't be any case where the txid is knowable but (an) outpoint is
> not. Removal of the txid and monitoring for any one of the outputs
> should be a strict reduction in the false positive rate for a given
> filter size (the filter will contain strictly fewer elements and the
> client will match for the same (or usually, fewer) number).
>
> I _think_ dropping txids as matt suggests is an obvious win that costs
> nothing. Replacing inputs with scripts as I suggested has some
> trade-offs.
>

I may have interpreted this differently. So wallets need a way to know when
the transactions they send get confirmed (for obvious usability reasons and
so for automatic fee-bumping). One way is to match the spent outpoints
against the filter, which I think of as the standard. Another would be to
match the txid of the spending transaction against the first, which only
works if the transaction is not malleable. Another would be to match the
change output script against the first, assuming the wallet does not reuse
change addresses and that the spending transaction does in fact have a
change output.

Now lets say these pieces of data, txids, output scripts, and spent
outpoints are in three separate filters that a wallet can download
separately or choose not to download. The spent outpoint method is the most
reliable and has no caviats. It also allows for theft detection as Peter
notes, which is a very nice property indeed. If the wallet uses the txid
matching though, the txid filter would be smaller because there are fewer
txids per block than inputs. So there could be some bandwidth savings to
that approach. The change output watching is probably the nicest in some
ways because the client needs the output filter anyway. If the transaction
has no change output with a unique script, the client could watch for any
of the other outputs on the spending tx, but may get more false positives
depending on the degree of address reuse.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180517/01b8298f/attachment-0001.html>;

📅 Original date posted:2018-05-17 📝 Original message:> ...

2023-06-07T20:12:12+02:00

In reply to nevent1q…j0x2
_________________________

📅 Original date posted:2018-05-17
📝 Original message:>
> I think lite clients cross checking is something which very likely
> will never be implemented by anyone, and probably not stay working
> (due to under-usage) if it is implemented. This thought is driven by
> three things (1) the bandwidth overhead of performing the check, (2)
> thinking about the network-interacting-state-machine complexity of it,
> and by the multitude of sanity checks that lite clients already don't
> implement (e.g. when a lite client noticed a split tip it could ask
> peers for the respective blocks and check at least the stateless
> checks, but none has ever done that), and...
>

In my opinion, it's overly pessimistic to design the protocol in an
insecure way because some light clients historically have taken shortcuts.
If the protocol can provide clients the option of getting additional
security, it should.

On the general topic, Peter makes a good point that in many cases filtering
by txid of spending transaction may be preferable to filtering by outpoint
spend, which has the nice benefit that there are obviously fewer txs in a
block than txins. This wouldn't work for malleable transactions though.

I'm open to the idea of splitting the basic filter into three separate
filters based on data type, but there are some bandwidth concerns. First,
the GCS encoding gets better compression with a greater number of elements,
though as I recall in my analysis, that starts to tail off at ~1000
elements per filter with P=20, in which case it's not as much of a concern
given current block sizes. The other is that clients need to download
and/or store the filter header chain for each filter type, which are 32
bytes each per block. So if a client is expected to download all three
filter types anyway, or even two of three, it's less efficient in these
terms. It would be possible though to split the filters themselves, but
still have the basic filter header cover all three filters. This would mean
that full nodes could not support just a subset of the basic filters --
they'd have to compute all of them to compute the filter header.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180517/c25d9cf4/attachment.html>;

📅 Original date posted:2018-04-03 📝 Original message:Hey. ...

2023-06-07T20:11:27+02:00

In reply to nevent1q…chlm
_________________________

📅 Original date posted:2018-04-03
📝 Original message:Hey. This idea sounds quite interesting. It'd be helpful to see some more
numbers to evaluate it.

- How much bandwidth is consumed by redundant tx INVs currently? What is
this as a % of overall bandwidth usage?
- How would filtering txs through N=2 links affect network propagation?
This probably requires simulation to determine.
- Do you propose setting filters on inbound peers as well?

On Mon, Apr 2, 2018 at 3:18 PM, Gleb Naumenko via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Hi all,
> I have a couple of ideas regarding transaction relay protocol and wanted
> to share it with and probably get some feedback.
>
> I did some emulation and simulation and found out that around 90% of INV
> messages sent by public-IP nodes are idle (duplicate), obviously because
> each node creates 8 connections. I also realized that sending INV messages
> is a significant part of the overall bandwidth consumed by a public-IP
> node. At a larger scale, this will result in people not able to run a
> public-IP node.
>
> My idea is in some sense similar to BIP37 but applied to public-IP nodes.
> Here I want to emphasize that all the nodes will still receive *all* of the
> transactions. A new protocol should also keep the same zero-trust,
> robustness, decentralization guarantees and latency.
>
> Idea: while joining the network, a new node agrees on some filter with
> each of 8 nodes it connects to. So that NewNode <-> Node_A will be used to
> relay only a subset of transactions, NewNode <-> Node_B for another subset.
> This will significantly decrease the redundancy.
>
> To keep the guarantees, I would keep some redundancy (for example, each
> transaction INV is sent over 2 links).
>
> To make it robust to attacks, I have 2 extensions in my mind:
> 1. Set reconciliation (for a subset of transactions) with *other* nodes.
> Getting a bloom filter of a subset of the mempool transactions from Node_B
> may help to figure out whether Node_A is malicious, very slow, etc.
> 2. Rotating the filters every N minutes (N < 10)
>
> I can see some issues with latency here, but I believe this problem has a
> solution.
>
> Feedback is appreciated!
>
> If you want to look at a draft of the proposal — please let me know.
> If there were any similar ideas — please let me know.
>
> Best,
> Gleb
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180403/89d5a284/attachment.html>;

📅 Original date posted:2017-12-15 📝 Original message:One of ...

2023-06-07T20:08:46+02:00

In reply to nevent1q…ecpx
_________________________

📅 Original date posted:2017-12-15
📝 Original message:One of the ideas that Greg Maxwell brought up in the "'Compressed' headers
stream" thread is the possibility of a header sync mechanism that allowed
parallel download from multiple peers. With the current getheaders/headers
semantics, headers must be downloaded sequentially from genesis. In my
testing, I saw that syncing headers directly from a colocated node took <5s
whereas syncing normally from network peers takes ~5 min for me, which goes
to show that 5s is an upper bound on the time to process all headers if
they are locally available. So if we can introduce new p2p messages for
header sync, what would they look like? Here's one idea.

A new getheadersv2 request would include a start height for the range of
headers requested and a commitment to the last block in the chain that you
want to download. Then you find N peers that are all on the same chain,
partition the range of headers from 0 to the chain height minus some
reasonable reorg safety buffer (~6 blocks), and send download requests in
parallel. So how do we know that the peers are on the same chain and that
their headers served connect into this chain?

When you connect to outbound peers and are in IBD, you will query them for
a Merkle Mountain Range commitment to all headers up to a height X (which
is 6ish blocks before their start height from the version message). Then
you choose the commitment that the majority of the queried peers sent (or
some other heuristic), and these become your download peers. Every
getheadersv2 request includes the start height, X, and the chain
commitment. The headersv2 response messages include all of the headers
followed by a merkle branch linking the last header into the chain
commitment. Headers are processed in order as they arrive and if any of the
headers are invalid, you can ban/disconnect all peers that committed to it,
drop the buffer of later headers and start over.

That's the basic idea. Here are other details:

- This would require an additional 32-byte MMR commitment for each header
in memory.
- When a node receives a headersv2 request and constructs a merkle proof
for the last header, it checks against the sent commitment. In the case of
a really deep reorg, that check would fail, and the node can instead
respond with an updated commitment hash for that height.
- Another packet is needed, getheaderchain or something, that a syncing
peer first sends along with a header locator and an end height. The peer
responds with headerchain, which includes the last common header from the
locator along with the chain commitment at that height and a merkle branch
proving inclusion of that header in the chain.
- Nodes would cache chain commitments for the last ~20 blocks (somewhat
arbitrary), and refuse to serve chain commitments for heights before that.

Thoughts? This is a pretty recycled idea, so please point me at prior
proposals that are similar as well.

-jimpo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20171215/267e7064/attachment-0001.html>;

📅 Original date posted:2017-12-11 📝 Original message:On ...

2023-06-07T20:08:40+02:00

In reply to nevent1q…nce8
_________________________

📅 Original date posted:2017-12-11
📝 Original message:On Mon, Dec 11, 2017 at 1:04 PM, Gregory Maxwell <greg at xiph.org> wrote:

> On Mon, Dec 11, 2017 at 8:40 PM, Jim Posen via bitcoin-dev
> <bitcoin-dev at lists.linuxfoundation.org> wrote:
> > Firstly, I don't like the idea of making the net header encoding
> dependent
> > on the specific header validation rules that Bitcoin uses (eg. the fact
> that
> > difficulty is only recalculated every 2016 blocks). This would be
> coupling
>
> In the last proposal I recall writing up, there was a one byte flag on
> each header to indicate what was included.
>
>
Is there a link somewhere to that proposal? The only thing I could find was
your forwarded email
<https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-September/014904.html>;
on
this thread.

> Nbits _never_ needs to be sent even with other consensus rules because
> its more or less necessarily a strict function of the prior headers.
> This still holds in every clone of Bitcoin I'm aware of; sending it
> with the first header in a group probably makes sense so it can be
> checked independently.
>
> > with insufficient benefit.
>
> another >18% reduction in size beyond the removal of prev. is not
> insubstantial by any means. I don't think it should lightly be
> ignored.
>
>
Omitting nBits entirely seems reasonable, I wrote up a possible
implementation here
<https://github.com/bitcoin/bitcoin/compare/master...jimpo:compact-headers-difficulty>;.
The downside is that it is more complex because it leaks into the
validation code. The extra 4 byte savings is certainly nice though.

> Prev omission itself is not, sadly, magically compatible: I am quite
> confident that if there is a bitcoin hardfork it would recover the
> nbits/4-guarenteed always-zero bits of prev to use as extra nonce for
> miners. This has been proposed many times, implemented at least once,
> and the current requirement for mining infrastructure to reach inside
> the coinbase txn to increment a nonce has been a reliable source of
> failures. So I think we'd want to have the encoding able to encode
> leading prev bits.
>
> Many altcoins also change the header structures. If the better thing
> is altcoin incompatible, we should still do it. Doing otherwise would
> competitively hobble Bitcoin especially considering the frequent
> recklessly incompetent moves made by various altcoins and the near
> total lack of useful novel development we've seen come out of the
> clones.
>
> Probably the most important change in a new header message wouldn't be
> the encoding, but it would be changing the fetching mechanism so that
> header sets could be pulled in parallel, etc.
>
> I would rather not change the serialization of existing messages,
> nodes are going to have to support speaking both messages for a long
> time, and I think we already want a different protocol flow for
> headers fetching in any case.
>

Can you elaborate on how parallel header fetching might work? getheaders
requests could probably already be pipelined, where the node requests the
next 2,000 headers before processing the current batch (though would make
sense to check that they are all above min difficulty first).

I'm open to more ideas on how to optimize the header download or design the
serialization format to be more flexible, but I'm concerned that we forgo a
40-45% bandwidth savings on the current protocol for a long time because
something better might be possible later on or there might be a hard fork
that at some point requires another upgrade. I do recognize that supporting
multiple serialization formats simultaneously adds code complexity, but in
this case the change seems simple enough to me that the tradeoff is worth
it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20171211/995788a9/attachment-0001.html>;

📅 Original date posted:2017-12-11 📝 Original message:I want ...

2023-06-07T20:08:39+02:00

In reply to nevent1q…2e58
_________________________

📅 Original date posted:2017-12-11
📝 Original message:I want to resurrect this thread from August/September because it seems like
a significant improvement for light clients at very little cost. From the
mailing list, it seems like this got stalled in determining how many more
bytes could be save in addition to the prev_block.

The ideas I've gathered from Greg Maxwell's forwarded email are:

1. Omit nBits altogether and have the receiving node determine it from
chain context.
2. Include nBits only on headers with a height that is a multiple of 2016
since it does not change in between.
3. Compress nTime to two bytes by using the bounds on allowed values from
the consensus rules.

I propose just moving ahead with only the exclusion of the prev_block, as
IMO the other savings are not worth the added complexity.

Firstly, I don't like the idea of making the net header encoding dependent
on the specific header validation rules that Bitcoin uses (eg. the fact
that difficulty is only recalculated every 2016 blocks). This would be
coupling together the two layers, breaking net compatibility for some alts,
and possibly making consensus rule changes even more difficult for a
savings with insufficient benefit. So if you buy that argument, I'm not in
favor of #2 or #3.

Option 1 is still viable, though it has some downsides. The implementation
leaks into the validation code, whereas calculating prev_block can occur
just at the net layer (see implementation below). Also, nodes would now be
*required* to sync the header chain from the genesis block, whereas they
had the option of starting from some checkpoint before.

So switching gears, I'd like to ask what the best way to actually implement
this change is. Solutions I can think of are:

1. New headers command name like "cmpctheaders" or "headersv2".
2. Change serialization of existing headers message in a new protocol
version.
3. Change serialization of existing headers message with new service bit.

I wrote up some proof-of-concept implementations in Core a) just omitting
prev_block
<https://github.com/bitcoin/bitcoin/compare/master...jimpo:compact-headers>;
and b) omitting nBits as well
<https://github.com/bitcoin/bitcoin/compare/master...jimpo:compact-headers-difficulty>;.
If people think a) is reasonable, I'll write up a BIP.

> Hi everyone, the Bitcoin headers are probably the most condensed and
> important piece of data in the world, their demand is expected to grow.
> When sending a stream of continuous block headers, a common case in IBD and
> in disconnected clients, I think there is a possible optimization of the
> transmitted data: The headers after the first could avoid transmitting the
> previous hash cause the receiver could compute it by double hashing the
> previous header (an operation he needs to do anyway to verify PoW). In a
> long stream, for example 2016 headers, the savings in bandwidth are about
> 32/80 ~= 40% without compressed headers 2016*80=161280 bytes with
> compressed headers 80+2015*48=96800 bytes What do you think? In
> OpenTimestamps calendars we are going to use this compression to give
> lite-client a reasonable secure proofs (a full node give higher security
> but isn't feasible in all situations, for example for in-browser
> verification) To speed up sync of a new client Electrum starts with the
> download of a file <https://headers.electrum.org/blockchain_headers>;
> ~36MB containing the first 477637 headers. For this kind of clients could
> be useful a common http API with fixed position chunks to leverage http
> caching. For example /headers/2016/0 returns the headers from the genesis
> to the 2015 header included while /headers/2016/1 gives the headers from
> the 2016th to the 4031. Other endpoints could have chunks of 20160 blocks
> or 201600 such that with about 10 http requests a client could fast sync
> the headers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20171211/86e7642f/attachment.html>;