npub103…2kpam

📅 Original date posted:2022-06-21
📝 Original message:
Thanks for the summary Laolu, very informative.
> One other cool topic that came up is the concept of leveraging recursive musig2 (so musig2 within musig2) to make channels even _more_ multi-sigy.
A minor point but terminology can get frustratingly sticky if it isn't agreed on early. Can we refer to it as nested MuSig2 going forward rather than recursive MuSig2? It is a more accurate description in my opinion and going through some old transcripts the MuSig2 authors [0] also refer it to nested MuSig2 (as far as I can make out).
Rene Pickhardt brought up the issue of latency with regards to nested/recursive MuSig2 (or nested FROST for threshold) on Bitcoin StackExchange [1]. Was this discussed at the LN Summit? I don't know how all the Lightning implementations treat latency currently (how long a channel counterparty has to provide a needed signature before moving to a unhappy path) but Rene's concern is delays in the regular completions of a nested MuSig2 or FROST scheme could make them unviable for the Lightning channel use case depending on the exact setup and physical location of signers etc.
MuSig2 obviously generates an aggregated Schnorr signature and so even nested MuSig2 require the Lightning protocol to recognize and verify Schnorr signatures which it currently doesn't right? So is the current thinking that Schnorr signatures will be supported first with a Schnorr 2-of-2 on the funding output (using OP_CHECKSIGADD and enabling the nested schemes) before potentially supporting non-nested MuSig2 between the channel counterparties on the funding output later? Or is this still in the process of being discussed?
[0]: https://btctranscripts.com/london-bitcoin-devs/2020-06-17-tim-ruffing-schnorr-multisig/
[1]: https://bitcoin.stackexchange.com/questions/114159/how-do-the-various-lightning-implementations-treat-latency-how-long-do-they-wai
--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
------- Original Message -------
On Wednesday, June 8th, 2022 at 03:38, Olaoluwa Osuntokun <laolu32 at gmail.com> wrote:
> Hi y'all,
>
> Last week nearly 30 (!) Lightning developers and researchers gathered in
> Oakland, California for three day to discuss a number of matters related to
> the current state and evolution of the protocol. This time around, we had
> much better representation for all the major Lightning Node implementations
> compared to the last LN Dev Summit (Zurich, Oct 2021).
>
> Similar to the prior LN Dev Summit, notes were kept throughout the day that
> attempted on a best effort basis to capture the relevant discussions,
> decisions, and new relevant research or follow up areas to circle back on.
> Last time around, I sent out an email that summarized some key takeaways
> (from my PoV) of the last multi-day dev summit [1]. What follows in this
> email is a similar summary/recap of the three day summit. Just like last
> time: if you attended and felt I missed out on a key point, or inadvertently
> misrepresented a statement/idea, please feel free to reply, correcting or
> adding additional detail.
>
> The meeting notes in full can be found here:
> https://docs.google.com/document/d/1KHocBjlvg-XOFH5oG_HwWdvNBIvQgxwAok3ZQ6bnCW0/edit?usp=sharing
>
> # Simple Taproot Channels
>
> During the last summit, Taproot was a major discussion topic as though the
> soft fork had been deployed, we we're all still watching the 🟩 's stack up
> on the road to ultimately activation. Fast forward several months later and
> Taproot has now been fully activated, with ecosystem starting to
> progressively deploy more and more advanced systems/applications that take
> advantage of the new features.
>
> One key deployment model that came out of the last LN Dev summit was the
> concept of an iterative roadmap that progressively revamped the system to
> use more taprooty features, instead of a "big bang" approach that would
> attempt to package up as many things as possible into one larger update. At
> a high level the iterative roadmap proposed that we unroll an existing
> larger proposal [2] into more bite sized pieces that can be incrementally
> reviewed, implemented, and ultimately deployed (see my post on the LN Dev
> Summit 2021 for more details).
>
> ## Extension BOLTs
>
> Riiight before we started on the first day, I wrote up a minimal proposal
> that attempted to tackle the first two items of the Taproot iterative
> deployment schedule (musig2 funding outputs and simple tapscript mapping)
> [3]. I called the proposal "Simple Taproot Channels" as it set out to do a
> mechanical mapping of the current commitment and script structure to a more
> taprooty domain. Rather than edit 4 or 5 different BOLTs with a series of
> "if this feature bit applies" nested clauses, I instead opted to create a
> new standalone "extension bolt" that defines _new_ behavior on top of the
> existing BOLTs, referring to the BOLTs when necessary. The style of the
> document was inspired by the "proposals" proposal (very meta), which was
> popularized by cdecker and adopted by t-bast with his documents on
> Trampoline and Blinded Paths.
>
> If the concept catches on, extension BOLTs provide us with a new way to
> extend the spec: rather than insert everything in-line, we could instead
> create new standalone documents for larger features. Having a single self
> contained document makes the proposal easier to review, and also gives the
> author more room to provide any background knowledge, primaries, and also
> rationale. Overtime, as the new extensions become widespread (eg: taproot is
> the default channel type), we can fold in the extensions back to the main
> set of "core" BOLTs (or make new ones as relevant).
>
> Smaller changes to the spec like deprecating an old field or tightening up
> some language will likely still follow the old approach of mutating the
> existing BOLTs, but larger overhauls like the planned PTLC update may find
> the extension BOLTs to be a better tool.
>
> ## Tapscript, Musig2, and Lightning
>
> As mentioned above the Simple Taproot Channels proposal does two main
> things:
> 1. Move the existing 2-of-2 p2wsh segwit v0 funding output to a _single
> key_ p2tr output, with the single key actually being an aggregated musig2
> key.
>
> 2. Map all our existing scripts to the tapscript domain, using the
> internal key (keyspend path) for things like revocations, which an
> potentially allow nodes to store less state for HTLCs.
>
> Of the two components #1 is by far the trickiest. Musig2 is a very elegant
> protocol (not to mention the spec which y'all should totally check out) but
> as the signatures aren't deterministic (like RFC 6979 [5]), both signers
> need to "protect themselves at all times" to ensure they don't ever re-use
> nonces, which can lead to a private key leak (!!).
>
> Rather than try to create some sort of psuedo-deterministic nonces scheme
> (which maaybe works until the Blockstream Research team squints vaguely in
> its direction), I opted to just make all nonces 100% ephemeral and tied to
> the lifetime of a connection. Musig2 defines something called a public
> nonces, which is actually two individual 33-byte nonces. This value needs to
> be exchanged before signing can begin (but can be sent before sides know
> they're aggregated keys). One important thing to note is that given that the
> channels today have _asymmetric_ state, we actually need a _pair_ of public
> nonces: one that I'll use to sign my commitment, and one I'll use to sign
> yours. Lightning channels w/ symmetric state like eltoo can get by w/ only
> exchange a single set of nonces, as there's only one message per state.
>
> Nonce exchange takes place in a few places:
>
> * During initial funding: I send my public nonce in the open_channel
> message, you send yours in the accept_channel message. After this
> exchange we can both generate signatures for the refund commitment
> transactions.
>
> * After the channel is "ready" we send another set of nonces, so we can
> sign the next state. This is similar to the existing revocation key
> exchange: I need your next nonce/key before I can sign a new state.
>
> * Upon channel re-establishment a _new_ set of nonces is sent, as they're
> 100% ephemeral. The current draft also requires that if you were
> re-transmitting a sig, then you use the _new_ nonces to sign again, as
> it's possible you went to retransmit but left off an expired/trimmed
> HLTC (could lead to nonce re-use and also needing to remember nonces).
>
> * Each time I revoke my channel, I send to you a single nonce, my "local
> nonce" (naming needs some work here), which lets you sign for a new
> state.
>
> * Each time I send a new sig, I also send you another nonce, my "remote"
> nonce", which
>
> * When I send a shutdown (co-op close) I send a single public nonce so we
> can sign the next co-opc close offer.
>
> * When I send a closing_signed I send another nonce so once you send your
> offer, we sign another set.
>
> The final flows aren't 100% yet finalized, as we'll need some
> implementations drafted to make sure the nonce handling and script mapping
> works out properly.
>
> ### Lightning Channels & Recursive Musig2
>
> One other cool topic that came up is the concept of leveraging recursive
> musig2 (so musig2 within musig2) to make channels even _more_ multi-sigy.
> The benefit here is that Bob & Carol can each have their individual keys
> (which might actually be aggregated keys themselves) and make a channel w/
> Alice, who only knows of them as Barol, and doesn't know there're actually
> another pair of keys at play. This is _really_ cool as it allows node
> operators, wallets, and lightning platforms to experiment with various
> key/signing trees that may add more security, redundancy, or flexibility.
>
> When this first came up, someone brought up the fact that while the scheme
> is "known" the initial paper as they weren't sure how to actually write a
> proof for it. During the session, someone emailed one of the musig2 authors
> asking for more details, and if it's safe to implement and roll out.
> Thankfully they quickly replied and explained that the proof recursive musig
> (pls someone correct me again here if I'm wrong) wasn't left out due to
> impossibility, but that a proof in the existing Random Oracle Model (which
> was used to derive a bound for the number of nonces needed) would lead to a
> blow up in the number of nonces required. Attempting to write the proof in
> some other model would likely lead to better results (proved w/ two nonces
> as base musig2), but would end up being pretty complicated, so hard to read
> and even review for correctness.
>
> Assuming everything checks out, then a useful mental model explained by the
> musig2 BIP author is a sort of tree structure. Assuming I'm a signer, and we
> assemble the other signer as a sibling leaf in a binary tree, then I just
> need to wait for the sibling nonce/key, before I can aggregate that into
> the final value. So if there're 3 signers, I wait for the regular public
> nonce, but the other signers sum their respective nonces into a single
> nonce, then send that to me. A similar operation is carried out for key
> aggregation, with the rest of the protocol being mostly the same.
>
> Ultimately, even if wallets/nodes aren't ready to roll something like this
> out today, we at least want to make sure the proposed flow is compatible
> with Simple Taproot Channels, and ideally we'd have a toy implementation to
> verify out understanding and show it's possible/sound. I volunteered to hack
> up a simple recursive musig2 demo, as there doesn't seem to be any code in
> the wild that implements it.
>
> ## Lightning Gossip
>
> # Gossip V2: Now Or Later?
>
> Another big topic related to Taproot was the question of how we should
> update the gossip network: the gossip protocol today has all channels
> validated by node, which requires that the nodes understand how to
> reconstruct the funding output based on the set of advertised keys. The
> protocol today assumes a segwit v0 p2wsh multi-sig is used. Assuming we had
> everything implemented today, a node wouldn't be able to advertise its new
> taproot channels to the rest of the public graph as they wouldn't understand
> how to validate it.
>
> This presents a new opportunity: we already need to rework gossip for
> taproot, so should we go ahead and re-design the entire thing with an eye
> for better privacy and future extensibility?
>
> A proposal for the "re-design the entire thing" was floated in the past by
> Rusty [6]. It does away with the strict coupling of channels to channel
> announcements, and instead moves them to the _node_ level. Each node would
> then advertise the set of "outputs" they have control of, which would then
> be mapped to the total capacity of a node, without requiring that these
> outputs self identify themselves on-chain as Lightning Channels. This also
> opens up the door to different, potentially more privacy preserving
> proofs-of-channel-ownership (something something zkp).
>
> On the other hand, we could just follow the path of Simple Taproot Channels
> and map musig2+schnorr onto the existing gossip network. This is less
> changes in total, with the main benefit being the ability to only send 1 sig
> (aggregated musig2 sig of keys) instead of 4 individual sigs. I made a very
> lofty proposal in this direction here [7].
>
> Ultimately we decided to take the "just musig2 aspects" from gossip v1.5
> (not the real name), and the "let's refresh all the messages w/ TLV
> goodness" from the gossip v2 proposal. This gives us a smaller package to
> implement, and lets us potentially rejigger the messages to be more
> extensible and remove cruft like the node color that almost nothing uses,
> but we all validate/store.
>
> The follow up work in this area is a more concrete proposal that updates the
> relevant gossip messages to be taproot aware and TLV'd and also update the
> set of requirements w.r.t _how_ to validate the channels in the first place
> (so given two keys verify that applying the keyagg method of musig2 lead to
> what' in the funding output).
>
> Gossip v2 will likely happen "eventually", but the rather large design space
> needs to be explored a bit more so we can properly analyze exactly what
> privacy and extensibility properties we'll get out of it.
>
> # Applying Mini Sketch to LN Gossip
>
> One issue we have today, is that other than the initial scid query mechanism
> added to the protocol, there isn't a great way to ensure you have all the
> latest updates your peer has. These days, many nodes pretty aggressively
> rate limit other nodes, so you might even have trouble sending out your
> update in the first place. A recent paper (that I haven't actually fully
> read yet) [8] analyzes the gossip network today to work out things like:
> exactly how long it takes things to propagate, total bandwidth usage, etc.
>
> Minisketch [9] (the grandchild of IBLTs ;)), is an efficient set
> reconciliation protocol that was designed for Bitcoin p2p mempool syncing,
> but can be applied to other protocols. An attendee has been working on
> brushing off some older work to try to see how we could apply it to the LN
> protocol to give nodes a more bandwidth efficient way to sync channel
> updates, and also achieve better update propagation. This supplements some
> existing investigative work done by Alex Meyers [10], with more concrete
> designs w.r.t: what goes into the sketch, and the various size parameters
> that need to be chosen.
>
> # Channel Jamming
>
> An attendee gave a talk on the various proposed solutions to channel
> jamming, evaluating them on several axis including: punishment/monetary,
> local vs global reputation, feasibility of mechanism design, UX
> implications, and implementation complexity. The presenter didn't present a
> new concrete proposal, but instead went through the various trade-offs,
> ultimately concluding that they factor monetary penalties wherein the funds
> are distributed across the route, rather than being provably burnt to
> miners. However they alluded to some future upcoming work that attempts a
> more rigorous analysis of the proposed solutions, their tradeoffs, and
> potential ways we can parametrize solutions to be more effective (how much
> should they pay, etc).
>
> For those looking to brush up on the latest state of research/mitigations in
> this area, I recommend this blog post by Bitmex research [11].
>
> # Onion Messages & DoS
>
> The topic of DoS concerns related to onion messages (in isolation, so not
> necessarily related to things like bolt12 that take advantage of them came
> up. During a white boarding session some argued that DoS isn't actually
> much of an issue, as nodes can leverage "back propagation congestion
> control" to inform the source (who may not actually be the sender) that
> they'll start to drop or limit their packets, with each node doing this
> iteratively until the actual source of the spam has been clamped. A few
> lofty designs were thrown around, but more work needs to be done to
> concretely specify something so it can be properly analyzed.
>
> On the other side of the spectrum, rather than attempt to rate limit at the
> node level (which each node having their own policy), nodes could opt
> instead to forward _anything_ as long as the sender pays them enough. I
> proposed a lofty approach that combined AMP and Onion Messages earlier this
> year [12]. At a high level I make an AMP payment, which pushes extra coins
> to all nodes on a route, and also drops off a special identifier to them.
> When I send an onion message I include this identifier, with each node
> performing their own account w.r.t the amount of bandwidth an ID has
> remaining.
>
> Ultimately a few implementations are pretty close to deploying their
> implementation of onion messages, so no matter the intended use case, it
> would be good to have code deployed along side to either rate limit or price
> resource consumption accordingly. Otherwise, we might end up in a scenario
> where DoS concerns were brushed aside, but end up being a huge issue later.
>
> # Blinded Paths, QR Codes & Invoices
>
> Blinded paths [13] is a new-er proposal to solve the "last mile" privacy
> issue when receiving payments on LN. Today invoices to unadvertised channels
> contain a set of hop hints, which are anchored at public nodes in the graph,
> and also leak the scid of the unadvertised channel (points on-chain to the
> channel receiving payments). A solution for the on-chain leak, SCID channel
> aliases [15] are in the process of being widely rolled out. Channel aliases
> instead use a random value in the invoice, allowing receiving nodes to break
> that on-chain link and even rotate out the value periodically. With the
> on-chain leak addressed, it's still the case that you give away your
> "position" in the network, since as a sender I know that you're connected to
> node N with a private channel.
>
> Blinded paths address this node-level last mile privacy leak by replacing
> hop hints with a new cryptographically blinded path. At a high level, the
> receiver can construct a "hop hint" of length greater than 1, gather the
> public keys of each of the nodes, then blinded them such that: the sender
> can use them for path finding, but doesn't actually now exactly _which_
> nodes they actually are.
>
> There're two type of blinded paths: those in onion messages and those used
> for actual payments. The latter variant was only formalized earlier this
> year, as before people were mainly interested in using them to fetch BOLT 12
> invoice via onion messages. One issue that pops up when attempting to use
> blinded paths for normal payments is: the size of the resulting invoice. As
> blinded paths are actually fragments of publicly known paths, as a receiver,
> you want to stuff as many of them into the invoice as possible, since they
> MUST be taken in order to route towards you. Invoices are typically
> communicated via QR codes, which have a hard limit w.r.t the amount of
> information that can be packed in. On the other hand for invoice fetching,
> all that matters is that a path exists, so you can get by with stuffing less
> of then in a QR code.
>
> As a result, blinded paths aren't necessarily compatible with the widely
> deployed BOLT 11 based QR codes. Instead a way to fetch invoice on demand is
> required. Both BOLT-12 and LN-URL provide standardized ways for nodes to
> fetch invoices, though their transport/signalling medium of choice differs.
> Blinded routes are technically compatible with BOLT 11 invoices, but may be
> hampered by the fact that you can only include so many routes.
>
> Another consideration is that unlike hop hints, blinded paths require more
> maintain once, as since they traverse public route, policy changes like a
> fee update may invalidate an entire set set of routes. One proposed solution
> is that forwarding nodes should observe their older policy for a period of
> time (so a grace period), and also that blinded paths should have an
> explicit expiry (similar to the existing invoice expiry).
>
> One other implication is that the set of routes the receiver includes matters
> more: if they don't send enough or select them poorly, the sender may never be
> able to reach them even though a path exists in theory. More hands on
> experience is needed so the spec authors can better guide implementations and
> wallets w.r.t best practices.
>
> # Friend-of-a-friend Balance Sharing & Probing
>
> A presentation was given on friend-of-a-friend balance sharing [16]. The
> high level idea is that if we share _some_ information within a local
> radius, then this gives the sender more information to choose a path that's
> potentially more reliable. The tradeoff here ofc is that nodes will be
> giving away more information that can potentially be used to ascertain
> payment flows. In an attempt to minimize the amount of information shared,
> the presenter proposed that just 2 bits of information be shared. Some
> initial simulations showed that sharing local information actually performed
> better than sharing global information (?). Some were puzzled w.r.t how
> that's possible, but assuming the slides+methods are published others can
> dig further into the model/parameter used to signal the inclusion.
>
> Arguably, information like this is already available via probing, so one
> line of thinking is something like: "why not just share _some_ of it" that
> may actually lead to less internal failures? This is related to a sort of
> tension between probing as a tool to increase payment reliability and also
> as a tool to degrade privacy in the network. On the other hand, others
> argued that probing provides natural cover traffic, since they actually
> _are_ payments, though they may not be intended to succeed.
>
> On the topic of channel probing, a sort of makeshift protocol was devised to
> make it harder in practice, sacrificing too much on the axis of payment
> reliability. At a high level it proposes that:
>
> * nodes more diligently set both their max_htlc amount, as well as the
> max_htlc_value_in_flight amount
>
> * a 50ms (or select other value) timer should be used when sending out
> commitment signatures, independent of HTLC arrival
>
> * nodes leverage the max_htlc value to set a false ceiling on the max in
> flight parameter
>
> * for each HTLC sent/forwarded, select 2 other channels at random and
> reduce the "fake" in-flight ceiling for a period of time
>
> Some more details still need to be worked out, but some felt that this would
> kick start more research into this area, and also make balance mapping
> _slightly_ more difficult. From afar, it may be the case that achieving
> balance privacy while also achieving acceptable levels of payment
> reliability might be at odds with each other.
>
> # Eltoo & ANYPREVOUT
>
> One of the attendees is currently working on both fully implementing eltoo,
> as well as specifying the exact channel funding+update interaction were it
> to be rolled out align side the existing penalty based channels in the
> protocol. As this version of eltoo is based on Taproot, we were able to
> compare notes a bit to find the overlapping set of changes (nonce handling,
> etc), which permits cross review of the proposals. This type of work is
> cool, as only by fully implementing something end to end can you reaaally
> work out all the edge cases and nuances.
>
> ANYPREVOUT as hasn't changed significantly as of late. An attendee shared
> plans to create a sort of mega all-future-feasible-soft-forks fork of
> bitcoind, that would package up various unmerged (from bitcoind's) proposal
> soft fork packages into an easy to run+install binary/project attached to a
> signet. The hop is that by giving developers an easy way to interact with
> proposed soft fork proposals (vs debasing some ancient pull request), wider
> participation in testing/implementation/review can be facilitated.
>
> # Trampoline Routing
>
> There was a presentation on Trampoline routing explaining the motivation,
> history, and current state of the proposal. The two main cases we've
> narrowed down on are:
>
> 1. A mobile user doesn't necessarily want to sync the _entire_ graph, so
> they can use trampoline to maintain a subset and still be able to send
> payments.
>
> 2. A mobile user wants to be able to instate a payment, go offline, and
> return at a later time to learn about the final state of the payment.
>
> Use case #2 seems to be the most promising when combined with other
> proposals for holding HTLCs at an origin node (call it an "LSP") [13].
> Combined together, this would allow a mobile node to send a payment, then go
> offline, with the LSP being able to retry the payment either continuously or
> only when it knows the receiver is online to accept the payment. This may
> potentially dramatically improve the UX for LN on mobile, as things suddenly
> become a lot more asynchronous: I do something go offline, and the LSP node
> can fulfil the payment in the background, then wait for me to come online to
> settle the final. hop.
>
> Trampoline can also be composed well with blinded routes (blinded route from
> last trampoline to receiver) and also MPP (internal nodes can split
> themselves with local information).
>
> One added trade-off is that since the sender doesn't know the entire route,
> they need to sort of overshoot w.r.t fees and CTLVs. This is something
> we've known for a while, but until Trampoline is more widely rolled out, we
> won't have a very good feel w.r.t how much extra senders will need to
> allocate.
>
> # Node Fee Optimization & Fee Rate Cards
>
> Over the past few years, a common thread we've seen across successful
> routing nodes is dynamic fee setting as a way to encourage/discourage
> traffic. A routing nodes can utilize the set of fees of a channel to either
> make it too expensive for other nodes to route through (it's already
> depleted don't try unless you'll give be 10 mil sats, which no one would) or
> very cheap, which'll incentivize flows in the other direction. If all nodes
> are constantly sending out updates of this nature, then it can generate a
> lot of traffic, and also sort of leak more balance information overtime
> (which some nodes are already doing: using fees/max_htlc to communicate
> available balances).
>
> One attendee proposed allowing nodes to express a sort of fee gradient via a
> static curve/bucket/function, instead of dynamically communicating what the
> latest state of the fee+liquidity distribution looks like. A possible
> manifestation could be a series of buckets, each of which with varying fee
> rates. If your payment consumes 50% of channel balance, then you pay this
> rate, otherwise if it's 5% you pay this rate, etc, etc. This might allow for
> nodes to capture the same dynamics as they do with more dynamic fee updates,
> but in a way that leaks less information and also consumes less gossip
> bandwidth.
>
> # The Return of Splicing
>
> Splicing is one of those things that was discussed a long time ago, but was
> never really fully implemented and rolled out. A few attendees have started
> to take a closer look at the problem, building off of the interactive-tx
> scheme that the dual-funding protocol extension uses. The main intricacy
> discussed was if concurrent splices should be allowed or not, and if so, how
> we would handle the various edge cases. As an example, if I propose a splice
> to add more funds via my input, but that turns out to already be spent, then
> the splicing transaction we created is invalid and can never be confirmed.
> However if we allow _another_ splice to take place, and another one, and
> another one, then ideally _one_ of them will confirm and serve as the new
> anchor for the channel.
>
> In a world of concurrent splices, the question of "what is my Lightning
> balance" becomes even more murky. Wallet and implementations will likely
> want to show the most pessimistic value, while also ensuring that the user
> is able to effectively account for where all their funds and what they can
> spend on/off chain.
>
> # LN-URL + BOLT 12
>
> LN-URL and BOLT 12 are both standardized ways that answer the question of:
> how can I fetch an invoice from Bob? LN-URL differs from BOLT 12 in that it
> uses the existing BOLT 11 invoice format, and uses an HTTP based protocol
> for the negotiation process. BOLT 12 on the other hand is a suite of
> protocol additions that includes (amongst other things) a new invoice format
> (yay TLV!) and also a way to use onion messages to fetch an invoice _via_
> the network.
>
> Assuming blinded paths is widely rolled out, then the question of how
> invoices are obtained becomes more important as blinded paths means that you
> can't fit much in the traditional QR encoding. As a result, fetching
> invoices on demand may become a more common place flow, with all its
> trade-offs. There was a group discussion on how we could sort of unifying
> everything either by allowing BOLT 12 to be used over LN-URL or the other
> way around.
>
> One proposal was to add a new query parameter to the normal LN-URL QR code
> contents. This would mean that when a wallet goes to scan an LN-URL QR code,
> if they know of the extra param, and what BOLT 12, they can just use the
> enclosed offer to fetch the invoice.
>
> An alternative proposal was to instead extract the BOLT 12 _invoice_ format
> from the greater BOLT 12 "Offers" proposal. Assuming blinded paths is only
> specified w.r.t BOLT 12 _invoices_, then this would mean an LN-URL extension
> could be rolled out that allowed returning BOLT 12 invoice rather than BOLT
> 11 invoices. This would allow the ecosystem to slowly transition to a shared
> invoice format, even if there may be fundamental disagreements w.r.t _how_
> the invoices should be fetched in the first place.
>
> It's worth noting that both of these proposals can be combined:
>
> * If a wallet knows how to BOLT 12 Offers, they can take the enclosed
> offer and run w/ it.
>
> * If they don't know about Offers, but can send w/ the BOLT _invoice_
> format, then they can fetch that and complete the payment.
>
> This might be a nice middle ground as it would tend all
> wallets/implementations to being able to decode and send w/ a BOLT 12
> _invoice_, and leave the question of _how_ it should be fetched up to the
> application/wallet/service. In the end, if paths never quite intersect, then
> it's still possible to add route blinding to BOLT 11, with LN-URL sticking
> with that invoice format to take advantage of the new privacy enhancements
>
> [1]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-November/003336.html
> [2]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-October/003278.html
> [3]: https://github.com/lightning/bolts/pull/995
> [4]: https://github.com/jonasnick/bips/blob/musig2/bip-musig2.mediawiki
> [5]: https://datatracker.ietf.org/doc/html/rfc6979
> [6]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003470.html
> [7]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-March/003526.html
> [8]: https://arxiv.org/abs/2205.12737
> [9]: https://bitcoinops.org/en/topics/minisketch/
> [10]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-April/003551.html
> [11]: https://blog.bitmex.com/preventing-channel-jamming/
> [12]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003498.html
> [13]: https://github.com/lightning/bolts/pull/765
> [14]: https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-October/003307.html
> [15]: https://github.com/lightning/bolts/pull/910
> [16]: https://github.com/lightning/bolts/pull/780
>
> -- Laolu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20220621/89814a22/attachment-0001.html>

📅 Original date posted:2022-02-05
📝 Original message:Thanks for this Bastien (and Gloria for initially posting about this).
I sympathetically skimmed the eclair PR (https://github.com/ACINQ/eclair/pull/2113) dealing with replaceable transactions fee bumping.
There will continue to be a (hopefully) friendly tug of war on this probably for the rest of Bitcoin's existence. I am sure people like Luke, Prayank etc will (rightfully) continue to raise that Lightning and other second layer protocols shouldn't demand that policy rules be changed if there is a reason (e.g. DoS vector) for those rules on the base network. But if there are rules that have no upside, introduce unnecessary complexity for no reason and make Lightning implementers like Bastien's life miserable attempting to deal with them I really hope we can make progress on removing or simplifying them.
This is why I think it is important to understand the rationales for introducing the rules in the first place (and why it is safe to remove them if indeed it is) and being as rigorous as possible on the rationales for introducing additional rules. It sounds like from Gloria's initial post we are still at a brainstorming phase (which is fine) but knowing what we know today I really hope we can learn from the mistakes of the original BIP 125, namely the Core implementation not matching the BIP and the sparse rationales for the rules. As Bastien says this is not criticizing the original BIP 125 authors, 7 years is a long time especially in Bitcoin world and they probably weren't thinking about Bastien sitting down to write an eclair PR in late 2021 (and reviewers of that PR) when they wrote the BIP in 2015.
--
Michael Folkson
Email: michaelfolkson at [protonmail.com](http://protonmail.com/)
Keybase: michaelfolkson
PGP: 43ED C999 9F85 1D40 EAF4 9835 92D6 0159 214C FEE3
------- Original Message -------
On Monday, January 31st, 2022 at 3:57 PM, Bastien TEINTURIER via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi Gloria,
>
> Many thanks for raising awareness on these issues and constantly pushing
> towards finding a better model. This work will highly improve the
> security of any multi-party contract trying to build on top of bitcoin
> (because most multi-party contracts will need to have timeout conditions
> and participants will need to make some transactions confirm before a
> timeout happens - otherwise they may lose funds).
>
> For starters, let me quickly explain why the current rules are hard to
> work with in the context of lightning (but I believe most L2 protocols
> will have the same issues). Feel free to skip this part if you are
> already convinced.
>
> ## Motivation
>
> The biggest pain point is BIP 125 rule 2.
> If I need to increase the fees of a time-sensitive transaction because
> the feerate has been rising since I broadcast it, I may need to also pay
> high fees just to produce a confirmed utxo that I can use. I'm actually
> paying a high fee twice instead of once (and needlessly using on-chain
> space, our scarcest asset, because we could have avoided that additional
> transaction!).
>
> It also has some annoying "non-determinism".
> Imagine that my transaction has been evicted from my mempool because its
> feerate was too low. I could think "Great, that means I don't have to
> apply BIP 125 restrictions, I can just fund this transaction as if it
> were a new one!". But actually I do, because my transaction could still
> be in miner's mempools and I have no way of knowing it...this means that
> whenever I have broadcast a transaction, I must assume that I will
> always need to abide by whatever replacement rules the network applies.
>
> Fortunately, as far as I understand it, this rule only exists because of
> a previous implementation detail of bitcoin core, so there's simply no
> good reason to keep it.
>
> The second biggest pain point is rule 3. It prevents me from efficiently
> using my capital while it's unconfirmed. Whenever I'm using a big utxo
> to fund a transaction, I will get a big change output, and it would
> really be a waste to be unable to use that change output to fund other
> transactions. In order to be capital-efficient, I will end up creating
> descendant trees for my time-sensitive transactions. But as Gloria
> explained, replacing all my children will cost me an absurdly large
> amount of fees. So what I'm actually planning to do instead is to RBF
> one of the descendants high enough to get the whole tree confirmed.
> But if those descendants' timeouts were far in the future, that's a
> waste, I paid a lot more fees for them than I should have. I'd like to
> just replace my transaction and republish the invalidated children
> independently.
>
> Rule 4 doesn't hurt as much as the two previous ones, I don't have too
> much to say about it.
>
> To be fair to the BIP 125 authors, all of these scenarios were very hard
> to forecast at the time this BIP was created. We needed years to build
> on those rules to get a better understanding of their limitations and if
> the rationale behind them made sense in the long term.
>
> ## Proposals
>
> I believe that now is a good time to re-think those, and I really like
> Gloria's categorization of the design constraints.
>
> I'd like to propose a different way of looking at descendants that makes
> it easier to design the new rules. The way I understand it, limiting the
> impact on descendant transactions is only important for DoS protection,
> not for incentive compatibility. I would argue that after evictions,
> descendant transactions will be submitted again (because they represent
> transactions that people actually want to make), so evicting them does
> not have a negative impact on mining incentives (in a world where blocks
> are full most of the time).
>
> I'm curious to hear other people's thoughts on that. If it makes sense,
> I would propose the following very simple rules:
>
> 1. The transaction's ancestor absolute fees must be X% higher than the
> previous transaction's ancestor fees
> 2. The transaction's ancestor feerate must be Y% higher than the
> previous transaction's ancestor feerate
>
> I believe it's completely ok to require increasing both the fees and
> feerate if we don't take descendants into account, because you control
> your ancestor set - whereas the descendant set may be completely out of
> your control.
>
> This is very easy to use by wallets, because the ancestor set is easy to
> obtain. And an important point is that the ancestor set is the same in
> every mempool, whereas the descendant set is not (your mempool may have
> rejected the last descendants, while other people's mempools may still
> contain them).
>
> Because of that reason, I'd like to avoid having a rule that relies on
> some size of the replaced descendant set: it may be valid in your
> mempool but invalid in someone else's, which makes it exploitable for
> pinning attacks.
>
> I believe these rules are incentive compatible (again, if you accept
> the fact that the descendants will be re-submitted and mined as well,
> so their fees aren't lost).
>
> Can we choose X and Y so that these two rules are also DoS-resistant?
> Unfortunately I'm not sure, so maybe we'll need to add a third rule to
> address that. But before we do, can someone detail what it costs for a
> node to evict a descendant tree? Given that bitcoin core doesn't allow
> chains of more than 25 transactions, the maximum number of transactions
> being replaced will be bounded by 25 * N (where N is the number of
> outputs of the transaction being replaced). If it's just O(n) pruning of
> a graph, maybe that's ok? Or maybe we make X or Y depend on the number
> of outputs of the transaction being replaced (this would need very
> careful thoughts)?
>
> If you made it this far, thanks for reading!
> A couple of comments on the previous messages:
>
>> Currently, if we see a transaction
>> that has the same txid as one in the mempool, we reject it as a
>> duplicate, even if the feerate is much higher. It's unclear to me if
>> we have a very strong reason to change this, but noting it as a
>> limitation of our current replacement policy.
>
> I don't see a strong reason from an L2 protocol's point of view yet, but
> there are many unkown unknowns. But from a miner incentive's point of
> view, we should keep the transaction with the higher feerate, shouldn't
> we? In that case it's also a more efficient use of on-chain space, which
> is a win, right?
>
>> We might have a more-or-less long transition period during which we support both...
>
> Yes, this is a long term thing.
> Even if bitcoin core releases a new version with updated RBF rules, as a
> wallet you'll need to keep using the old rules for a long time if you
> want to be safe.
>
> But it's all the more reason to try to ship this as soon as possible,
> this way maybe our grand-children will be able to benefit from it ;)
> (just kidding on the timespan obviously).
>
> Cheers,
> Bastien
>
> Le lun. 31 janv. 2022 à 00:11, Antoine Riard via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> a écrit :
>
>> Hi Gloria,
>>
>> Thanks for this RBF sum up. Few thoughts and more context comments if it can help other readers.
>>
>>> For starters, the absolute fee pinning attack is especially
>>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>> LN channel. The mempool is rather full, so their pre-negotiated
>>> commitment transactions' feerates would not be considered high
>>> priority by miners. Bob broadcasts his commitment transaction and
>>> attaches a very large child (100KvB with 100,000sat in fees) to his
>>> anchor output. Alice broadcasts her commitment transaction with a
>>> fee-bumping child (200vB with 50,000sat fees which is a generous
>>> 250sat/vB), but this does not meet the absolute fee requirement. She
>>> would need to add another 50,000sat to replace Bob's commitment
>>> transaction.
>>
>> Solving LN pinning attacks, what we're aiming for is enabling a fair feerate bid between the counterparties, thus either forcing the adversary to overbid or to disengage from the confirmation competition. If the replace-by-feerate rule is adopted, there shouldn't be an incentive for Bob to
>> pick up the first option. Though if he does, that's a winning outcome for Alice, as one of the commitment transactions confirms and her time-sensitive second-stage HTLC can be subsequently confirmed.
>>
>>> It's unclear to me if
>>> we have a very strong reason to change this, but noting it as a
>>> limitation of our current replacement policy. See [#24007][12].
>>
>> Deployment of Taproot opens interesting possibilities in the vaults/payment channels design space, where the tapscripts can commit to different set of timelocks/quorum of keys. Even if the pre-signed states stay symmetric, whoever is the publisher, the feerate cost to spend can fluctuate.
>>
>>> While this isn't completely broken, and the user interface is
>>> secondary to the safety of the mempool policy
>>
>> I think with L2s transaction broadcast backend, the stability and clarity of the RBF user interface is primary. What we could be worried about is a too-much complex interface easing the way for an attacker to trigger your L2 node to issue policy-invalid chain of transactions. Especially, when we consider that an attacker might have leverage on chain of transactions composition ("force broadcast of commitment A then commitment B, knowing they will share a CPFP") or even transactions size ("overload commitment A with HTLCs").
>>
>>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>> mempool, apply the current rules (absolute fees must increase and
>>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>>> feerate-only rule.
>>
>> How this new replacement rule would behave if you have a parent in the "replace-by-feerate" half but the child is in the "replace-by-fee" one ?
>>
>> If we allow the replacement of the parent based on the feerate, we might decrease the top block absolute fees.
>>
>> If we block the replacement of the parent based on the feerate because the replacement absolute fees aren't above the replaced package, we still preclude a pinning vector. The child might be low-feerate junk and even attached to a low ancestor-score branch.
>>
>> If I'm correct on this limitation, maybe we could turn off the "replace-by-fee" behavior as soon as the mempool is fulfilled with a few blocks ?
>>
>>> * Rate-limit how many replacements we allow per prevout.
>>
>> Depending on how it is implemented, though I would be concerned it introduces a new pinning vector in the context of shared-utxo. If it's a hardcoded constant, it could be exhausted by an adversary starting at the lowest acceptable feerate then slowly increasing while still not reaching
>> the top of the mempool. Same if it's time-based or block-based, no guarantee the replacement slot is honestly used by your counterparty.
>>
>> Further, an above-the-average replacement frequency might just be the reflection of your confirmation strategy reacting to block schedule or mempools historical data. As long as the feerate penalty is paid, I lean to allow replacement.
>> (One solution could be to associate per-user "tag" to the LN transactions, where each "tag" would have its own replacement slots, but privacy?)
>>
>>> * Rate-limit transaction validation in general, per peer.
>>
>> I think we could improve on the Core's new transaction requester logic. Maybe we could bind the peer announced flow based on the feerate score (modulo validation time) of the previously validated transactions from that peer ? That said, while related to RBF, it sounds to me that enhancing Core's rate-limiting transaction strategy is a whole discussion in itself [0]. Especially ensuring it's tolerant to the specific requirements of LN & consorts.
>>
>>> What should they be? We can do some arithmetic to see what happens if
>>> you start with the biggest/lowest feerate transaction and do a bunch
>>> of replacements. Maybe we end up with values that are high enough to
>>> prevent abuse and make sense for applications/users that do RBF.
>>
>> That's a good question.
>>
>> One observation is that the attacker can always renew the set of DoSy utxos to pursue the attack. So maybe we could pick up constants scaled on the block size ? That way an attacker would have to burn fees, thus deterring them from launching an attack. Even if the attackers are miners, they have to renounce their income to acquire new DoSy utxos. If a low-fee period, we could scale up the constants ?
>>
>> Overall, I think there is the deployment issue to warn of. Moving to a new set of RBF rules implies for a lot of Bitcoin applications to rewrite their RBF logics. We might have a more-or-less long transition period during which we support both...
>>
>> Cheers,
>> Antoine
>>
>> [0] https://github.com/bitcoin/bitcoin/pull/21224
>>
>> Le jeu. 27 janv. 2022 à 09:10, Gloria Zhao via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> a écrit :
>>
>>> Hi everyone,
>>>
>>> This post discusses limitations of current Bitcoin Core RBF policy and
>>> attempts to start a conversation about how we can improve it,
>>> summarizing some ideas that have been discussed. Please reply if you
>>> have any new input on issues to be solved and ideas for improvement!
>>>
>>> Just in case I've screwed up the text wrapping again, another copy can be
>>> found here: https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff
>>>
>>> ## Background
>>>
>>> Please feel free to skip this section if you are already familiar
>>> with RBF.
>>>
>>> Nodes may receive *conflicting* unconfirmed transactions, aka
>>> "double spends" of the same inputs. Instead of always keeping the
>>> first transaction, since v0.12, Bitcoin Core mempool policy has
>>> included a set of Replace-by-Fee (RBF) criteria that allows the second
>>> transaction to replace the first one and any descendants it may have.
>>>
>>> Bitcoin Core RBF policy was previously documented as BIP 125.
>>> The current RBF policy is documented [here][1]. In summary:
>>>
>>> 1. The directly conflicting transactions all signal replaceability
>>> explicitly.
>>>
>>> 2. The replacement transaction only includes an unconfirmed input if
>>> that input was included in one of the directly conflicting
>>> transactions.
>>>
>>> 3. The replacement transaction pays an absolute fee of at least the
>>> sum paid by the original transactions.
>>>
>>> 4. The additional fees pays for the replacement transaction's
>>> bandwidth at or above the rate set by the node's *incremental relay
>>> feerate*.
>>>
>>> 5. The sum of all directly conflicting transactions' descendant counts
>>> (number of transactions inclusive of itself and its descendants)
>>> does not exceed 100.
>>>
>>> We can split these rules into 3 categories/goals:
>>>
>>> - **Allow Opting Out**: Some applications/businesses are unable to
>>> handle transactions that are replaceable (e.g. merchants that use
>>> zero-confirmation transactions). We (try to) help these businesses by
>>> honoring BIP125 signaling; we won't replace transactions that have not
>>> opted in.
>>>
>>> - **Incentive Compatibility**: Ensure that our RBF policy would not
>>> accept replacement transactions which would decrease fee profits
>>> of a miner. In general, if our mempool policy deviates from what is
>>> economically rational, it's likely that the transactions in our
>>> mempool will not match the ones in miners' mempools, making our
>>> fee estimation, compact block relay, and other mempool-dependent
>>> functions unreliable. Incentive-incompatible policy may also
>>> encourage transaction submission through routes other than the p2p
>>> network, harming censorship-resistance and privacy of Bitcoin payments.
>>>
>>> - **DoS Protection**: Limit two types of DoS attacks on the node's
>>> mempool: (1) the number of times a transaction can be replaced and
>>> (2) the volume of transactions that can be evicted during a
>>> replacement.
>>>
>>> Even more abstract: our goal is to make a replacement policy that
>>> results in a useful interface for users and safe policy for
>>> node operators.
>>>
>>> ## Motivation
>>>
>>> There are a number of known problems with the current RBF policy.
>>> Many of these shortcomings exist due to mempool limitations at the
>>> time RBF was implemented or result from new types of Bitcoin usage;
>>> they are not criticisms of the original design.
>>>
>>> ### Pinning Attacks
>>>
>>> The most pressing concern is that attackers may take advantage of
>>> limitations in RBF policy to prevent other users' transactions from
>>> being mined or getting accepted as a replacement.
>>>
>>> #### SIGHASH_ANYONECANPAY Pinning
>>>
>>> BIP125#2 can be bypassed by creating intermediary transactions to be
>>> replaced together. Anyone can simply split a 1-input 1-output
>>> transaction off from the replacement transaction, then broadcast the
>>> transaction as is. This can always be done, and quite cheaply. More
>>> details in [this comment][2].
>>>
>>> In general, if a transaction is signed with SIGHASH\_ANYONECANPAY,
>>> anybody can just attach a low feerate parent to this transaction and
>>> lower its ancestor feerate. Even if you require SIGHASH\_ALL which
>>> prevents an attacker from changing any outputs, the input can be a
>>> very low amount (e.g. just above the dust limit) from a low-fee
>>> ancestor and still bring down the ancestor feerate of the transaction.
>>>
>>> TLDR: if your transaction is signed with SIGHASH\_ANYONECANPAY and
>>> signals replaceability, regardless of the feerate you broadcast at, an
>>> attacker can lower its mining priority by adding an ancestor.
>>>
>>> #### Absolute Fee
>>>
>>> The restriction of requiring replacement transactions to increase the
>>> absolute fee of the mempool has been described as "bonkers." If the
>>> original transaction has a very large descendant that pays a large
>>> amount of fees, even if it has a low feerate, the replacement
>>> transaction must now pay those fees in order to meet Rule #3.
>>>
>>> #### Package RBF
>>>
>>> There are a number of reasons why, in order to enable Package RBF, we
>>> cannot use the same criteria.
>>>
>>> For starters, the absolute fee pinning attack is especially
>>> problematic if we apply the same rules (i.e. Rule #3 and #4) in
>>> Package RBF. Imagine that Alice (honest) and Bob (adversary) share a
>>> LN channel. The mempool is rather full, so their pre-negotiated
>>> commitment transactions' feerates would not be considered high
>>> priority by miners. Bob broadcasts his commitment transaction and
>>> attaches a very large child (100KvB with 100,000sat in fees) to his
>>> anchor output. Alice broadcasts her commitment transaction with a
>>> fee-bumping child (200vB with 50,000sat fees which is a generous
>>> 250sat/vB), but this does not meet the absolute fee requirement. She
>>> would need to add another 50,000sat to replace Bob's commitment
>>> transaction.
>>>
>>> Disallowing new unconfirmed inputs (Rule #2) in Package RBF would be
>>> broken for packages containing transactions already in the mempool,
>>> explained [here][7].
>>>
>>> Note: I originally [proposed][6] Package RBF using the same Rule #3
>>> and #4 before I realized how significant this pinning attack is. I'm
>>> retracting that proposal, and a new set of Package RBF rules would
>>> follow from whatever the new individual RBF rules end up being.
>>>
>>> #### Same Txid Different Witness
>>>
>>> Two transactions with the same non-witness data but different
>>> witnesses have the same txid but different wtxid, and the same fee but
>>> not necessarily the same feerate. Currently, if we see a transaction
>>> that has the same txid as one in the mempool, we reject it as a
>>> duplicate, even if the feerate is much higher. It's unclear to me if
>>> we have a very strong reason to change this, but noting it as a
>>> limitation of our current replacement policy. See [#24007][12].
>>>
>>> ### User Interface
>>>
>>> #### Using Unconfirmed UTXOs to Fund Replacements
>>>
>>> The restriction of only allowing confirmed UTXOs for funding a
>>> fee-bump (Rule #2) can hurt users trying to fee-bump their
>>> transactions and complicate wallet implementations. If the original
>>> transaction's output value isn't sufficient to fund a fee-bump and/or
>>> all of the user's other UTXOs are unconfirmed, they might not be able
>>> to fund a replacement transaction. Wallet developers also need to
>>> treat self-owned unconfirmed UTXOs as unusable for fee-bumping, which
>>> adds complexity to wallet logic. For example, see BDK issues [#144][4]
>>> and [#414][5].
>>>
>>> #### Interface Not Suitable for Coin Selection
>>>
>>> Currently, a user cannot simply create a replacement transaction
>>> targeting a specific feerate or meeting a minimum fee amount and
>>> expect to meet the RBF criteria. The fee amount depends on the size of
>>> the replacement transaction, and feerate is almost irrelevant.
>>>
>>> Bitcoin Core's `bumpfee` doesn't use the RBF rules when funding the
>>> replacement. It [estimates][13] a feerate which is "wallet incremental
>>> relay fee" (a conservative overestimation of the node's incremental
>>> relay fee) higher than the original transaction, selects coins for
>>> that feerate, and hopes that it meets the RBF rules. It never fails
>>> Rule #3 and #4 because it uses all original inputs and refuses to
>>> bump a transaction with mempool descendants.
>>>
>>> This is suboptimal, but is designed to work with the coin selection
>>> engine: select a feerate first, and then add fees to cover it.
>>> Following the exact RBF rules would require working the other way
>>> around: based on how much fees we've added to the transaction and its
>>> current size, calculate the feerate to see if we meet Rule #4.
>>>
>>> While this isn't completely broken, and the user interface is
>>> secondary to the safety of the mempool policy, we can do much better.
>>> A much more user-friendly interface would depend *only* on the
>>> fee and size of the original transactions.
>>>
>>> ### Updates to Mempool and Mining
>>>
>>> Since RBF was first implemented, a number of improvements have been
>>> made to mempool and mining logic. For example, we now use ancestor
>>> feerates in mining (allowing CPFP), and keep track of ancestor
>>> packages in the mempool.
>>>
>>> ## Ideas for Improvements
>>>
>>> ### Goals
>>>
>>> To summarize, these seem to be desired changes, in order of priority:
>>>
>>> 1. Remove Rule #3. The replacement should not be *required* to pay
>>> higher absolute fees.
>>>
>>> 2. Make it impossible for a replacement transaction to have a lower
>>> mining score than the original transaction(s). This would eliminate
>>> the `SIGHASH\_ANYONECANPAY` pinning attack.
>>>
>>> 3. Remove Rule #2. Adding new unconfirmed inputs should be allowed.
>>>
>>> 4. Create a more helpful interface that helps wallet fund replacement
>>> transactions that aim for a feerate and fee.
>>>
>>> ### A Different Model for Fees
>>>
>>> For incentive compatibility, I believe there are different
>>> formulations we should consider. Most importantly, if we want to get
>>> rid of the absolute fee rule, we can no longer think of it as "the
>>> transaction needs to pay for its own bandwidth," since we won't always
>>> be getting additional fees. That means we need a new method of
>>> rate-limiting replacements that doesn't require additional fees every
>>> time.
>>>
>>> While it makes sense to think about monetary costs when launching a
>>> specific type of attack, given that the fees are paid to the miner and
>>> not to the mempool operators, maybe it doesn't make much sense to
>>> think about "paying for bandwidth". Maybe we should implement
>>> transaction validation rate-limiting differently, e.g. building it
>>> into the P2P layer instead of the mempool policy layer.
>>>
>>> Recently, Suhas gave a [formulation][8] for incentive compatibility
>>> that made sense to me: "are the fees expected to be paid in the next
>>> (N?) blocks higher or lower if we process this transaction?"
>>>
>>> I started by thinking about this where N=1 or `1 + p`.
>>> Here, a rational miner is looking at what fees they would
>>> collect in the next block, and then some proportion `p` of the rest of
>>> the blocks based on their hashrate. We're assuming `p` isn't *so high*
>>> that they would be okay with lower absolute fees in the next 1 block.
>>> We're also assuming `p` isn't *so low* that the miner doesn't care
>>> about what's left of the mempool after this block.
>>>
>>> A tweak to this formulation is "if we process this transaction, would
>>> the fees in the next 1 block higher or lower, and is the feerate
>>> density of the rest of the mempool higher or lower?" This is pretty
>>> similar, where N=1, but we consider the rest of the mempool by feerate
>>> rather than fees.
>>>
>>> ### Mining Score of a Mempool Transaction
>>>
>>> We are often interested in finding out what
>>> the "mining score" of a transaction in the mempool is. That is, when
>>> the transaction is considered in block template building, what is the
>>> feerate it is considered at?
>>>
>>> Obviously, it's not the transaction's individual feerate. Bitcoin Core
>>> [mining code sorts][14] transactions by their ancestor feerate and
>>> includes them packages at a time, keeping track of how this affects the
>>> package feerates of remaining transactions in the mempool.
>>>
>>> *ancestor feerate*: Ancestor feerate is easily accessible information,
>>> but it's not accurate either, because it doesn't take into account the
>>> fact that subsets of a transaction's ancestor set can be included
>>> without it. For example, ancestors may have high feerates on their own
>>> or we may have [high feerate siblings][8].
>>>
>>> TLDR: *Looking at the current ancestor feerate of a transaction is
>>> insufficient to tell us what feerate it will be considered at when
>>> building a block template in the future.*
>>>
>>> *min(individual feerate, ancestor feerate)*: Another
>>> heuristic that is simple to calculate based on current mempool tooling
>>> is to use the [minimum of a transaction's individual score and its
>>> ancestor score][10] as a conservative measure. But this can
>>> overestimate as well (see the example below).
>>>
>>> *min ancestor feerate(tx + possible ancestor subsets)* We can also
>>> take the minimum of every possible ancestor subset, but this can be
>>> computationally expensive since there can be lots and lots of ancestor
>>> subsets.
>>>
>>> *max ancestor feerate(tx + possible descendant subsets)*: Another idea
>>> is to use the [maximum ancestor score of the transaction + each of its
>>> descendants][9]. This doesn't work either; it has the same blindspot
>>> of ancestor subsets being mined on their own.
>>>
>>> #### Mining Score Example
>>>
>>> Here's an example illustrating why mining score is tricky to
>>> efficiently calculate for mempool transactions:
>>>
>>> Let's say you have same-size transactions A (21sat/vB), B (1sat/vB),
>>> C(9sat/vB), D(5sat/vB).
>>> The layout is: grandparent A, parent B, and two children C and D.
>>>
>>> ```
>>> A
>>> ^
>>> B
>>> ^ ^
>>> C D
>>> ```
>>>
>>> A miner using ancestor packages to build block templates will first
>>> include A with a mining score of 21. Next, the miner will include B and
>>> C with a mining score of 6. This leaves D, with a mining score of 5.
>>>
>>> Note: in this case, mining by ancestor feerate results in the most
>>> rational decisions, but [a candidate set-based approach][10] which
>>> makes ancestor feerate much less relevant could
>>> be more advantageous in other situations.
>>>
>>> Here is a chart showing the "true" mining score alongside the values
>>> calculating using imperfect heuristics described above. All of them
>>> can overestimate or underestimate.
>>>
>>> ```
>>> A B C D
>>> mining score | 21 | 6 | 6 | 5 |
>>> ancestor feerate | 21 | 11 | 10.3 | 9 |
>>> min(individual, ancestor) | 21 | 1 | 9 | 5 |
>>> min(tx + ancestor subsets) | 21 | 1 | 5 | 3 |
>>> max(tx + descendants subsets) | 21 | 9 | 9 | 5 |
>>>
>>> ```
>>>
>>> Possibly the best solution for finding the "mining score" of a
>>> transaction is to build a block template, see what feerate each
>>> package is included at. Perhaps at some cutoff, remaining mempool
>>> transactions can be estimated using some heuristic that leans
>>> {overestimating, underestimating} depending on the situation.
>>>
>>> Mining score seems to be relevant in multiple places: Murch and I
>>> recently [found][3] that it would be very important in
>>> "ancestor-aware" funding of transactions (the wallet doesn't
>>> incorporate ancestor fees when using unconfirmed transactions in coin
>>> selection, which is a bug we want to fix).
>>>
>>> In general, it would be nice to know the exact mining priority of
>>> one's unconfirmed transaction is. I can think of a few block/mempool
>>> explorers who might want to display this information for users.
>>>
>>> ### RBF Improvement Proposals
>>>
>>> After speaking to quite a few people, here are some suggestions
>>> for improvements that I have heard:
>>>
>>> * The ancestor score of the replacement must be {5, 10, N}% higher
>>> than that of every original transaction.
>>>
>>> * The ancestor score of the replacement must be 1sat/vB higher than
>>> that of every original transaction.
>>>
>>> * If the original transaction is in the top {0.75MvB, 1MvB} of the
>>> mempool, apply the current rules (absolute fees must increase and
>>> pay for the replacement transaction's new bandwidth). Otherwise, use a
>>> feerate-only rule.
>>>
>>> * If fees don't increase, the size of the replacement transaction must
>>> decrease by at least N%.
>>>
>>> * Rate-limit how many replacements we allow per prevout.
>>>
>>> * Rate-limit transaction validation in general, per peer.
>>>
>>> Perhaps some others on the mailing list can chime in to throw other
>>> ideas into the ring and/or combine some of these rules into a sensible
>>> policy.
>>>
>>> #### Replace by Feerate Only
>>>
>>> I don't think there's going to be a single-line feerate-based
>>> rule that can incorporate everything we need.
>>> On one hand, a feerate-only approach helps eliminate the issues
>>> associated with Rule #3. On the other hand, I believe the main concern
>>> with a feerate-only approach is how to rate limit replacements. We
>>> don't want to enable an attack such as:
>>>
>>> 1. Attacker broadcasts large, low-feerate transaction, and attaches a
>>> chain of descendants.
>>>
>>> 2. The attacker replaces the transaction with a smaller but higher
>>> feerate transaction, attaching a new chain of descendants.
>>>
>>> 3. Repeat 1000 times.
>>>
>>> #### Fees in Next Block and Feerate for the Rest of the Mempool
>>>
>>> Perhaps we can look at replacements like this:
>>>
>>> 1. Calculate the directly conflicting transactions and, with their
>>> descendants, the original transactions. Check signaling. Limit the
>>> total volume (e.g. can't be more than 100 total or 1MvB or something).
>>>
>>> 2. Find which original transactions would be in the next ~1 block. The
>>> replacement must pay at least this amount + X% in absolute fees. This
>>> guarantees that the fees of the next block doesn't decrease.
>>>
>>> 3. Find which transactions would be left in the mempool after that ~1
>>> block. The replacement's feerate must be Y% higher than the maximum
>>> mining score of these transactions. This guarantees that you now have
>>> only *better* candidates in your after-this-block mempool than you did
>>> before, even if the size and fees the transactions decrease.
>>>
>>> 4. Now you have two numbers: a minimum absolute fee amount and a
>>> minimum feerate. Check to see if the replacement(s) meet these
>>> minimums. Also, a wallet would be able to ask the node "What fee and
>>> feerate would I need to put on a transaction replacing this?" and use
>>> this information to fund a replacement transaction, without needing to
>>> guess or overshoot.
>>>
>>> Obviously, there are some magic numbers missing here. X and Y are
>>> TBD constants to ensure we have some kind of rate limiting for the
>>> number of replacements allowed using some set of fees.
>>>
>>> What should they be? We can do some arithmetic to see what happens if
>>> you start with the biggest/lowest feerate transaction and do a bunch
>>> of replacements. Maybe we end up with values that are high enough to
>>> prevent abuse and make sense for applications/users that do RBF.
>>>
>>> ### Mempool Changes Need for Implementation
>>>
>>> As described in the mining score section above,
>>> we may want additional tooling to more accurately assess
>>> the economic gain of replacing transactions in our mempool.
>>>
>>> A few options have been discussed:
>>>
>>> * Calculate block templates on the fly when we need to consider a
>>> replacement. However, since replacements are [quite common][11]
>>> and the information might be useful for other things as well,
>>> it may be worth it to cache a block template.
>>>
>>> * Keep a persistent block template so that we know what transactions
>>> we would put in the next block. We need to remember the feerate
>>> at which each transaction was included in the template, because an
>>> ancestor package may be included in the same block template in
>>> multiple subsets. Transactions included earlier alter the ancestor
>>> feerate of the remaining transactions in the package. We also need
>>> to keep track of the new feerates of transactions left over.
>>>
>>> * Divide the mempool into two layers, "high feerate" and "low
>>> feerate." The high feerate layer contains ~1 block of packages with
>>> the highest ancestor feerates, and the low feerate layer contains
>>> everything else. At the edge of a block, we have a Knapsacky problem
>>> where the next highest ancestor feerate package might not fit, so we
>>> would probably want the high feerate layer ~2MvB or something to avoid
>>> underestimating the fees.
>>>
>>> ## Acknowledgements
>>>
>>> Thank you to everyone whose RBF-related suggestions, grievances,
>>> criticisms and ideas were incorporated in this document:
>>> Andrew Chow, Matt Corallo, Suhas Daftuar, Christian Decker,
>>> Mark Erhardt, Lloyd Fournier, Lisa Neigut, John Newbery,
>>> Antoine Poinsot, Antoine Riard, Larry Ruane,
>>>
>>> S3RK and Bastien Teinturier.
>>>
>>> Thanks for reading!
>>>
>>> Best,
>>> Gloria
>>> [1]: https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replacements.md
>>> [2]: https://github.com/bitcoin/bitcoin/pull/23121#issuecomment-929475999
>>> [3]: https://github.com/Xekyo/bitcoin/commit/d754b0242ec69d42c570418aebf9c1335af0b8ea
>>> [4]: https://github.com/bitcoindevkit/bdk/issues/144
>>> [5]: https://github.com/bitcoindevkit/bdk/issues/414
>>> [6]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>>> [7]: https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a#new-unconfirmed-inputs-rule-2
>>> [8]: https://github.com/bitcoin/bitcoin/pull/23121#discussion_r777131366
>>> [9]: https://github.com/bitcoin/bitcoin/pull/22290#issuecomment-865887922
>>> [10]: https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
>>> [11]: https://github.com/bitcoin/bitcoin/pull/22539#issuecomment-885763670
>>> [12]: https://github.com/bitcoin/bitcoin/pull/24007
>>> [13]: https://github.com/bitcoin/bitcoin/blob/1a369f006fd0bec373b95001ed84b480e852f191/src/wallet/feebumper.cpp#L114
>>>
>>> [14]: https://github.com/bitcoin/bitcoin/blob/cf5bb048e80d4cde8828787b266b7f5f2e3b6d7b/src/node/miner.cpp#L310-L320
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev at lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20220205/3641f38d/attachment-0001.html>