{"type":"rich","version":"1.0","author_name":"npub1j66aek8wm7jq8vaffs2l8w43evyywd4q7tcnqyge6xqezz0vcpks7jm42m","author_url":"https://nostr.ae/npub1j66aek8wm7jq8vaffs2l8w43evyywd4q7tcnqyge6xqezz0vcpks7jm42m","provider_name":"njump","provider_url":"https://nostr.ae","html":"📅 Original date posted:2016-05-07\n📝 Original message:There was some confusion over the following email which was posted to the list\nwhich appears to have been cancelled before a decision could be reached.\n\nPlease note the email seems inflammatory in the \"acknowledgement\" section and\nreally should have been rewritten to contain specific details of the objection\nand corrections expected.\n\nTo be clear posts to the mailing list are either approved, or rejected for not\nmeeting the posting standards. This allows the author to make a quick correction\nand resubmit. All rejections are cc'd to\nhttps://lists.ozlabs.org/pipermail/bitcoin-dev-moderation/\nfor transparency. Sometimes moderators get delayed - this week has been a busy\nwith lots of distractions one for everyone :)\n\nI'm copying the entire message below:\n\n---------- Forwarded message ----------\nFrom: Tom \u003ctomz at freedommail.ch\u003e\nTo: bitcoin-dev at lists.linuxfoundation.org, Matt Corallo\n\u003clf-lists at mattcorallo.com\u003e\nCc:\nDate: Fri, 06 May 2016 13:31:15 +0100\nSubject: Re: [bitcoin-dev] Compact Block Relay BIP\nOn Monday 02 May 2016 22:13:22 Matt Corallo via bitcoin-dev wrote:\n\nThanks for putting in the time to make a spec!\n\nIt looks good already, but I do think some more improvements can be made.\n\n\n\u003e ===Intended Protocol Flow===\nI'm not a fan of the solution that a CNode should keep state and talk to\nits remote nodes differently while announcing new blocks.\nIts too complicated and ultimately counter-productive.\n\nThe problem is that an individual node needs to predict network behaviour in\nadvance. With the downside that if it guesses wrong that both nodes end up\npaying for the wrong guess.\nThis is not a good way to design a p2p layer.\n\n\n\nI would suggest that a new block is announced to all nodes equally and then\nindividual nodes can respond with a request of either a 'compact' or a\nnormal block.\nThis is much more in line with the current design as well.\n\nDetection if remote nodes support compact blocks, for the purpose of\nrequesting a compact-block, can be done either via a network-bit or just a\nprotocol version. Or something else entirely, if you have better\nsuggestions.\n\n\n\n\u003e Variable-length integers: bytes are a MSB base-128 encoding of the\n\u003e number.\n\u003e The high bit in each byte signifies whether another digit follows.\n\u003e [snip bitwise spec]\n\nI suggest just referring to UTF-8 which describes this just fine.\nit is good practice to refer to existing specs when possible and not copy\nthe details.\n\n\u003e ====Short transaction IDs====\n\u003e Short transaction IDs are used to represent a transaction without\n\u003e sending a full 256-bit hash. They are calculated by:\n\u003e # single-SHA256 hashing the block header with the nonce appended (in\n\u003e little-endian)\n\u003e # XORing each 8-byte chunk of the double-SHA256 transaction hash with\n\u003e each corresponding 8-byte chunk of the hash from the previous step\n\u003e # Adding each of the XORed 8-byte chunks together (in little-endian)\n\u003e iteratively to find the short transaction ID\n\nI don't think this is needed. Just use the first 8 bytes.\nThe reason to do xor-ing doesn't hold up and extra complexity is unneeded.\nEspecially since you mention some lines down;\n\n\u003e The short transaction ID calculation is designed to take absolutely\n\u003e minimal processing time during block compaction to avoid introducing\n\u003e serious DoS vulnerabilities\n\n\n==Acknowledgements==\n\nI think you need to acknowledge some more people, or just remove this\nparagraph.\n\nCheers\n\n\n---------- Forwarded message ----------\nFrom: bitcoin-dev-request at lists.linuxfoundation.org\nTo:\nCc:\nDate: Fri, 06 May 2016 12:31:23 +0000\nSubject: confirm 37d25406a07ab77823fba5f9b450438c410ccd75\nIf you reply to this message, keeping the Subject: header intact,\nMailman will discard the held message.  Do this if the message is\nspam.  If you reply to this message and include an Approved: header\nwith the list password in it, the message will be approved for posting\nto the list.  The Approved: header can also appear in the first line\nof the body of the reply.\n\n\nOn Mon, May 2, 2016 at 3:13 PM, Matt Corallo via bitcoin-dev \u003c\nbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\n\u003e Hi all,\n\u003e\n\u003e The following is a BIP-formatted design spec for compact block relay\n\u003e designed to limit on wire bytes during block relay. You can find the\n\u003e latest version of this document at\n\u003e https://github.com/TheBlueMatt/bips/blob/master/bip-TODO.mediawiki.\n\u003e\n\u003e There are several TODO items left on the document as indicated.\n\u003e Additionally, the implementation linked at the bottom of the document\n\u003e has a few remaining TODO items as well:\n\u003e\n\u003e  * Only request compact-block-announcement from one or two peers at a\n\u003e time, as the spec requires.\n\u003e  * Request new blocks using MSG_CMPCT_BLOCK where appropriate.\n\u003e  * Fill prefilledtxn with more than just the coinbase, as noted by the\n\u003e spec, up to 10K in transactions.\n\u003e\n\u003e Luke (CC'd): Can you assign a BIP number?\n\u003e\n\u003e Thanks,\n\u003e Matt\n\u003e\n\u003e \u003cpre\u003e\n\u003e   BIP: TODO\n\u003e   Title: Compact block relay\n\u003e   Author: Matt Corallo \u003cbip at bluematt.me\u003e\n\u003e   Status: Draft\n\u003e   Type: Standards Track\n\u003e   Created: 2016-04-27\n\u003e \u003c/pre\u003e\n\u003e\n\u003e ==Abstract==\n\u003e\n\u003e Compact blocks on the wire as a way to save bandwidth for nodes on the\n\u003e P2P network.\n\u003e\n\u003e The key words \"MUST\", \"MUST NOT\", \"REQUIRED\", \"SHALL\", \"SHALL NOT\",\n\u003e \"SHOULD\", \"SHOULD NOT\", \"RECOMMENDED\", \"MAY\", and \"OPTIONAL\" in this\n\u003e document are to be interpreted as described in RFC 2119.\n\u003e\n\u003e ==Motivation==\n\u003e\n\u003e Historically, the Bitcoin P2P protocol has not been very bandwidth\n\u003e efficient for block relay. Every transaction in a block is included when\n\u003e relayed, even though a large number of the transactions in a given block\n\u003e are already available to nodes before the block is relayed. This causes\n\u003e moderate inbound bandwidth spikes for nodes when receiving blocks, but\n\u003e can cause very significant outbound bandwidth spikes for some nodes\n\u003e which receive a block before their peers. When such spikes occur, buffer\n\u003e bloat can make consumer-grade internet connections temporarily unusable,\n\u003e and can delay the relay of blocks to remote peers who may choose to wait\n\u003e instead of redundantly requesting the same block from other, less\n\u003e congested, peers.\n\u003e\n\u003e Thus, decreasing the bandwidth used during block relay is very useful\n\u003e for many individuals running nodes.\n\u003e\n\u003e While the goal of this work is explicitly not to reduce block transfer\n\u003e latency, it does, as a side effect reduce block transfer latencies in\n\u003e some rather significant ways. Additionally, this work forms a foundation\n\u003e for future work explicitly targeting low-latency block transfer.\n\u003e\n\u003e ==Specification==\n\u003e\n\u003e ===Intended Protocol Flow===\n\u003e TODO: Diagrams\n\u003e\n\u003e The protocol is intended to be used in two ways, depending on the peers\n\u003e and bandwidth available, as discussed [[#Implementation_Details|later]].\n\u003e The \"high-bandwidth\" mode, which nodes may only enable for a few of\n\u003e their peers, is enabled by setting the first boolean to 1 in a\n\u003e \"sendcmpct\" message. In this mode, peers send new block announcements\n\u003e with the short transaction IDs already, possibly even before fully\n\u003e validating the block. In some cases no further round-trip is needed, and\n\u003e the receiver can reconstruct the block and process it as usual\n\u003e immediately. When some transactions were not available from local\n\u003e sources (ie mempool), a getblocktxn/blocktxn roundtrip is neccessary,\n\u003e bringing the best-case latency to the same 1.5*RTT minimum time that\n\u003e nodes take today, though with significantly less bandwidth usage.\n\u003e\n\u003e The \"low-bandwidth\" mode is enabled by setting the first boolean to 0 in\n\u003e a \"sendcmpct\" message. In this mode, peers send new block announcements\n\u003e with the usual inv/headers announcements (as per BIP130, and after fully\n\u003e validating the block). The receiving peer may then request the block\n\u003e using a MSG_CMPCT_BLOCK getdata reqeuest, which will receive a response\n\u003e of the header and short transaction IDs. In some cases no further\n\u003e round-trip is needed, and the receiver can reconstruct the block and\n\u003e process it as usual, taking the same 1.5*RTT minimum time that nodes\n\u003e take today, though with significantly less bandwidth usage. When some\n\u003e transactions were not available from local sources (ie mempool), a\n\u003e getblocktxn/blocktxn roundtrip is neccessary, bringing the best-case\n\u003e latency to 2.5*RTT, again with significantly less bandwidth usage than\n\u003e today. Because TCP often exhibits worse transfer latency for larger data\n\u003e sizes (as a multiple of RTT), total latency is expected to be reduced\n\u003e even when full the 2.5*RTT transfer mechanism is used.\n\u003e\n\u003e ===New data structures===\n\u003e Several new data structures are added to the P2P network to relay\n\u003e compact blocks: PrefilledTransaction, HeaderAndShortIDs,\n\u003e BlockTransactionsRequest, and BlockTransactions. Additionally, we\n\u003e introduce a new variable-length integer encoding for use in these data\n\u003e structures.\n\u003e\n\u003e For the purposes of this section, CompactSize refers to the\n\u003e variable-length integer encoding used across the existing P2P protocol\n\u003e to encode array lengths, among other things, in 1, 3, 5 or 9 bytes.\n\u003e\n\u003e ====New VarInt====\n\u003e TODO: I just copied this out of the src...Something that is\n\u003e wiki-formatted and more descriptive should be used here isntead.\n\u003e\n\u003e Variable-length integers: bytes are a MSB base-128 encoding of the number.\n\u003e The high bit in each byte signifies whether another digit follows. To make\n\u003e sure the encoding is one-to-one, one is subtracted from all but the last\n\u003e digit.\n\u003e Thus, the byte sequence a[] with length len, where all but the last byte\n\u003e has bit 128 set, encodes the number:\n\u003e\n\u003e (a[len-1] \u0026 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] \u0026 0x7F)+1))\n\u003e\n\u003e Properties:\n\u003e * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes)\n\u003e * Every integer has exactly one encoding\n\u003e * Encoding does not depend on size of original integer type\n\u003e * No redundancy: every (infinite) byte sequence corresponds to a list\n\u003e   of encoded integers.\n\u003e\n\u003e 0:         [0x00]  256:        [0x81 0x00]\n\u003e 1:         [0x01]  16383:      [0xFE 0x7F]\n\u003e 127:       [0x7F]  16384:      [0xFF 0x00]\n\u003e 128:  [0x80 0x00]  16511: [0x80 0xFF 0x7F]\n\u003e 255:  [0x80 0x7F]  65535: [0x82 0xFD 0x7F]\n\u003e 2^32:           [0x8E 0xFE 0xFE 0xFF 0x00]\n\u003e\n\u003e Several uses of New VarInts below are \"differentially encoded\". For\n\u003e these, instead of using raw indexes, the number encoded is the\n\u003e difference between the current index and the previous index, minus one.\n\u003e For example, a first index of 0 implies a real index of 0, a second\n\u003e index of 0 thereafter refers to a real index of 1, etc.\n\u003e\n\u003e ====PrefilledTransaction====\n\u003e A PrefilledTransaction structure is used in HeaderAndShortIDs to provide\n\u003e a list of a few transactions explicitly.\n\u003e\n\u003e {|\n\u003e |Field Name||Type||Size||Encoding||Purpose\n\u003e |-\n\u003e |index||New VarInt||1-3 bytes||[[#New_VarInt|New VarInt]],\n\u003e differentially encoded since the last PrefilledTransaction in a\n\u003e list||The index into the block at which this transaction is\n\u003e |-\n\u003e |tx||Transaction||variable||As encoded in \"tx\" messages||The transaction\n\u003e which is in the block at index index.\n\u003e |}\n\u003e\n\u003e ====HeaderAndShortIDs====\n\u003e A HeaderAndShortIDs structure is used to relay a block header, the short\n\u003e transactions IDs used for matching already-available transactions, and a\n\u003e select few transactions which we expect a peer may be missing.\n\u003e\n\u003e {|\n\u003e |Field Name||Type||Size||Encoding||Purpose\n\u003e |-\n\u003e |header||Block header||80 bytes||First 80 bytes of the block as defined\n\u003e by the encoding used by \"block\" messages||The header of the block being\n\u003e provided\n\u003e |-\n\u003e |nonce||uint64_t||8 bytes||Little Endian||A nonce for use in short\n\u003e transaction ID calculations\n\u003e |-\n\u003e |shortids_length||CompactSize||1, 3, 5, or 9 bytes||As used elsewhere to\n\u003e encode array lengths||The number of short transaction IDs in shortids\n\u003e |-\n\u003e |shortids||List of uint64_ts||8*shortids_length bytes||Little\n\u003e Endian||The short transaction IDs calculated from the transactions which\n\u003e were not provided explicitly in prefilledtxn\n\u003e |-\n\u003e |prefilledtxn_length||CompactSize||1, 3, 5, or 9 bytes||As used\n\u003e elsewhere to encode array lengths||The number of prefilled transactions\n\u003e in prefilledtxn\n\u003e |-\n\u003e |prefilledtxn||List of PrefilledTransactions||variable\n\u003e size*prefilledtxn_length||As defined by PrefilledTransaction definition,\n\u003e above||Used to provide the coinbase transaction and a select few which\n\u003e we expect a peer may be missing\n\u003e |}\n\u003e\n\u003e ====BlockTransactionsRequest====\n\u003e A BlockTransactionsRequest structure is used to list transaction indexes\n\u003e in a block being requested.\n\u003e\n\u003e {|\n\u003e |Field Name||Type||Size||Encoding||Purpose\n\u003e |-\n\u003e |blockhash||Binary blob||32 bytes||The output from a double-SHA256 of\n\u003e the block header, as used elsewhere||The blockhash of the block which\n\u003e the transactions being requested are in\n\u003e |-\n\u003e |indexes_length||New VarInt||1-3 bytes||As defined in [[#New_VarInt|New\n\u003e VarInt]]||The number of transactions being requested\n\u003e |-\n\u003e |indexes||List of New VarInts||1-3 bytes*indexes_length||As defined in\n\u003e [[#New_VarInt|New VarInt]], differentially encoded||The indexes of the\n\u003e transactions being requested in the block\n\u003e |}\n\u003e\n\u003e ====BlockTransactions====\n\u003e A BlockTransactions structure is used to provide some of the\n\u003e transactions in a block, as requested.\n\u003e\n\u003e {|\n\u003e |Field Name||Type||Size||Encoding||Purpose\n\u003e |-\n\u003e |blockhash||Binary blob||32 bytes||The output from a double-SHA256 of\n\u003e the block header, as used elsewhere||The blockhash of the block which\n\u003e the transactions being provided are in\n\u003e |-\n\u003e |transactions_length||New VarInt||1-3 bytes||As defined in\n\u003e [[#New_VarInt|New VarInt]]||The number of transactions provided\n\u003e |-\n\u003e |transactions||List of Transactions||variable||As encoded in \"tx\"\n\u003e messages||The transactions provided\n\u003e |}\n\u003e\n\u003e ====Short transaction IDs====\n\u003e Short transaction IDs are used to represent a transaction without\n\u003e sending a full 256-bit hash. They are calculated by:\n\u003e # single-SHA256 hashing the block header with the nonce appended (in\n\u003e little-endian)\n\u003e # XORing each 8-byte chunk of the double-SHA256 transaction hash with\n\u003e each corresponding 8-byte chunk of the hash from the previous step\n\u003e # Adding each of the XORed 8-byte chunks together (in little-endian)\n\u003e iteratively to find the short transaction ID\n\u003e\n\u003e ===New messages===\n\u003e A new inv type (MSG_CMPCT_BLOCK == 4) and several new protocol messages\n\u003e are added: sendcmpct, cmpctblock, getblocktxn, and blocktxn.\n\u003e\n\u003e ====sendcmpct====\n\u003e # The sendcmpct message is defined as a message containing a 1-byte\n\u003e integer followed by a 8-byte integer where pchCommand == \"sendcmpct\".\n\u003e # The first integer SHALL be interpreted as a boolean (and MUST have a\n\u003e value of either 1 or 0)\n\u003e # The second integer SHALL be interpreted as a little-endian version\n\u003e number. Nodes sending a sendcmpct message MUST currently set this value\n\u003e to 1.\n\u003e # Upon receipt of a \"sendcmpct\" message with the first and second\n\u003e integers set to 1, the node SHOULD announce new blocks by sending a\n\u003e cmpctblock message.\n\u003e # Upon receipt of a \"sendcmpct\" message with the first integer set to 0,\n\u003e the node SHOULD NOT announce new blocks by sending a cmpctblock message,\n\u003e but SHOULD announce new blocks by sending invs or headers, as defined by\n\u003e BIP130.\n\u003e # Upon receipt of a \"sendcmpct\" message with the second integer set to\n\u003e something other than 1, nodes SHOULD treat the peer as if they had not\n\u003e received the message (as it indicates the peer will provide an\n\u003e unexpected encoding in cmpctblock, and/or other, messages)\n\u003e # Nodes SHOULD check for a protocol version of \u003e= 70014 before sending\n\u003e sendcmpct messages.\n\u003e # Nodes MUST NOT send a request for a MSG_CMPCT_BLOCK object to a peer\n\u003e before having received a sendcmpct message from that peer.\n\u003e\n\u003e ====MSG_CMPCT_BLOCK====\n\u003e # getdata messages may now contain requests for MSG_CMPCT_BLOCK objects.\n\u003e # Upon receipt of a getdata containing a request for a MSG_CMPCT_BLOCK\n\u003e object with the hash of a block which was recently announced and after\n\u003e having sent the requesting peer a sendcmpct message, nodes MUST respond\n\u003e with a cmpctblock message containing appropriate data representing the\n\u003e block being requested.\n\u003e # MSG_CMPCT_BLOCK inv objects MUST NOT appear anywhere except for in\n\u003e getdata messages.\n\u003e\n\u003e ====cmpctblock====\n\u003e # The cmpctblock message is defined as as a message containing a\n\u003e serialized HeaderAndShortIDs message and pchCommand == \"cmpctblock\".\n\u003e # Upon receipt of a cmpctblock message after sending a sendcmpct\n\u003e message, nodes SHOULD calculate the short transaction ID for each\n\u003e unconfirmed transaction they have available (ie in their mempool) and\n\u003e compare each to each short transaction ID in the cmpctblock message.\n\u003e # After finding already-available transactions, nodes which do not have\n\u003e all transactions available to reconstruct the full block SHOULD request\n\u003e the missing transactions using a getblocktxn message.\n\u003e # A node MUST NOT send a cmpctblock message unless they are able to\n\u003e respond to a getblocktxn message which requests every transaction in the\n\u003e block.\n\u003e # A node MUST NOT send a cmpctblock message without having validated\n\u003e that the header properly commits to each transaction in the block, and\n\u003e properly builds on top of the existing chain with a valid proof-of-work.\n\u003e A node MAY send a cmpctblock before validating that each transaction in\n\u003e the block validly spends existing UTXO set entries.\n\u003e\n\u003e ====getblocktxn====\n\u003e # The getblocktxn message is defined as as a message containing a\n\u003e serialized BlockTransactionsRequest message and pchCommand ==\n\u003e \"getblocktxn\".\n\u003e # Upon receipt of a properly-formatted getblocktxnmessage, nodes which\n\u003e recently provided the sender of such a message a cmpctblock for the\n\u003e block hash identified in this message MUST respond with an appropriate\n\u003e blocktxn message. Such a blocktxn message MUST contain exactly and only\n\u003e each transaction which is present in the appropriate block at the index\n\u003e specified in the getblocktxn indexes list, in the order requested.\n\u003e\n\u003e ====blocktxn====\n\u003e # The blocktxn message is defined as as a message containing a\n\u003e serialized BlockTransactions message and pchCommand == \"blocktxn\".\n\u003e # Upon receipt of a properly-formatted requested blocktxn message, nodes\n\u003e SHOULD attempt to reconstruct the full block by:\n\u003e ## Taking the prefilledtxn transactions from the original cmpctblock and\n\u003e placing them in the marked positions.\n\u003e ## For each short transaction ID from the original cmpctblock, in order,\n\u003e find the corresponding transaction either from the blocktxn message or\n\u003e from other sources and place it in the first available position in the\n\u003e block.\n\u003e # Once the block has been reconstructed, it shall be processed as\n\u003e normal, keeping in mind that short transaction IDs are expected to\n\u003e occasionally collide, and that nodes MUST NOT be penalized for such\n\u003e collisions, wherever they appear.\n\u003e\n\u003e ===Implementation Notes===\n\u003e # For nodes which have sufficient inbound bandwidth, sending a sendcmpct\n\u003e message with the first integer set to 1 to up to three peers is\n\u003e RECOMMENDED. If possible, it is RECOMMENDED that those peers be selected\n\u003e based on their past performance in providing blocks quickly. This will\n\u003e allow them to receive some blocks in only 0.5*RTT between them and the\n\u003e sending peer. It will also reduce their block transfer latency in other\n\u003e cases due to the smaller amount of data transmitted. Nodes MUST NOT send\n\u003e such sendcmpct messages to all peers, as it encourages wasting outbound\n\u003e bandwidth across the network.\n\u003e\n\u003e # All nodes SHOULD send a sendcmpct message to all appropriate peers.\n\u003e This will reduce their outbound bandwidth usage by allowing their peers\n\u003e to request compact blocks instead of full blocks.\n\u003e\n\u003e # Nodes with limited inbound bandwidth SHOULD request blocks using\n\u003e MSG_CMPCT_BLOCK/getblocktxn requests, when possible. While this\n\u003e increases worst-case message round-trips, it is expected to reduce\n\u003e overall transfer latency as TCP is more likely to exhibit poor\n\u003e throughput on low-bandwidth nodes.\n\u003e\n\u003e # Nodes sending cmpctblock messages SHOULD make an attempt to not place\n\u003e too many transactions into prefilledtxn (ie should limit prefilledtxn to\n\u003e only around 10KB of transactions). When in doubt, nodes SHOULD only\n\u003e include the coinbase transaction in prefilledtxn.\n\u003e\n\u003e # Nodes MAY pick one nonce per block they wish to send, and only build a\n\u003e cmpctblock message once for all peers which they wish to send a given\n\u003e block to. Nodes SHOULD NOT use the same nonce across multiple different\n\u003e blocks.\n\u003e\n\u003e # Nodes MAY impose additional requirements on when they announce new\n\u003e blocks by sending cmpctblock messages. For example, nodes with limited\n\u003e outbound bandwidth MAY choose to announce new blocks using inv/header\n\u003e messages (as per BIP130) to conserve outbound bandwidth.\n\u003e\n\u003e # Note that the MSG_CMPCT_BLOCK section does not require that nodes\n\u003e respond to MSG_CMPCT_BLOCK getdata requests for blocks which they did\n\u003e not recently announce. This allows nodes to calculate cmpctblock\n\u003e messages at announce-time instead of at request-time. Thus, nodes MUST\n\u003e NOT request blocks using MSG_CMPCT_BLOCK getdatas unless it is in\n\u003e response to an inv/headers block announcement (as per BIP130), and MUST\n\u003e NOT request blocks using MSG_CMPCT_BLOCK getdatas in response to headers\n\u003e messages which were, themselves, responses to getheaders requests.\n\u003e\n\u003e # While the current version sends transactions with the same encodings\n\u003e as is used in tx messages and elsewhere in the protocol, the version\n\u003e field in sendcmpct is intended to allow this to change in the future.\n\u003e For this reason, it is recommended that the code used to decode\n\u003e PrefilledTransaction and BlockTransactions messages be prepared to take\n\u003e a different transaction encoding, if and when the version field in\n\u003e sendcmpct changes in a future BIP.\n\u003e\n\u003e ==Justification==\n\u003e\n\u003e ====Protocol design====\n\u003e There have been many proposals to save wire bytes when relaying blocks.\n\u003e Many of them have a two-fold goal of reducing block relay time and thus\n\u003e rely on the use of significant processing power in order to avoid\n\u003e introducing additional worst-case RTTs. Because this work is not focused\n\u003e primarily on reducing block relay time, its design is much simpler (ie\n\u003e does not rely on set reconciliation protocols). Still, in testing at the\n\u003e time of writing, nodes are able to relay blocks without the extra\n\u003e getblocktxn/blocktxn RTT around 90% of the time. With a smart\n\u003e compact-block-announcement policy, it is thus expected that this work\n\u003e might allow blocks to be relayed between nodes in 0.5*RTT instead of\n\u003e 1.5*RTT at least 75% of the time.\n\u003e\n\u003e ====Use of New VarInts====\n\u003e Bitcoin has long had a variable-length integer implementation (referred\n\u003e to as CompactSize in this document), making a second a strange protocol\n\u003e quirk. However, in this protocol most of our variable-length integers\n\u003e are between 0 and 2000. For both encodings, small numbers (\u003c100) are\n\u003e encoded as 1-byte. For numbers over 250, the CompactSize encoding begins\n\u003e to use 3 bytes instead of 1, whereas the New VarInt encoding uses 2.\n\u003e Because the primary motivation for this work is to save bytes during\n\u003e block relay, the extra byte of saving per transaction-difference is\n\u003e considered worth the extra design complexity.\n\u003e\n\u003e ====Short transaction ID calculation====\n\u003e The short transaction ID calculation is designed to take absolutely\n\u003e minimal processing time during block compaction to avoid introducing\n\u003e serious DoS vulnerabilities such as those introduced by the\n\u003e bloom-filtering in BIP 37. As such, it is possible for a node to\n\u003e construct one compact-block representation of a block for relay to\n\u003e multiple peers. Additionally, only one cryptographic hash (2 SHA rounds)\n\u003e is used when calculating the short transaction IDs for an entire block.\n\u003e\n\u003e The XOR-and-add method is used for calculating short transaction IDs\n\u003e primarily because it is fast and is reasonably able to limit the ability\n\u003e of an attacker who does not know the block hash or nonce to cause\n\u003e collisions in short transaction IDs. If an attacker were able to cause\n\u003e such collisions, filling mempools (and, thus, blocks) with them would\n\u003e cause poor network propagation of new (or non-attacker, in the case of a\n\u003e miner) blocks.\n\u003e\n\u003e The 8-byte nonce in short transaction ID calculation is used to\n\u003e introduce additional entropy on a per-node level. While the use of 8\n\u003e bytes is sufficient for an attacker to maliciously cause short\n\u003e transaction ID collisions in their own block relay, this would have less\n\u003e of an effect than if such an attacker were relaying headers/invs and not\n\u003e responding to requests for the full block.\n\u003e\n\u003e ==Backward compatibility==\n\u003e\n\u003e Older clients remain fully compatible and interoperable after this change.\n\u003e\n\u003e ==Implementation==\n\u003e\n\u003e https://github.com/TheBlueMatt/bitcoin/tree/udp\n\u003e\n\u003e ==Acknowledgements==\n\u003e\n\u003e Thanks to Gregory Maxwell for the initial suggestion as well as a lot of\n\u003e back-and-forth design and significant testing.\n\u003e\n\u003e ==Copyright==\n\u003e\n\u003e This document is placed in the public domain.\n\u003e\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\n\n\n-- \nJohnathan Corgan\nCorgan Labs - SDR Training and Development Services\nhttp://corganlabs.com\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160507/028cac26/attachment-0001.html\u003e"}