{"type":"rich","version":"1.0","author_name":"npub15fmnxm546tg2sv0l7elusrvqsgezdzdg3m0flpml5fr2qf3f5slskxlq6h","author_url":"https://nostr.ae/npub15fmnxm546tg2sv0l7elusrvqsgezdzdg3m0flpml5fr2qf3f5slskxlq6h","provider_name":"njump","provider_url":"https://nostr.ae","html":"📅 Original date posted:2011-12-21\n🗒️ Summary of this message: Bitcoin faces scalability issues due to the rapid growth of the blockchain. Partitioning the transaction/block space through a Distributed Hash Table (DHT) approach can provide distributed validation and storage. However, ensuring trust and security while querying other nodes for validation and avoiding double spending remains a challenge. Designing a system based on assigning clients to serve a part of the block chain, block index, and bitcoin addresses can help address these issues.\n📝 Original message:DHTs and Bitcoin:\n\nFirst, lets define the problem we want to solve: scalability - when bitcoin takes over all credit card transactions (!), and even before that, we will meet a scalability problem. The blockchain will grow rapidly, (1MB/10min or 50GB/yr) and we will constantly have transactions pending to get into a block. Further, the clients will turn into toasters just from validating all transactions. At the same time we have a level of validation and block chain distribution that we really don't need - today txes are validated by 100k clients in the future that could be 100M clients, and they are stored at way more locations than they are today. So... all this calls for a partition of the transaction/block space, and for a more flexible than 1MB / block setup.\n\nFirst things first. The partitioning of the tx space. One way to partition the tx space is through a partition in hash, namely the DHT approach. There might be other schemes, but as we already have both the ability to share addresses and maintain a hash space it seems obvious.\n\nSo we would like a scheme that provides distributed validation and storage keeping a similar level of trust and security as we have today. We hence need to be able to query another node for validation and ensure it is not pulling our leg (Sybil...).\n\nThere are two important aspects of bitcoin:\n1. transaction signing / validation\n2. to avoid double spending\n\n1. Is a a simple and inclusive problem to solve, 2 is more complex and exclusive. 1. can to a large extend be solved by asking for transactions and validating these against the block chain - it is hard to cheat as you can match blocks containing your transaction with the block chain headers, requiring a false node to perform heavy proof of work tasks.\nIf we on the other hand query other nodes for 2. just blocking an answer would be enough to enable a double spend. (at least seen from the one node querying).\n\nToday you can, assuming you have en up to date block chain, only block pending tx'es which gives you an approximate 10 minutes scale for cheating by double spending. If we create a setup where we distribute the block index and the block chain, we can fake any older transaction as well, and leave a node to believe that a tx has not been spend. The obvious way around it is to ensure a high level of connectedness and to query several geographically distributed nodes if a tx has already been spend. But this can be quite hard and also, you don't want to flood the network with to many extra commands.\n\nIf we design the system based on the above conclusions we get:\n\n1. A client is, based on the hash of its ip:port assigned to serve a part of the block chain, a part of the block index and possibly also a part of the bitcoin addresses (hash160).\n\n2. Further, the client can announce that it also serves any other hashspace fractions - e.g. to enable notification of payments to its bitcoin address or use of its coins (txouts).\n\n3. On validation of a tx, the txins are queried for at the clients serving these and a possible double spend can be monitored. We need to query more clients to ensure we are not cheated by one. And we need to maintain the requirement that they come from separate A.B address spaces (so they don't just setup a matching hash from playing with C.D and ports).\n\n4. The proper nodes are found using Chord DHT scheme (other schemes might be suitable as well).\n\nThin clients keep their spendable coins to a minimum and use only one bitcoin address, that way they will only serve and listen to 3 hash fractions. If we split the current space into 4096 parts we get roughly 100 clients for each hash space.\n\nThe (only?) new attack vector, compared to the current system is the possibility that a client has only evil peers within one hash range and hence can be fooled into believing an old tx can be spend again.\n\nThe new scheme will scale well as each client will only serve a part of the hashspace and hence the number of validations and block storage can be kept at a minimum. Further, it scales well for thin clients vs more full clients as you can add as many or as few (down to 1-3) hash space parts as you want, so the new scheme includes the old scheme in the limit of subscribing to all hash space parts.\n\nI might have overlooked something - so please fill in some comments...\n\nCheers,\n\nMichael\n\n\nOn 18/12/2011, at 22:19, Stefan Thomas wrote:\n\n\u003e Hey Chris,\n\u003e \n\u003e\u003e The storage would be distributed, messages are routed on behalf of others, which makes finding the origin of the query hard to find (think Tor)\n\u003e \n\u003e This type of intermediate routing makes Tor slow. Bitcoin does not and imho should not make anonymity guarantees. Many users do not need them.\n\u003e \n\u003e Let those who want anonymity connect through Tor, Freenet, etc. It's easy to add anonymity via an extra layer, but it is impossible to add performance on top of a slow system.\n\u003e \n\u003e That's really the only thing I wanted to point out - if you do DHTs, focus on performance, not anonymity. :)\n\u003e \n\u003e Cheers,\n\u003e \n\u003e Stefan\n\u003e \n\u003e On 12/17/2011 2:37 PM, Christian Decker wrote:\n\u003e\u003e A while back I had proposed a similar idea to the DHT, although my main goal was to reduce the need for broadcasts.\n\u003e\u003e \n\u003e\u003e My idea was to structure the network in a hypercube and use prefixes to address different parts of the network, and use those prefixes also to find the location where an item (transaction, block, ...) should be stored. Each vertex in the hypercube is a small, highly connected, cluster of nodes. The storage would be distributed, messages are routed on behalf of others, which makes finding the origin of the query hard to find (think Tor), each node would have to store only O(log(p)) items, with p being the prefix length, maximum number of hops is equal to the dimension of the hypercube O(log(n)).\n\u003e\u003e \n\u003e\u003e Newly created transaction will be sent directly to the location they'll be stored and miners retrieve new transactions at regular intervals. It might increase delays to the confirmations, but it reduces the number of broadcasts and storage requirements on nodes greatly.\n\u003e\u003e \n\u003e\u003e Regards,\n\u003e\u003e Chris\n\u003e\u003e \n\u003e\u003e \n\u003e\u003e On Sat, Dec 17, 2011 at 2:13 PM, Michael Grønager \u003cgronager at ceptacle.com\u003e wrote:\n\u003e\u003e Hey Eric,\n\u003e\u003e \n\u003e\u003e Two comments.\n\u003e\u003e \n\u003e\u003e 1.\n\u003e\u003e The ability to query for transactions belonging to pubkeys or bitcoin addresses is supported today by several implementations:\n\u003e\u003e * blockexplorer.com\n\u003e\u003e * bitcoin-js\n\u003e\u003e * my own libBTC (will more on this soon)\n\u003e\u003e \n\u003e\u003e To query for transactions you need to use json-rpc and not the bitcoin protocol, however. But still the purpose is the same: to be able to build thin clients that can rely on a server for           storing the blockchain and keeping connected on the p2p network.\n\u003e\u003e \n\u003e\u003e The reason for not having these queries part of the standard protocol (I think) are as they breaks anonymity, and that you would actually encourage people to participate in the p2p.\n\u003e\u003e \n\u003e\u003e 2. The second part you mention, to some how move the storage of the blockchain into a DHT based storage would be quite nice. The benefit of this is that it could be a way to integrate the smaller clients into the network without breaking the anonymity. But it should be thought out quite carefully. Further, if each client only store a fraction of the blockchain we should work out what fraction that need to be in order to ensure a similar service level. I would be           happy to work with you on this.\n\u003e\u003e \n\u003e\u003e Cheers,\n\u003e\u003e \n\u003e\u003e Michael\n\u003e\u003e \n\u003e\u003e On 17/12/2011, at 08:41, Eric Lombrozo wrote:\n\u003e\u003e \n\u003e\u003e\u003e Hey, guys.\n\u003e\u003e\u003e \n\u003e\u003e\u003e I haven't posted here before so I'll introduce myself. My name's Eric,\n\u003e\u003e\u003e I've been developing cryptocurrency-related\n\u003e\u003e\u003e software for several months now, I've implemented some libraries for\n\u003e\u003e\u003e dealing with core bitcoin datastructures, made\n\u003e\u003e\u003e some custom builds of bitcoind and interfaced it with a few apps I've written.\n\u003e\u003e\u003e \n\u003e\u003e\u003e In doing so, I've come to appreciate just how little of the potential\n\u003e\u003e\u003e for the bitcoin protocol is being exploited right now...\n\u003e\u003e\u003e not only in terms of the script features but in terms of the potential\n\u003e\u003e\u003e commands and node types that could exist.\n\u003e\u003e\u003e \n\u003e\u003e\u003e For instance, the protocol spec at\n\u003e\u003e\u003e https://en.bitcoin.it/wiki/Protocol_specification only has 16 commands\n\u003e\u003e\u003e listed and\n\u003e\u003e\u003e only one service type...despite having a full 12 bytes for a command\n\u003e\u003e\u003e code and a full eight bytes for a services\n\u003e\u003e\u003e type.\n\u003e\u003e\u003e \n\u003e\u003e\u003e The fact that only one node service type is specified is probably due\n\u003e\u003e\u003e to the fact that the satoshi client was written\n\u003e\u003e\u003e to be a standalone monolithic app that took care of all the essential\n\u003e\u003e\u003e needs for a network of peers.\n\u003e\u003e\u003e i.e. block chain storage/management, transaction signing/verification,\n\u003e\u003e\u003e key generation/wallet management, block mining, etc...\n\u003e\u003e\u003e However, I think there's an urgent need for breaking up all these\n\u003e\u003e\u003e different tasks into separate components that can run as independent\n\u003e\u003e\u003e services on different types of devices.\n\u003e\u003e\u003e \n\u003e\u003e\u003e One of the big issues I'm dealing with now pertains to block chain\n\u003e\u003e\u003e storage. As of right now, it is implemented as sequential\n\u003e\u003e\u003e disk files using Berkeley DB in the satoshi client. Then you have\n\u003e\u003e\u003e other projects that have been using SQL tables, etc...\n\u003e\u003e\u003e But I believe the direction this really needs to move towards is some\n\u003e\u003e\u003e sort of distributed hash table...and the database queries\n\u003e\u003e\u003e should be performed using the bitcoin protocol itself. Perhaps adding\n\u003e\u003e\u003e a few more commands. As things stand right now,\n\u003e\u003e\u003e the only way to query for transactions or blocks is by their hash. And\n\u003e\u003e\u003e once a transaction gets incorporated into a block and\n\u003e\u003e\u003e removed from the transaction pool, one can no longer query it by the\n\u003e\u003e\u003e transaction hash without stepping outside the bitcoin protocol.\n\u003e\u003e\u003e We need access to the disk file that stores the blocks whether it be\n\u003e\u003e\u003e via Berkeley DB or SQL or whatever.\n\u003e\u003e\u003e \n\u003e\u003e\u003e I propose an extension to the bitcoin protocol to provide methods for\n\u003e\u003e\u003e performing more sophisticated queries, such as \"Give me\n\u003e\u003e\u003e an inventory of transactions involving this particular public key\" or\n\u003e\u003e\u003e \"Give me an inventory all transactions in the last n blocks with\n\u003e\u003e\u003e unredeemed outputs.\" This could be done by adding a few more commands.\n\u003e\u003e\u003e \n\u003e\u003e\u003e Furthermore, I propose a new network services type for nodes that\n\u003e\u003e\u003e serve as block chain/transaction pool storage.\n\u003e\u003e\u003e \n\u003e\u003e\u003e Of couse, any peer that wishes to verify the integrity of the block\n\u003e\u003e\u003e chain would still have to download at the very least\n\u003e\u003e\u003e all the block headers...and to be completely sure, also all the blocks\n\u003e\u003e\u003e themselves...and verify everything. But it would be\n\u003e\u003e\u003e very nice to be able to run thin services that can rely on other\n\u003e\u003e\u003e network peers to do this work. It is still possible to attain\n\u003e\u003e\u003e a high level of confidence in the integrity by querying multiple peers\n\u003e\u003e\u003e for similar objects and comparing. It is also possible\n\u003e\u003e\u003e to run your own dedicated block chain storage servers which you trust.\n\u003e\u003e\u003e \n\u003e\u003e\u003e There are other ideas I have for other types of services, too.\n\u003e\u003e\u003e \n\u003e\u003e\u003e Anyhow, I'm just throwing this out there...if anyone's interested I'd\n\u003e\u003e\u003e love to develop these ideas further and help put together some\n\u003e\u003e\u003e specs.\n\u003e\u003e\u003e \n\u003e\u003e\u003e -Eric Lombrozo\n\u003e\u003e\u003e \n\u003e\u003e\u003e ------------------------------------------------------------------------------\n\u003e\u003e\u003e Learn Windows Azure Live!  Tuesday, Dec 13, 2011\n\u003e\u003e\u003e Microsoft is holding a special Learn Windows Azure training event for\n\u003e\u003e\u003e developers. It will provide a great way to learn Windows Azure and what it\n\u003e\u003e\u003e provides. You can attend the event by watching it streamed LIVE online.\n\u003e\u003e\u003e Learn more at http://p.sf.net/sfu/ms-windowsazure\n\u003e\u003e\u003e _______________________________________________\n\u003e\u003e\u003e Bitcoin-development mailing list\n\u003e\u003e\u003e Bitcoin-development at lists.sourceforge.net\n\u003e\u003e\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development\n\u003e\u003e \n\u003e\u003e \n\u003e\u003e \n\u003e\u003e ------------------------------------------------------------------------------\n\u003e\u003e Learn Windows Azure Live!  Tuesday, Dec 13, 2011\n\u003e\u003e Microsoft is holding a special Learn Windows Azure training event for\n\u003e\u003e developers. It will provide a great way to learn Windows Azure and what it\n\u003e\u003e provides. You can attend the event by watching it streamed LIVE online.\n\u003e\u003e Learn more at http://p.sf.net/sfu/ms-windowsazure\n\u003e\u003e _______________________________________________\n\u003e\u003e Bitcoin-development mailing list\n\u003e\u003e Bitcoin-development at lists.sourceforge.net\n\u003e\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development\n\u003e\u003e \n\u003e\u003e \n\u003e\u003e \n\u003e\u003e ------------------------------------------------------------------------------\n\u003e\u003e Learn Windows Azure Live!  Tuesday, Dec 13, 2011\n\u003e\u003e Microsoft is holding a special Learn Windows Azure training event for \n\u003e\u003e developers. It will provide a great way to learn Windows Azure and what it \n\u003e\u003e provides. You can attend the event by watching it streamed LIVE online.  \n\u003e\u003e Learn more at \n\u003e\u003e http://p.sf.net/sfu/ms-windowsazure\n\u003e\u003e \n\u003e\u003e \n\u003e\u003e _______________________________________________\n\u003e\u003e Bitcoin-development mailing list\n\u003e\u003e \n\u003e\u003e Bitcoin-development at lists.sourceforge.net\n\u003e\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development\n\u003e \n\u003e ------------------------------------------------------------------------------\n\u003e Learn Windows Azure Live!  Tuesday, Dec 13, 2011\n\u003e Microsoft is holding a special Learn Windows Azure training event for \n\u003e developers. It will provide a great way to learn Windows Azure and what it \n\u003e provides. You can attend the event by watching it streamed LIVE online.  \n\u003e Learn more at http://p.sf.net/sfu/ms-windowsazure\n\u003e _______________________________________________\n\u003e Bitcoin-development mailing list\n\u003e Bitcoin-development at lists.sourceforge.net\n\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development"}