{"type":"rich","version":"1.0","author_name":"npub1sgs97fe0n9wehe6zw7drcxdz4cy9yt9pfqjv8gasz5jlk4zezc0quppx3c","author_url":"https://nostr.ae/npub1sgs97fe0n9wehe6zw7drcxdz4cy9yt9pfqjv8gasz5jlk4zezc0quppx3c","provider_name":"njump","provider_url":"https://nostr.ae","html":"📅 Original date posted:2017-04-07\n📝 Original message:-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\nOn 04/07/2017 11:39 AM, Bram Cohen via bitcoin-dev wrote:\n\u003e Expanding on this question a bit, it's optimized for parallel\n\u003e access, but hard drive access isn't parallel and memory accesses\n\u003e are very fast, so shouldn't the target of optimization be about\n\u003e cramming as much as possible in memory and minimizing disk\n\u003e accesses?\n\nWhile this may seem to be the case it is not generally optimal. The\nquestion is overly broad as one may or may not be optimizing for any\ncombination of:\n\nstartup time (first usability)\nwarm-up time (priming)\nshutdown time (flush)\nfault tolerance (hard shutdown survivability)\ntop block validation (read speed)\nfull chain validation (read/write speed)\nRAM consumption\nDisk consumption\nQuery response\nServers (big RAM)\nDesktops (small RAM)\nMining (fast validation)\nWallets (background performance)\nSSD vs. HDD\n\nBut even limiting the question to input validation, all of these\nconsiderations (at least) are present.\n\nIdeally one wants the simplest implementation that is optimal under\nall considerations. While this may be a unicorn, it is possible to\nachieve a simple implementation (relative to alternatives) that allows\nfor the trade-offs necessary to be managed through configuration (by\nthe user and/or implementation).\n\nShoving the entire data set into RAM has the obvious problem of\nlimited RAM. Eventually the OS will be paging more of the data back to\ndisk (as virtual RAM). In other words this does not scale, as a change\nin hardware disproportionately impacts performance. Ideally one wants\nthe trade between \"disk\" and \"memory\" to be made by the underlying\nplatform, as that is its purpose. Creating one data structure for disk\nand another for memory not only increases complexity, but denies the\nplatform visibility into this trade-off. As such the platform\neventually ends up working directly against the optimization.\n\nAn on-disk structure that is not mapped into memory by the application\nallows the operating system to maintain as much or as little state in\nmemory as it considers optimal, given the other tasks that the user\nhas given it. In the case of memory mapped files (which are optimized\nby all operating systems as central to their virtual memory systems)\nit is possible for everything from zero to the full store to be memory\nresident.\n\nOptimization for lower memory platforms then becomes a process of\nreducing the need for paging. This is the purpose of a cache. The seam\nbetween disk and memory can be filled quite nicely by a small amount\nof cache. On high RAM systems any cache is actually a de-optimization\nbut on low RAM systems it can prevent excessive paging. This is\ndirectly analogous to a CPU cache. There are clear optimal points in\nterms of cache size, and the implementation and management of such a\ncache can and should be internal to a store. Of course a cache cannot\nprovide perfect scale all the way to zero RAM, but it scales quite\nwell for actual systems.\n\nWhile a particular drive may not support parallel operations one\nshould not assume that a disk-based store does not benefit from\nparallelism. Simply refer to the model described above and you will\nsee that with enough memory the entire blockchain can be\nmemory-resident, and for high performance operations a fraction of\nthat is sufficient for a high degree of parallelism.\n\nIn practice a cache of about 10k transactions worth of outputs is\noptimal for 8GB RAM. This requires just a few blocks for warm-up,\nwhich can be primed in inconsequential time at startup. Fault\ntolerance can be managed by flushing after all writes, which also\nreduces shutdown time to zero. For higher performance systems,\nflushing can be disabled entirely, increasing shutdown time but also\ndramatically increasing write performance. Given that the blockchain\nis a cache, this is a very reasonable trade-off in some scenarios. The\nmodel works just as well with HDD as SSD, although certainly SSD\nperforms better overall.\n\ne\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v2.0.22 (GNU/Linux)\n\niQEcBAEBCAAGBQJY5+7GAAoJEDzYwH8LXOFOsAsH/3QK55aWH6sAi6OsTwV1FLZV\nY/2SSjwn1vUh55MDkPpCxDwV99JqVwpk0vGM8mGg5s4ZS8sxOPqwGiBz/SZWbF9v\noStJS0DjUPnbYtI/mrC30GuAYVcKnc5DFDHvjX6f0xrLIzViFR7eiW0npUH6Xipt\nRI9Mockaf1CqqGExtbIqWal0YDEQGH0ekXRp7uEjh8nPUoKqTVvxDCgqVooQfvfx\nEeKX9ruSv/r91EM1JQuH8HBBF7+R24tmMtwbpGx0zrDg5ytpIyrRzVH/ze1Mj2a3\nZxThvofGzhKcDiTPWiJI11DBYUvhSH4Kx0uWLzFUA0gxPfWkZQKJWNDl2CEwljk=\n=C7rD\n-----END PGP SIGNATURE-----"}