Nostr notes by Tomas [ARCHIVE]

📅 Original date posted:2017-04-08 📝 Original message:> ...

2023-06-07T17:59:42Z

In reply to nevent1q…lgpf
_________________________

📅 Original date posted:2017-04-08
📝 Original message:> Please no conspiracy theory like stepping on someone’s toes. I believe
> it’s always nice to challenge the established model. However, as I’m
> trying to make some hardfork design, I intend to have a stricter UTXO
> growth limit. As you said "protocol addressing the UTXO growth, might not
> be worth considering protocol improvements*, it sounds like UTXO growth
> limit wouldn’t be very helpful for your model, which I doubt.

Thank you. I realize that this particular phrase implies that in my
design, outputs are less costly then inputs, *in total resource costs*,
which I can not defend without completely ignoring base load script
verification. I rephrased it.

Tomas

📅 Original date posted:2017-04-08 📝 Original message:> I ...

2023-06-07T17:59:41Z

In reply to nevent1q…mvnq
_________________________

📅 Original date posted:2017-04-08
📝 Original message:> I don’t fully understand your storage engine. So the following deduction
> is just based on common sense.
>
> a) It is possible to make unlimited number of 1-in-100-out txs
>
> b) The maximum number of 100-in-1-out txs is limited by the number of
> previous 1-in-100-out txs
>
> c) Since bitcrust performs not good with 100-in-1-out txs, for anti-DoS
> purpose you should limit the number of previous 1-in-100-out txs.
>
> d) Limit 1-in-100-out txs == Limit UTXO growth
>
> I’m not surprised that you find an model more efficient than Core. But I
> don’t believe one could find a model that doesn’t become more efficient
> with UTXO growth limitation.

My efficiency claims are *only* with regards to order validation. If we
assume all transactions are already pre-synced and verified, bitcrust's
order validation is very fast, and (only slightly) negatively effected
by input-counts.

Most total time is spend during base load script validation, and UTXO
growth is the definitely the limiting factor there, as the model here
isn't all that different from Core's.

> Maybe you could try an experiment with regtest? Make a lot 1-in-100-out
> txs with many blocks, then spend all the UTXOs with 100-in-1-out txs.
> Compare the performance of bitcrust with core. Then repeat with
> 1-in-1-out chained txs (so the UTXO set is always almost empty)
>

Again, this really depends on whether we focus on full block validation,
in which case the 100-1, 1-100 distinction will be the similar to Core,
or only regard order validation, in which case Bitcrust will have this
odd reversal.

> One more question: what is the absolute minimum disk and memory usage in
> bitcrust, compared with the pruning mode in Core?

As bitcrust doesn't support this yet, I cannot give accurate numbers,
but I've provided some numbers estimates earlier in the thread.

Rereading my post and these comments, I may have stepped on some toes
with regards to SegWit's model. I like SegWit (though I may have a
slight preference for BIP140), and I understand the reasons for the
"discount", so this was not my intention. I just think that the reversal
of costs during peak load order validation is a rather interesting
feature of using spend-tree based validation.

Tomas

📅 Original date posted:2017-04-08 📝 Original message:On ...

2023-06-07T17:59:41Z

In reply to nevent1q…hjv5
_________________________

📅 Original date posted:2017-04-08
📝 Original message:On Sat, Apr 8, 2017, at 02:44, Gregory Maxwell wrote:
> As you note that the output costs still bound the resource
> requirements.

Resource cost is not just a measure of storage requirement; data that
needs to be accessed during peak load induce more cost then data only
used during base load or only rarely used.

> Latency related costs in Bitcoin Core also do not depend on the number
> of outputs in transactions in a block. When a transaction is handled
> it goes into an in-memory buffer and only gets flushed later if isn't
> spent before the buffer fills. A block will take more time to
> validate with more inputs, same as you observer, but the aggregate
> resource usage for users depends significantly on outputs (so, in fact
> there is even further misaligned incentives than just the fact that
> small outputs have a outsized long term cost).

In Core, when a block comes the inputs are checked against the UTXO set
(which grows with outputs) even if pre-synced, to verify order. Am I
wrong there? This is not in the case in bitcrust; it is instead checked
against the spend-tree (which grows with inputs).

How "significant" this is, I neither know nor claim, but it is an
interesting difference.

> Then I think you may want to retract the claim that "As this solution,
> reversing the costs of outputs and inputs, [...] updates to the
> protocol addressing the UTXO growth, might not be worth considering
> *protocol improvements* "

I think you are being a bit harsh here . I am also clearly explaining
the difference only applies to peak load, and just making a suggestion.
I simply want to stress the importance of protocol / implementation
separation as even though you are correct UTXO data is always a resource
cost for script validation (as I also state), the ratio of different
costs are not necessarily *identical* across implementation.

Note that the converse also holds: In bitcrust, if the last few blocks
contain many inputs, the peak load verification for this block is
slower. This is not the case in Core.

Tomas

📅 Original date posted:2017-04-07 📝 Original ...

2023-06-07T17:59:40Z

In reply to nevent1q…76jg
_________________________

📅 Original date posted:2017-04-07
📝 Original message:Answering both,

On Fri, Apr 7, 2017 at 11:18 AM, Gregory Maxwell via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>>

>> I'm still lost on this-- AFAICT your proposals long term resource

>> requirements are directly proportional to the amount of
>> unspent output
>> data, which grows over time at some fraction of the total transaction
>> volume (plus the rate of spending which is more or less a constant).
>>

>> Can you help out my understanding here?

>>

On Fri, Apr 7, 2017, at 20:39, Bram Cohen wrote:

> Expanding on this question a bit, it's optimized for parallel access,
> but hard drive access isn't parallel and memory accesses are very
> fast, so shouldn't the target of optimization be about cramming as
> much as possible in memory and minimizing disk accesses?

The long term *minimal disk storage* requirement, can obviously not be
less then all the unspent outputs. Minimal disk requirements is not
something bitcrust attempts to address.

The storage that is accessed during peak load (block validation with
pre-synced transactions), is minimized as this only needs the
transaction index (to lookup ptrs from hashes), the tip of the spend-
tree and the tip of the spend-index (together to check double
spents/spending non-existing outputs). These not only easily fit in
RAM, but are accessed in a cache efficient way. *These* only grow with
inputs as the spend tree contains one record per input referencing the
output being spent.

Script validation is also not something bitcrust *directly* addresses;
it uses libbitcoinconsensus for the actual validation and lookups to
outputs are mostly similar. They are kept fast by trusting the OS on MRU
caching of transaction-outputs; I don't think that for this part the
UTXO index has much drawbacks,. Bitcrust seems to have a small advantage
due to the awesomeness of Rayon's parallelization and the lock-free data
structures, but a disadvantage in that keeping all spent outputs
decreases spatial locality of reference. Script validation is not the
innovative part.

Tomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170407/f7db76c7/attachment.html>;

📅 Original date posted:2017-04-07 📝 Original message:Hi ...

2023-06-07T17:59:39Z

In reply to nevent1q…w4gr
_________________________

📅 Original date posted:2017-04-07
📝 Original message:Hi Eric,

On Fri, Apr 7, 2017, at 21:55, Eric Voskuil via bitcoin-dev wrote:
> Optimization for lower memory platforms then becomes a process of
> reducing the need for paging. This is the purpose of a cache. The seam
> between disk and memory can be filled quite nicely by a small amount
> of cache. On high RAM systems any cache is actually a de-optimization
> but on low RAM systems it can prevent excessive paging. This is
> directly analogous to a CPU cache.

I am not entirely sure I agree with that, or understand it correctly.

If -for example - the data of some application is a set of records
which can be sorted from least frequently used to most frequently used
then doing just that sort will beat any application-layer cache.
Regardless of size of data and size of RAM, you simply allow the OS to
use disk caching or memory map caching to work its magic .

In fact, I would argue that an application-layer cache *only* makes
sense if the data model shows a *hard* distinction between often and not
often used data. If usage-frequency is a continuous line, caching is
best left to the OS by focussing on proper spatial and temporal locality
of reference of your data, because the OS has much more information to
make the right decision.

📅 Original date posted:2017-04-07 📝 Original message:Thank ...

2023-06-07T17:59:38Z

In reply to nevent1q…28x5
_________________________

📅 Original date posted:2017-04-07
📝 Original message:Thank you,

The benches are running in Google Cloud Engine; currently on 8 vCPU
32gb, but I tend to switch hardware regularly.

Roughly, the results are better for Bitcrust with high end hardware and
the difference for total block validations is mostly diminished at 2
vCPU, 7,5 gb.

Note that the spend-tree optimization primarily aims to improve peak
load order validation; when a block with pre-synced transactions comes
in, but this is tricky to accurately bench with Core using this simple
method of comparison by logs.

I will upgrade to, and show the results against 0.14 in the next weeks.

Best,

Tomas

On Fri, Apr 7, 2017, at 16:14, Greg Sanders wrote:

> Interesting work.

>

> I was wondering if you could tellank us what specs for the machine
> being used as preliminary benchmark is here:
> https://bitcrust.org/results ?
>

> I'd be interested to also see comparisons with 0.14 which has some
> improvements for script validation with more cores.
>

> On Fri, Apr 7, 2017 at 4:47 AM, Tomas via bitcoin-dev <bitcoin-
> dev at lists.linuxfoundation.org> wrote:
>> Thank you Marcos,

>>

>> Though written in Rust, bitcrust-db is definitely usable as
>> pluggable
>> module as its interface will be roughly some queries, add_tx and

>> add_block with blobs and flags. (Bitcrust internally uses a

>> deserialize-only model, keeping references to the blobs with the
>> parsed
>> data).

>>

>> However, from Core's side I believe network and storage are
>> currently
>> rather tightly coupled, which will make this far from trivial.

>>

>> Regardless, I am also hoping (with funding & a team) to build a
>> Bitcrust
>> networking component as well to bring a strong competitor to the
>> market.
>>

>> best,

>> Tomas

>>

>>

>>

>>

>> On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote:

>> > Hi Tomas,

>> >

>> > I've read it and think it is an excellent work, I'd like to see it
>> > integrated into bitcoin-core as a 'kernel module'.

>> >

>> > I see there are a lot of proof of concepts out there, IMO
>> > every one
>> > deserve a room in the bitcoin client as a selectable feature, to
>> > make the
>> > software more flexible and less dictatorial, an user could easily
>> > select
>> > which features she wants to run.

>> >

>> > Best regards,

>> > Marcos

>> >

>> > > I have been working on a bitcoin implementation that uses a
>> > > different
>> > > approach to indexing for verifying the order of transactions.
>> > > Instead of
>> > > using an index of unspent outputs, double spends are verified by
>> > > using a
>> > > spend-tree where spends are scanned against spent outputs
>> > > instead of
>> > > unspent outputs.

>> > >

>> > > This allows for much better concurrency, as not only blocks, but
>> > > also
>> > > individual inputs can be verified fully in parallel.

>> > >

>> > > I explain the approach at https://bitcrust.org, source code is
>> > > available
>> > > at https://github.com/tomasvdw/bitcrust

>> > >

>> > > I am sharing this not only to ask for your feedback, but also to
>> > > call
>> > > for a clear separation of protocol and implementations: As this
>> > > solution, reversing the costs of outputs and inputs, seems to
>> > > have
>> > > excellent performance characteristics (as shown in the test
>> > > results),
>> > > updates to the protocol addressing the UTXO growth, might not be
>> > > worth
>> > > considering *protocol improvements* and it might be best to
>> > > address
>> > > these concerns as implementation details.

>> > >

>> > > Kind regards,

>> > > Tomas van der Wansem

>> > > tomas at bitcrust.org

>> > > Bitcrust

>> > > _______________________________________________

>> > > bitcoin-dev mailing list

>> > > bitcoin-dev at lists.linuxfoundation.org

>> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

>> > >

>> >

>> >

>> _______________________________________________

>> bitcoin-dev mailing list

>> bitcoin-dev at lists.linuxfoundation.org

>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170407/68d9cba6/attachment.html>;

📅 Original date posted:2017-04-11 📝 Original message:On ...

2023-06-07T17:59:37Z

In reply to nevent1q…9ypp
_________________________

📅 Original date posted:2017-04-11
📝 Original message:On Tue, Apr 11, 2017, at 11:41, Eric Voskuil wrote:
> It's not the headers/tx-hashes of the blocks that I'm referring to, it
> is the confirmation and spend information relative to all txs and all
> outputs for each branch. This reverse navigation (i.e. utxo
> information) is essential, must be persistent and is branch-relative.

That is exactly what is stored in the spend-tree.

>> As a simpler example, if two miners both mine a block at
>> approximately the same time and send it to each other, then surely
>> they would want to continue mining on their own block. Otherwise
>> they would be throwing away their own reward.

> That's not your concurrent validation scenario. In the scenario you
> described, the person chooses the weaker block of two that require
> validation because it's better somehow, not because it's his own
> (which does not require validation).

> Consistency is reached, despite seeing things at different times,
> because people use the same rules. If the economy ran on arbitrary
> block preference consistency would be elusive.

No but my example shows that it is up to the miner to choose which tip
to work on. This is not using different rules, it is just optimizing its
income. This means that the economy *does* run on arbitrary "block
preference", even if it is not running on arbitrary rules.

If two blocks are competing, a miner could optimize its decision which
to mine on, not just on whether one of the blocks is his own, but also
on fees, or on excessive validation costs.

> I read this as encoding the height at which a fork historically
> activated. If you intend to track activation for each branch that will
> not be "height-based" it will be history based.

I understand "height-based" was not the right wording, as it is of
course branch-specific. Per tip ruleset metadata, must be matched with
per-transaction ruleset metadata.

Tomas

📅 Original date posted:2017-04-07 📝 Original message:Thank ...

2023-06-07T17:59:37Z

In reply to nevent1q…ygzp
_________________________

📅 Original date posted:2017-04-07
📝 Original message:Thank you Marcos,

Though written in Rust, bitcrust-db is definitely usable as pluggable
module as its interface will be roughly some queries, add_tx and
add_block with blobs and flags. (Bitcrust internally uses a
deserialize-only model, keeping references to the blobs with the parsed
data).

However, from Core's side I believe network and storage are currently
rather tightly coupled, which will make this far from trivial.

Regardless, I am also hoping (with funding & a team) to build a Bitcrust
networking component as well to bring a strong competitor to the market.

best,
Tomas

On Fri, Apr 7, 2017, at 09:55, Marcos mayorga wrote:
> Hi Tomas,
>
> I've read it and think it is an excellent work, I'd like to see it
> integrated into bitcoin-core as a 'kernel module'.
>
> I see there are a lot of proof of concepts out there, IMO every one
> deserve a room in the bitcoin client as a selectable feature, to make the
> software more flexible and less dictatorial, an user could easily select
> which features she wants to run.
>
> Best regards,
> Marcos
>
> > I have been working on a bitcoin implementation that uses a different
> > approach to indexing for verifying the order of transactions. Instead of
> > using an index of unspent outputs, double spends are verified by using a
> > spend-tree where spends are scanned against spent outputs instead of
> > unspent outputs.
> >
> > This allows for much better concurrency, as not only blocks, but also
> > individual inputs can be verified fully in parallel.
> >
> > I explain the approach at https://bitcrust.org, source code is available
> > at https://github.com/tomasvdw/bitcrust
> >
> > I am sharing this not only to ask for your feedback, but also to call
> > for a clear separation of protocol and implementations: As this
> > solution, reversing the costs of outputs and inputs, seems to have
> > excellent performance characteristics (as shown in the test results),
> > updates to the protocol addressing the UTXO growth, might not be worth
> > considering *protocol improvements* and it might be best to address
> > these concerns as implementation details.
> >
> > Kind regards,
> > Tomas van der Wansem
> > tomas at bitcrust.org
> > Bitcrust
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev at lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >
>
>

📅 Original date posted:2017-04-11 📝 Original message:On ...

2023-06-07T17:59:36Z

In reply to nevent1q…qsma
_________________________

📅 Original date posted:2017-04-11
📝 Original message:On Tue, Apr 11, 2017, at 03:44, Eric Voskuil wrote:

> As I understand it you would split tx inputs and outputs and send them
> independently, and that you intend this to be a P2P network
> optimization - not a consensus rule change. So my comments are based
> on those inferences. If we are talking about consensus changes this
> conversation will end up in an entirely different place.

> I don't agree with the input/output relevance statements above. When a
> tx is announced the entire tx is relevant. It cannot be validated as
> outputs only. If it cannot be validated it cannot be stored by the
> node. Validating the outputs only would require the node store invalid
> transactions.

Splitting transactions only happens *on storage* and is just a minor
optimization compared to storing them in full. (actually a very recent
change with only marginally better results). This is simply because the
output scripts are read on script validation, and storing the outputs of
the transaction separately ensures better spatial locality of reference
(the inputs are just "in the way"). This is not relevant when using a
UTXO-index, because the outputs are then directly stored in the index,
where bitcrust has to read them from the transaction data.

It is not my intention to send them independently.

> I do accept that a double-spend detection is not an optimal criteria
> by which to discard a tx. One also needs fee information. But without
> double-spend knowledge the node has no rational way to defend itself
> against an infinity of transactions that spend the minimal fee but
> also have conflicting inputs (i.e. risking the fee only once). So tx
> (pool) validation requires double-spend knowledge and at least a
> summary from outputs.

Double spent information is still available to the network node and
could still be used for DoS protection, although I do believe
alternatives may exist.

>
> A reorg is conceptual and cannot be engineered out. What you are
> referring to is a restructuring of stored information as a consequence
> of a reorg. I don't see this as related to the above. The ability to
> perform reorganization via a branch pointer swap is based not on the
> order or factoring of validation but instead on the amount of
> information stored. It requires more information to maintain multiple
> branches.
>
> Transactions have confirmation states, validation contexts and spender
> heights for potentially each branch of an unbounded number of
> branches. It is this requirement to maintain that state for each
> branch that makes this design goal a very costly trade-off of space
> and complexity for reorg speed. As I mentioned earlier, it's the
> optimization for this scenario that I find questionable.

Sure, we can still call switching tips a "reorg". And it is indeed a
trade off as orphan blocks are stored, but a block in the spend tree
takes only ~12kb and contains the required state information.

I believe this trade off reduced complexity. For the earlier tree this
could be pruned.

> Because choosing the lesser amount of work is non-consensus behavior.
> Under the same circumstances (i.e. having seen the same set of blocks)
> two nodes will disagree on whether there is one confirmation or no
> confirmations for a given tx. This disagreement will persist (i.e. why
> take the weaker block only to turn around and replace it with the
> stronger block that arrives a few seconds or minutes later). It stands
> to reason that if one rejects a stronger block under a race condition,
> one would reorg out a stronger block when a weaker block arrives a
> little after the stronger block. Does this "optimization" then apply
> to chains of blocks too?

The blockchain is - by design - only eventually consistent across nodes.
Even if nodes would use the same "tip-selection" rules, you cannot rely
on all blocks being propagated and hence each transaction having the
same number of confirmations across all nodes.

As a simpler example, if two miners both mine a block at approximately
the same time and send it to each other, then surely they would want to
continue mining on their own block. Otherwise they would be throwing
away their own reward.

And yes, this can also happen over multiple blocks, but the chances of
consistency are vastly increased with each confirmation.

> Accepting a block that all previous implementations would have
> rejected under the same circumstance could be considered a hard fork,
> but you may be right.

I am not talking about rejecting blocks, I am only talking choosing on
which tip to mine.

> > Frankly, I think this is a bit of an exaggeration. Soft forks are
> > counted on a hand, and I don't think there are many - if any -
> > transactions in the current chain that have changed compliance
> > based on height.
>
> Hope is a bug.
>
> If you intend this to be useful it has to help build the chain, not
> just rely on hardwiring checkpoints once rule changes are presumed to
> be buried deeply enough to do so (as the result of other implementations
> ).
>
> I understand this approach, it was ours at one time. There is a
> significant difference, and your design is to some degree based on a
> failure to fully consider this. I encourage you to not assume any
> consensus-related detail is too small.

I am not failing to consider this, and I don't consider this too small .
But ensuring contextual transaction validity by "validate => valid with
rules X,Y,Z" and then checking the active rules (softfork activation) on
order validation, will give logically the same results as "validate with
X,Y,Z => valid". This is not "hardwiring checkpoints" at all.

> You cannot have a useful performance measure without full compliance.

I agree that the results are preliminary and I will post more if the
product reaches later stages.

> It's worth noting that many of your stated objectives, including
> modularity, developer platform, store isolation, consensus rule
> isolation (including optional use of libbitcoinconsensus) are implemente
> d.
>
> It seems like you are doing some good work and it's not my intent to
> discourage that. Libbitcoin is open source, I don't get paid and I'm
> not selling anything. But if you are going down this path you should
> be aware of it and may benefit from our successes as well as some of
> the other stuff :). And hopefully we can get the benefit of your
> insights as well.

Thank you, I will definitely further dive into libbitcoin, and see what
insights I can use for Bitcrust.

Tomas

📅 Original date posted:2017-04-08 📝 Original message:Thank ...

2023-06-07T17:59:34Z

In reply to nevent1q…vamq
_________________________

📅 Original date posted:2017-04-08
📝 Original message:Thank you for your elaborate response Eric,

On Sun, Apr 9, 2017, at 00:37, Eric Voskuil wrote:
> My point was that "Using a storage engine without UTXO-index" has been
> done, and may be a useful reference, not that implementation details
> are the same.

I haven't dived into libbitcoin V2/V3 enough to fully grasp it and
though your comments help, I still not fully do. I will answer below
what is related to bitcrust itself.

My post wasn't posted to claim innovation; I merely try to explain how
Bitcrust works and why it performs well.

> First, I remain confused on your comments pertaining to UTXO growth
> and network protocol. I followed your conversation with Greg and it
> remains unclear to me. From what I understand you have isolated order
> (double spend) from script validation. I think we all understand that
> script validation requires inputs and outputs while double spend
> detection requires correlation of inputs. What I do not understand is
> your choice of optimization axis.
>
> Detection of double spend is not useful in isolation. One must also
> validate scripts, which requires outputs. I can see that there is an
> opportunity to reject blocks (within the same branch) faster by
> validating for double spends before validating script. But unconfirmed
> transactions do not exist in a branch, and are therefore not truly
> conflicting, until they are mined. And even after they are mined
> conflicting txs remain potentially valid in other branches. So
> rejecting txs due to conflict comes down to a denial of service
> policy, which ultimately must be based on fee increment (e.g. RBF).
> But fees are based on the amount of the output value that remains
> unspent in the transaction. So this in turn requires the retrieval of
> outputs.
>
> And yet the remaining scenario of fast rejection of invalid blocks is
> not a meaningful optimization. Optimizing for the case where a block
> has valid and sufficient PoW and yet is invalid (for double spend) is
> counterproductive. And even so, the txs within the invalid block may
> be entirely valid independent of the block, so you are back to looking
> up their outputs to obtain fees in the case of a double spend or to
> validate script otherwise. In all cases you need to get the outputs.
>
> > Bitcrust simply scans the tree. Although earlier designs used a
> > skip-list, it turns out that accompanied by a spent-index lagging a
> > few blocks behind, raw scanning is faster then anything even though
> > it needs to scan ~5 blocks times ~4000 inputs before reaching the
> > first spent-index, the actual scan is highly cache efficient and
> > little more then a "REP SCASQ", reaching sub-microsecond per input
> > on each core *including* the lookup in the spend index.
>
> I realize that you see the implementation of the ordering validation
> as interesting detail, but I find it hard to justify contemplating the
> implementation in isolation from the output lookup requirement. And if
> one must looking up both outputs and spends for each validation, it
> makes more sense to co-locate that data.
>
> Recovering in one step all data necessary to validate a tx has real
> advantages over either interleaving queries and validation or
> splitting input vs. output validation queries into two steps. It is a
> significantly more test-friendly approach, has better performance
> characteristics, and simplifies code. I cannot see any reason to
> perform the data read for double spend validation in isolation of that
> for script validation.

You seem to ignore here the difference between base load and peak load.
If Compact blocks/XThin with further optimizations can presync nearly
100% of the transactions, and nodes can do as much as possible when a
transaction comes in, the time spent when a block comes in can be
minimized and a lot more transactions can be handled with the same
resources.

The reason for "splitting" is that for an incoming transaction the
spent-state of the outputs being spent isn't particularly relevant as
you seem to acknowledge. When the block comes in, the actual output data
isn't relevant.

The *only* thing that needs to be checked when a block comes in is the
order, and the spend-tree approach absolves the need to access outputs
here.

As it also absolves the need for reorgs this greatly simplifies the
design. I am not sure why you say that a one-step approach is more
"test-friendly" as this seems to be unrelated.

>
> If by results you are referring to performance numbers, it's very hard
> to draw any conclusions without a full benchmark. It's great that if
> you are able to boost Core, but from my perspective the numbers aren't
> especially compelling.
>

I fully agree and hopefully do not pretend to hide that my numbers are
premature without a full implementation. I just think they are promising
enough to convince at least myself to move on with this model.

> Despite the site's explanation I cannot think of any reason to ever
> validate two blocks at the same time. You would always prioritize the
> block with the greatest PoW. Doing otherwise just slows down the net
> validation in all but the pathological case where a miner has produced
> an *invalid* block with *more* PoW than another valid block which
> arrived at the node within the same second. Rejecting a *valid* block
> with more PoW in favor of one with *less* "processing" is a hard fork,
> so you probably wouldn't want to do that either. But with compact
> block validation times approaching 25ms it's hard to justify stopping
> a block validation for any reason.

I don't get what you are saying. Why pick the greatest PoW of two
competing blocks? If two blocks come in, an implementation is free to
choose whichever block to build on. Choosing so is not a "hardfork".
Parallel validation simply makes it easier to make an optimal choice,
for if two blocks come in, the one that is validated fastest can be
build upon without the risk of validationless mining.

>
> That's not to say parallel block validation difficult to do. If you
> can validate one block's full set of inputs in parallel (which is not
> novel) doing the same with additional blocks has trivial additional
> complexity.

I am not trying to claim novelty here.

> I am also interested in your previous comments about soft forks. These
> are material considerations that Greg touched on but it doesn't sound
> like you fully appreciate just yet. When a tx is pre-validated the
> rules applied must be the same rules as those of some future block.
> Yet a tx can be included in more than one block (different branches).
> Across branches and even in one branch, validation rules change, and
> can change back. The changes are based on accumulated branch history.
> Pre-validation can later become invalidated, and differently in
> different branches. And maintaining proper context requires either
> storing state that you are apparently not storing, or invalidating
> optimizations. Based on your comments you do not seem to be accounting
> for this in your storage assumptions or in your results. A recent post
> by Greg highlights the complexity and consensus criticality of these
> considerations.

Frankly, I think this is a bit of an exaggeration. Soft forks are
counted on a hand, and I don't think there are many - if any -
transactions in the current chain that have changed compliance based on
height. This makes this a compliance issue and not a performance issue
and the solution I have explained, to add height-based compliance as
meta data of validation seems to
be adequate and safe.

> The hash table store that I described can fully navigate the block
> tree and transaction DAG, since the stored tx, parent and point hashes
> are also natural keys and each link is navigable in constant time. It
> is also lock-free, can concurrently write any number of blocks during
> initial block download and supports read/write concurrency. It has
> successfully indexed and stored the entire blockchain from the P2P
> network in 16 minutes (locally). It also stores both confirmed and
> unconfirmed transactions in the same store, so there is nothing to
> write when a block is confirmed except for the block header/hashes and
> updates to spender heights for any output spent by the new block's
> txs. It is similarly capable of storage in the block table of weak
> chain blocks...
>

I think I get the gist of your approach and it sounds very interesting
and I will definitely dive in deeper.

It also seems sufficiently different from Bitcrust to merit competing on
(eventual) results instead of the complicated theory alone.

Best,
Tomas

📅 Original date posted:2017-04-06 📝 Original message:Hi ...

2023-06-07T17:59:33Z

In reply to nevent1q…2jhc
_________________________

📅 Original date posted:2017-04-06
📝 Original message:Hi Eric,

Thanks, but I get the impression that the similarity is rather
superficial.

To address your points:

> (1) higher than necessary storage space requirement due to storing the
> indexing data required for correlate the spends, and

Hmm. No. Spends are simply scanned in the spend-tree (full tree,
prunable, fully 5.6gb), or caught by the spend-index (bit index,
non-prunable, fully 180mb). Neither impose significant storage
requirements.

> 2) higher than necessary validation complexity and cost in terms of
> computing the spent-ness (including spender height) of an output.
>
> With the exception of de-linking (not deleted) in the case of reorgs, the
> entire store is append only, implemented in a small set of memory
> mapped file

I guess this is the key difference. As the spend-tree stores the spend
information in a tree structure, no reorgs are required, and the
resulting code is actually much less complex.

Bitcrust simply scans the tree. Although earlier designs used a
skip-list, it turns out that accompanied by a spent-index lagging a few
blocks behind, raw scanning is faster then anything even though it needs
to scan ~5 blocks times ~4000 inputs before reaching the first
spent-index, the actual scan is highly cache efficient and little more
then a "REP SCASQ", reaching sub-microsecond per input on each core
*including* the lookup in the spend index.

> I don't follow this part, maybe you could clarify. A spends index
> grows with the size of the spend set (forever) as it cannot be pruned,
> which certainly exceeds the size of the UTXO set (unless nothing is
> spent). The advantage is that you don't have to keep rewriting the
> store when you use a spends set (because the store can be append only).

My point is, that the spend tree grows per *input* of a transaction
instead of per *output* of a transaction, because this is what is
scanned on order validation.

The spend tree can be pruned because the spend index (~200mb) catches
early spends.

Disregarding the baseload script validation, the peak load order
validation of bitcrust is more negatively effected by a transaction with
many inputs than by a transaction of many outputs.

I encourage you to check out the results at https://bitcrust.org

Regards,
Tomas

On Fri, Apr 7, 2017, at 01:38, Eric Voskuil wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 04/06/2017 03:12 PM, Tomas via bitcoin-dev wrote:
>
> Hi Tomas,
>
> > I have been working on a bitcoin implementation that uses a
> > different approach to indexing for verifying the order of
> > transactions. Instead of using an index of unspent outputs, double
> > spends are verified by using a spend-tree where spends are scanned
> > against spent outputs instead of unspent outputs.
>
> This is the approach that genjix used in libbitcoin version2. With the
> exception of de-linking (not deleted) in the case of reorgs, the
> entire store is append only, implemented in a small set of memory
> mapped files. The downsides to the approach are:
>
> (1) higher than necessary storage space requirement due to storing the
> indexing data required for correlate the spends, and
>
> (2) higher than necessary validation complexity and cost in terms of
> computing the spent-ness (including spender height) of an output.
>
> His implementation used a hash table, so performance-wise it did quite
> well and would theoretically outperform a tree, O(1) vs. O(log2(N)).
>
> > This allows for much better concurrency, as not only blocks, but
> > also individual inputs can be verified fully in parallel.
>
> I was successful in parallelizing input validation (across the inputs
> of an unconfirmed tx and across the set of all inputs in a block)
> using the v2 store. However, it is not the case that the spends
> approach is necessary for concurrency.
>
> To resolve the above two problems the version3 store does not use a
> spends table/index. Nor does it store any table of UTXOs. Yet
> validation is highly parallelized. Instead of additional indexes it
> uses the tx hash table, augmented with 32 bits per output for spender
> height. So there is a O(1) cost of finding the tx and a O(N) cost of
> finding the spender height where N is the number of outputs in the tx.
> But because the number of outputs in a tx is bounded (by block size)
> this is constant time in the number of transactions.
>
> This works out much faster than the spends table, and without the
> storage cost or complexity disadvantages. It also scales with
> available hardware, as the memory mapped files become in-memory hash
> tables. For low memory machines we found it was important to implement
> an opaque UTXO cache to limit paging, but for higher end systems zero
> cache is optimal.
>
> > I am sharing this not only to ask for your feedback, but also to
> > call for a clear separation of protocol and implementations: As
> > this solution, reversing the costs of outputs and inputs, seems to
> > have excellent performance characteristics (as shown in the test
> > results), updates to the protocol addressing the UTXO growth, might
> > not be worth considering *protocol improvements* and it might be
> > best to address these concerns as implementation details.
>
> I don't follow this part, maybe you could clarify. A spends index
> grows with the size of the spend set (forever) as it cannot be pruned,
> which certainly exceeds the size of the UTXO set (unless nothing is
> spent). The advantage is that you don't have to keep rewriting the
> store when you use a spends set (because the store can be append only).
>
> Feel free to message me if you'd like to discuss in more detail, or to
> continue on the libbitcoin mailing list (copied).
>
> e
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQEcBAEBCAAGBQJY5tFpAAoJEDzYwH8LXOFOcMgH/2mw5iOvUYNwvZ2z0KKTSUOA
> Pd8d5mKoWvd94QxhQ+RyTbkEkMhHl75+zcBgRsfUTtZlBIe/Z0+OgVIN6ibEw+WD
> w7k3HqgQi9gLgydEelxTAX+z3dJ24n4kCCdKAmZbBuK+Yr/7AViugbEqYemKepku
> pRWZZS74MUvrYesc0xPn4Ao3DTzMjjY0K2mkuqV8jlwdfZjlAQX9pTx+iSCuMhkd
> HJ8w7s8QnjVnUeOlLe29mZwaFJPyOTLJMqgDE6s2sXacAy5QQbVCatygvDQ8A/wC
> ktBnKPFb2lGX3bGKu/KwABegBy/hyec+NP0wFR+0MVivCwTK1+SjeHu5MNOSVlM=
> =tfVj
> -----END PGP SIGNATURE-----

📅 Original date posted:2017-04-06 📝 Original message:I have ...

2023-06-07T17:59:32Z

In reply to nevent1q…w8pm
_________________________

📅 Original date posted:2017-04-06
📝 Original message:I have been working on a bitcoin implementation that uses a different
approach to indexing for verifying the order of transactions. Instead of
using an index of unspent outputs, double spends are verified by using a
spend-tree where spends are scanned against spent outputs instead of
unspent outputs.

This allows for much better concurrency, as not only blocks, but also
individual inputs can be verified fully in parallel.

I explain the approach at https://bitcrust.org, source code is available
at https://github.com/tomasvdw/bitcrust

I am sharing this not only to ask for your feedback, but also to call
for a clear separation of protocol and implementations: As this
solution, reversing the costs of outputs and inputs, seems to have
excellent performance characteristics (as shown in the test results),
updates to the protocol addressing the UTXO growth, might not be worth
considering *protocol improvements* and it might be best to address
these concerns as implementation details.

Kind regards,
Tomas van der Wansem
tomas at bitcrust.org
Bitcrust