The point is the *reasonable expectation of trust.
You trust elliptic curve multiplication only goes one way and cant be unwound.
It is a *similar application of trust to use a RNG. Is it necessary? No, use dice if you wanna.
But if HWRNG is compromised then the entire security layer of the internet is broken and we have much bigger problems.
