Our entire community is based on the idea that most of the world can hate you and still be wrong. And that this is not at all uncommon for many things.
My point isn’t to say that critics are bad or should always be ignored, but rather that they are inevitable and you shouldn’t take critics at their word. I’m sure you’ve had your own critics, and seen how the truth is often lost in resentment. The key is to practice discernment, and not just believe everything you hear, or equate “a lot of people say bad things” with “this must actually be bad.”
As for “how do you know,” the answer is simply “you can’t.” There are no guarantees in life. Any decision you make has a real chance of going wrong. And the more you commit to it, the worse the outcome if you turn out to be wrong.
But what many fail to do is accept that inaction is itself a choice—it’s just one that is hard to blame on you. Most of the problems in our world happen less because bad beings exist (they always have and always will exist) but because others chose not to do anything about what was happening. Having a villain to point at helps them feel better about doing nothing. Instead of taking a 10% chance that something goes wrong if they take action, they take a 90% chance something will go wrong if they do nothing—that’s not about ethics or making things better. It’s about avoiding accountability. Making sure no one can blame you when things go wrong. And so we live in a world where everyone avoids doing anything to help, for threat of being blamed for bad outcomes.
A democratic or anarchist system that fails to take actions is not less unethical—it is simply using its incompetence to diffuse responsibility so no one individual can be blamed. I think anarchism is a valid and good pursuit, but most political anarchists are naive anarchists that seem to think that removing all authorities without well designed distributed systems will just magically make things better, when in reality it would just remove accountability and make a miserable cesspool of beings blaming vague, unspecified others for all the problems. “Why won’t anyone do something about this??” becomes the rallying cry. And when someone actually does try to do something about it, and no one will help them, they start to try and accumulate the power to actually do it—and then are faced with you, saying “but what if you abuse the power?”
What matters ethically is universal subjective wellbeing. All organizational structures are just that: Ways of organizing communities and systems. Every organization concept has benefits, drawbacks and risks.

