The other problem I've found with being wrong 50% of the time, is if you aren't an expert in the topic, you don't know it's wrong, and I've found many people treat the LLM as infallible.
I've had to tell my estimators they cannot use LLMs for asking electrical code calculations. I've asked it to size wires, thinking it's all hard data that's in charts, and it confidently makes stuff up. If I wasn't already familiar with the charts, it wouldn't even raise a red flag.
