École des Bro-Arts on Nostr: We already use extensive robots.txt rules and user-agent blocking. Another option ...
We already use extensive robots.txt rules and user-agent blocking. Another option I've considered is adding a honeypot page for scrapers that immediately triggers fail2ban, and disallowing that from robots.txt, on the theory that well-behaved bots won't hit it.