انضم إلى نوستر
2026-05-18 13:21:05 UTC
in reply to

École des Bro-Arts on Nostr: We already use extensive robots.txt rules and user-agent blocking. Another option ...

We already use extensive robots.txt rules and user-agent blocking. Another option I've considered is adding a honeypot page for scrapers that immediately triggers fail2ban, and disallowing that from robots.txt, on the theory that well-behaved bots won't hit it.