I think they need to.
If they refuse to respect robots.txt then if an instance goes to court then that should be used as evidence against them as it clearly indicates the publisher doesn't want their content used in that way rather than it'd publicly available which it isn't
