Cynicus Rex@lemmy.ml to Privacy@lemmy.ml • 6 months agoHow to block AI Crawler Bots using robots.txt fileexternal-linkmessage-square8 fedilinkarrow-up146
arrow-up146external-linkHow to block AI Crawler Bots using robots.txt fileCynicus Rex@lemmy.ml to Privacy@lemmy.ml • 6 months agomessage-square8 Commentsfedilink
minus-squareCynicus Rex@lemmy.mlhexagonhexbear2·edit-26 months agoLies, as in that it's not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out? linkfedilink
minus-squareDa Bald Eagul@feddit.nlhexbear12·6 months agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered. linkfedilink
Lies, as in that it's not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.