Drunk & Root to Selfhosted@lemmy.worldEnglish • 5 months agoHow to combat large amounts of Ai scrapersmessage-square54fedilinkarrow-up1125arrow-down15file-text
arrow-up1120arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root to Selfhosted@lemmy.worldEnglish • 5 months agomessage-square54fedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-square@Sheldan@lemmy.worldlinkfedilinkEnglish2•5 months agoSome of them are at least honest and have it as a user agent.
minus-square@krakenfury@lemmy.sdf.orglinkfedilinkEnglish1•5 months agoIs ignoring robots.txt considered “honest”?
Some of them are at least honest and have it as a user agent.
Is ignoring robots.txt considered “honest”?
That’s not what I was talking about