Drunk & Root to Selfhosted@lemmy.worldEnglish • 5 months agoHow to combat large amounts of Ai scrapersmessage-square54fedilinkarrow-up1125arrow-down15file-text
arrow-up1120arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root to Selfhosted@lemmy.worldEnglish • 5 months agomessage-square54fedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-square@daniskarma@lemmy.dbzer0.comlinkfedilinkEnglish2•edit-25 months agoDo you have a proper robots.txt file? Do they do weird things like invalid url, invalid post tries? Weird user agents? Millions of times by the same ip sound much more like vulnerability proving than crawler. If that’s the case fail to ban or crowdsec. Should be easy to set up a rule to ban an inhumane number of hits per second on certain resources.
minus-squareDrunk & RootOPlinkfedilinkEnglish1•5 months agosince its the frontends i run getting scraped its the robots.txt included there
Do you have a proper robots.txt file?
Do they do weird things like invalid url, invalid post tries? Weird user agents?
Millions of times by the same ip sound much more like vulnerability proving than crawler.
If that’s the case fail to ban or crowdsec. Should be easy to set up a rule to ban an inhumane number of hits per second on certain resources.
since its the frontends i run getting scraped its the robots.txt included there