everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

  • @Typewar@infosec.pub
    link
    fedilink
    English
    2
    edit-2
    5 months ago

    I have a dumb question… what is preventing the crawlers from just eating the shit and just burn though the energy to get through the computational task?

    • SavvyWolf
      link
      fedilink
      English
      15 months ago

      It’ll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else’s site.