If you’re having issues with bot traffic, we have a trick on RetroLemmy. We use a cron that checks the server load, and when the load is too high it switches the instance into private mode (only logged in users can browse), when load goes back down again it switches to public mode again. Private mode means the bot requests use much fewer resources, but it also means the bots have no content to scrape and they probably run out of links to crawl very quickly.
This specific issue why slrpnk.net is down right now seems unrelated to the bot traffic, but I think once I am able to fix the server setup, I will move it to some kind hybrid where the main bulk of the bot traffic is taken on by a professional VPS instead of my home network. In the last months it was sometimes so bad that I coudn’t even browse normally on my otherwise fully seperated / vlan-ed personal PC as the main fiber gateway seems to have been bogged down by the scraper traffic.
We use a professional VPS, and the bot attacks got so bad that retrolemmy was down multiple times a day for sometimes an hour or more. This script helped so much that we didn’t have any bot related downtime since then; for almost 3 months now. Also the users notice no difference when the site is private, unless they are not logged in. And it goes back public after 5 minutes under the threshold.
The bot traffic has gotten so bad this year. Good luck with the server and bot traffic. I hope you find a solution.
If you’re having issues with bot traffic, we have a trick on RetroLemmy. We use a cron that checks the server load, and when the load is too high it switches the instance into private mode (only logged in users can browse), when load goes back down again it switches to public mode again. Private mode means the bot requests use much fewer resources, but it also means the bots have no content to scrape and they probably run out of links to crawl very quickly.
https://github.com/DaDosDude/Lemmy-bot-protection
@CafeFrog@lemmy.cafe @Kris@feddit.org @Five@lemmy.cafe
This specific issue why slrpnk.net is down right now seems unrelated to the bot traffic, but I think once I am able to fix the server setup, I will move it to some kind hybrid where the main bulk of the bot traffic is taken on by a professional VPS instead of my home network. In the last months it was sometimes so bad that I coudn’t even browse normally on my otherwise fully seperated / vlan-ed personal PC as the main fiber gateway seems to have been bogged down by the scraper traffic.
We use a professional VPS, and the bot attacks got so bad that retrolemmy was down multiple times a day for sometimes an hour or more. This script helped so much that we didn’t have any bot related downtime since then; for almost 3 months now. Also the users notice no difference when the site is private, unless they are not logged in. And it goes back public after 5 minutes under the threshold.
The bot traffic has gotten so bad this year. Good luck with the server and bot traffic. I hope you find a solution.
Might be helpful yes. I have bookmarked it for now. Thanks.