Preventing malicious bots to your site - strategies?

[evcom]

I know it's an old topic, but es prevention strategies, tools and bots evolve, I thought it's worth a new topic. 

Since a few days I get an unusual high number of visitors to my site. Instead of 10 users on average in a given time, there are now 100. Bots!
They visit different pages but so far no registrations or content spamming was observed. 

I tested whether they would respect robots.txt instructions. They don't. 

So I modified my htaccess file to prevent all countries, except the ones my normal users are from, to be blocked. This is effective and bots are no longer visiting, but it also blocks good bots, e.g. from search engines. 

I've tried adding the malicious IP addresses to the htaccess file, but it seems it's hard to catch 'em as they seem to be changing. When clicking on the IP address in the "Who is online" list, the bots seem to be from Moscow, Rio, Philadelphia...

What other options do I have to prevent these bots from sneaking around? Could they somehow be identified? If so how? The system log file does not show any unusual entries. 
Any tip is highly appreciated. Thanks

Edited November 12, 2021 by evcom

[Makoto]

Use Cloudflare if you aren't already, that's my number one recommendation.

It can do a tremendous job in reducing the number of malicious bots and scrapers that visit your website.

[evcom]

Thanks for this tip! Much appreciated. I've signed up with it and see what it can do. So far I had not really issues with bot traffic. But recently...

[Makoto]

Yeah, bots can become a huge pain. Usually malicious robots or scripts that are constantly scanning your website for vulnerabilities. I've been there trying to play the whack-a-mole game with them.

[evcom]

Actually I have not really been successful so far. Traffic is still 10x higher and IC reports 100 users online, thereof only about 10 are real/good bots.
Although Cloudflare offers tons of options, the bots or the traffic somehow get through. Only if I block entire countries, it get's to normal levels. Tried: 
- Blocking individual IPs
- Blocking ASN Numbers
- Using JS Challenges / Captcha
- Under attack mode (not so good for user experience...)
- Rate limiting
and of course all the other bells and whistles that can be turned on using the free Cloudflare plan. And I have the feeling the Pro Version does not really make a difference.

Or should I just give up and hope it goes back to normal levels one day?

[Makoto]

It's hard to say without looking at the traffic myself, trying to analyze what exactly they're doing specifically. Are they just trying to scrape your website, or trying to run malicious scripts? Things like that can give you a bit to go on.

In the latter case, enabling WAF rules could help, but you have to take a bit of care with those to ensure you don't cause false-positive triggers.

Blocking ASN's of entire web hosts where the malicious traffic is coming from is probably not a bad idea. Even if it seems like a hopeless endeavor, it may just take some time, monitoring, and persistence.

To some extent, though, you do have to account for this kind of traffic occurring and be able to scale with it as your website grows.

Edited November 13, 2021 by Makoto

[Sonya*]

I use CleanTalk Anti Spam Firewall There is a free IPS application for 14 days to try it out.

Edited November 14, 2021 by Sonya*

[IveLeft...]

1 hour ago, Sonya* said:

I use CleanTalk Anti Spam Firewall There is a free IPS application for 14 days to try it out.

That looks really good and well priced 

[evcom]

On 11/13/2021 at 11:30 PM, Makoto said:

Blocking ASN's of entire web hosts where the malicious traffic is coming from is probably not a bad idea. Even if it seems like a hopeless endeavor, it may just take some time, monitoring, and persistence.

That was it. I started blocking ASNs using Cloudflare. I found the resprective ASN numbers by entering the IP addresses into a Whois lookup site, providing also the corresponding ASNs. After blocking about 8 of them, bot traffic dissapeared.
 

[Makoto]

1 hour ago, evcom said:

That was it. I started blocking ASNs using Cloudflare. I found the resprective ASN numbers by entering the IP addresses into a Whois lookup site, providing also the corresponding ASNs. After blocking about 8 of them, bot traffic dissapeared.
 

Awesome! I'm glad you were able to get it taken care of!