Stop bot attacks?

[Genestoy]

Yesterday my site went down due to "using too many resources" on my server which has never happened before and was not caused by my members. I contacted my host and they said that I had "extremely high traffic" and several bots were crawling my site and the forum had literally thousands of page hits per minute so the server automatically shut down my forum, this continued for several hours and I have now been placed in a "penalty box". My question is since I understand after doing a search here that IPS 4 no longer uses or needs a robots.txt file so how do you prevent this from happening again and stop these bots?

Thanks for any assistance with this

[AndyF]

I do remember slightly more 'visits' when sites went from 2x to 3x (unsure if you had this when you moved to 3x from 2x) ? , although if memory serves most of the cause was due to the newer URL structure in 3x compared to 2x

Were you using (on 3x before you upgraded to IPS4) the "legacy" type URL's still ? By this I mean &showtopic=x / &showforum=x etc ?

You can still use a robots.txt file to slow "good" bots down with a crawl-delay line. Google you'll need to use their Webmaster tools to slow their bot down.

User-agent: *
Crawl-delay: 120

If its a specific bot causing problems (Baidu maybe?) you *might* have to consider temporarily banning it via .htaccess or suchlike, although this is not ideal I know but its better than having your site suspended or restricted due to resource use. Depends on how much you need/value SEO with your community niche I guess.

[Adlago]

This is my list of IP addresses and networks that bother you.
If you want to paste this list without changing, in your htaccess file

deny from 178.151.216.90
deny from 5.10.83.0/24
deny from 120.62.180.27
deny from 77.190.50.73
deny from 119.147.146.118
deny from 85.114.135.126
deny from 91.200.13.12
deny from 180.76.6.58
deny from 220.181.132.217
deny from 5.135.199.22
deny from 93.84.14.185
deny from 218.30.103.0/24
deny from 188.165.15.
deny from 78.137.18.108
deny from 46.229.164.111
deny from 5.9.76.35
deny from 23.22.131.24
deny from 70.39.246.37
deny from 54.91.137.217
deny from 46.229.164.0/24
deny from 185.65.245.134
deny from 136.243.36.97
deny from 164.132.161.0/24
deny from 51.255.65.0/24
deny from 192.243.55.0/24
deny from 151.80.31.0/24
deny from 136.243.36.80
deny from 136.243.36.93
deny from 78.46.174.55
deny from 123.126.113.0/24
deny from 46.229.168.0/24
deny from 46.161.9.20

 

[Genestoy]

Thanks to both of you have added both to my domain