Genestoy Posted July 10, 2017 Posted July 10, 2017 Yesterday my site went down due to "using too many resources" on my server which has never happened before and was not caused by my members. I contacted my host and they said that I had "extremely high traffic" and several bots were crawling my site and the forum had literally thousands of page hits per minute so the server automatically shut down my forum, this continued for several hours and I have now been placed in a "penalty box". My question is since I understand after doing a search here that IPS 4 no longer uses or needs a robots.txt file so how do you prevent this from happening again and stop these bots? Thanks for any assistance with this
AndyF Posted July 10, 2017 Posted July 10, 2017 I do remember slightly more 'visits' when sites went from 2x to 3x (unsure if you had this when you moved to 3x from 2x) ? , although if memory serves most of the cause was due to the newer URL structure in 3x compared to 2x Were you using (on 3x before you upgraded to IPS4) the "legacy" type URL's still ? By this I mean &showtopic=x / &showforum=x etc ? You can still use a robots.txt file to slow "good" bots down with a crawl-delay line. Google you'll need to use their Webmaster tools to slow their bot down. User-agent: * Crawl-delay: 120 If its a specific bot causing problems (Baidu maybe?) you *might* have to consider temporarily banning it via .htaccess or suchlike, although this is not ideal I know but its better than having your site suspended or restricted due to resource use. Depends on how much you need/value SEO with your community niche I guess.
Adlago Posted July 10, 2017 Posted July 10, 2017 This is my list of IP addresses and networks that bother you. If you want to paste this list without changing, in your htaccess file deny from 178.151.216.90 deny from 5.10.83.0/24 deny from 120.62.180.27 deny from 77.190.50.73 deny from 119.147.146.118 deny from 85.114.135.126 deny from 91.200.13.12 deny from 180.76.6.58 deny from 220.181.132.217 deny from 5.135.199.22 deny from 93.84.14.185 deny from 218.30.103.0/24 deny from 188.165.15. deny from 78.137.18.108 deny from 46.229.164.111 deny from 5.9.76.35 deny from 23.22.131.24 deny from 70.39.246.37 deny from 54.91.137.217 deny from 46.229.164.0/24 deny from 185.65.245.134 deny from 136.243.36.97 deny from 164.132.161.0/24 deny from 51.255.65.0/24 deny from 192.243.55.0/24 deny from 151.80.31.0/24 deny from 136.243.36.80 deny from 136.243.36.93 deny from 78.46.174.55 deny from 123.126.113.0/24 deny from 46.229.168.0/24 deny from 46.161.9.20
Genestoy Posted July 10, 2017 Author Posted July 10, 2017 Thanks to both of you have added both to my domain
Recommended Posts
Archived
This topic is now archived and is closed to further replies.