Gabriel Petrelli Posted August 30, 2009 Share Posted August 30, 2009 Sometimes I notice an abnormal amount of guests online, so I check the who's online list. When I check the who's online list I notice a lot of guests with the same IP address. Sometimes there can be upwards of 60 - 100 guests from the same ip address or sometimes even 200 - 500 guests. On the who's online list they appear to be checking profiles, reading topics, searching, or viewing other parts of the board. I've tried banning the ip addresses, but they always come back under a different ip address. I've also tried blocking bad bots with htaccess and blocking anything keyweb which does no good. When I did a whois on the ip address it appears to be coming from data centers in different locations for each different ip address. I know it's not a search engine, as Google, Yahoo, and Msn don't hit websites en mass like this. I don't think it's a DDos attack either since my website experiences no downtime or hindered performance when these ip addresses hit my website en mass. What is it and what can I do to stop it? Link to comment Share on other sites More sharing options...
tAPir Posted August 31, 2009 Share Posted August 31, 2009 Be careful you don't ban yourself :) Out of my 140 guests earlier, 130 were me. I have no idea what's causing it but suspected ajax refresh at one time. Link to comment Share on other sites More sharing options...
Gabriel Petrelli Posted August 31, 2009 Author Share Posted August 31, 2009 This occurred on IPB2 as well, I need the actual number of guests, bots viewing the forum for statistical purposes and not some bloated number. Link to comment Share on other sites More sharing options...
rct2·com Posted August 31, 2009 Share Posted August 31, 2009 I think you'll find that it is proably spiders crawling your site (even though you suggest that it isn't). Have you checked out who the IPs belong to? http://ws.arin.net/whois/ http://www.db.ripe.net/whois I think that there is a topic here somewhere (or maybe an article at resources) about constructing a robots.txt file that focuses spiders on topics and posts, stops them trying to reply etc. Link to comment Share on other sites More sharing options...
Gabriel Petrelli Posted August 31, 2009 Author Share Posted August 31, 2009 Why is there so many from the same ip address though? Even Google, Yahoo, Msn etc doesn't index my site in excess like that, with other search engines it's usually just one ip address. Link to comment Share on other sites More sharing options...
rct2·com Posted August 31, 2009 Share Posted August 31, 2009 I think there are so many from one IP address because the spiders are programmed to multithread/multitask. They don't have to wait for the response from one before they fire off another request. I often have Yahoo for example hitting me simultaneously from the same IP address. Link to comment Share on other sites More sharing options...
AaronM Posted September 1, 2009 Share Posted September 1, 2009 I keep getting Google showing up in my online list with 30, 40 & currently 99 Guests showing all using the same IP (213.180.209.10) This has been all day today & yesterday. I even placed this IP in my Ban filter yesterday but it still shows up as online??? Link to comment Share on other sites More sharing options...
ApertureForums Posted September 1, 2009 Share Posted September 1, 2009 Yahoo/Inktomi search bots are infamous for pounding on sites with 100-500 open connections to them at a time, its just how they crawl. Google and MSN usually only send one spider to your site to crawl it so its probably not those two, but if I had to guess it was any, Yahoo would be it. Try to add a line to your robots.txt in your sites root folder that looks like then it will usually take a day or two for that change to take effect but it should fix your problem, assuming its Yahoo User-agent: Slurp Crawl-delay: 5 Link to comment Share on other sites More sharing options...
AaronM Posted September 1, 2009 Share Posted September 1, 2009 Thanks for the info, will add that right now, much appreciated Link to comment Share on other sites More sharing options...
Gabriel Petrelli Posted September 6, 2009 Author Share Posted September 6, 2009 Is it ok to add the crawl delay to all bots/user agents? I added it like this in my robots.txt: User-agent: * Crawl-delay: 5 Then I specified the directories below I don't want search engines to index at all. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.