Invision Community 4: SEO, prepare for v5 and dormant account notifications By Matt Monday at 02:04 PM
Gabriel Petrelli Posted August 30, 2009 Posted August 30, 2009 Sometimes I notice an abnormal amount of guests online, so I check the who's online list. When I check the who's online list I notice a lot of guests with the same IP address. Sometimes there can be upwards of 60 - 100 guests from the same ip address or sometimes even 200 - 500 guests. On the who's online list they appear to be checking profiles, reading topics, searching, or viewing other parts of the board. I've tried banning the ip addresses, but they always come back under a different ip address. I've also tried blocking bad bots with htaccess and blocking anything keyweb which does no good. When I did a whois on the ip address it appears to be coming from data centers in different locations for each different ip address. I know it's not a search engine, as Google, Yahoo, and Msn don't hit websites en mass like this. I don't think it's a DDos attack either since my website experiences no downtime or hindered performance when these ip addresses hit my website en mass. What is it and what can I do to stop it?
tAPir Posted August 31, 2009 Posted August 31, 2009 Be careful you don't ban yourself :) Out of my 140 guests earlier, 130 were me. I have no idea what's causing it but suspected ajax refresh at one time.
Gabriel Petrelli Posted August 31, 2009 Author Posted August 31, 2009 This occurred on IPB2 as well, I need the actual number of guests, bots viewing the forum for statistical purposes and not some bloated number.
rct2·com Posted August 31, 2009 Posted August 31, 2009 I think you'll find that it is proably spiders crawling your site (even though you suggest that it isn't). Have you checked out who the IPs belong to? http://ws.arin.net/whois/ http://www.db.ripe.net/whois I think that there is a topic here somewhere (or maybe an article at resources) about constructing a robots.txt file that focuses spiders on topics and posts, stops them trying to reply etc.
Gabriel Petrelli Posted August 31, 2009 Author Posted August 31, 2009 Why is there so many from the same ip address though? Even Google, Yahoo, Msn etc doesn't index my site in excess like that, with other search engines it's usually just one ip address.
rct2·com Posted August 31, 2009 Posted August 31, 2009 I think there are so many from one IP address because the spiders are programmed to multithread/multitask. They don't have to wait for the response from one before they fire off another request. I often have Yahoo for example hitting me simultaneously from the same IP address.
AaronM Posted September 1, 2009 Posted September 1, 2009 I keep getting Google showing up in my online list with 30, 40 & currently 99 Guests showing all using the same IP (213.180.209.10) This has been all day today & yesterday. I even placed this IP in my Ban filter yesterday but it still shows up as online???
ApertureForums Posted September 1, 2009 Posted September 1, 2009 Yahoo/Inktomi search bots are infamous for pounding on sites with 100-500 open connections to them at a time, its just how they crawl. Google and MSN usually only send one spider to your site to crawl it so its probably not those two, but if I had to guess it was any, Yahoo would be it. Try to add a line to your robots.txt in your sites root folder that looks like then it will usually take a day or two for that change to take effect but it should fix your problem, assuming its Yahoo User-agent: Slurp Crawl-delay: 5
AaronM Posted September 1, 2009 Posted September 1, 2009 Thanks for the info, will add that right now, much appreciated
Gabriel Petrelli Posted September 6, 2009 Author Posted September 6, 2009 Is it ok to add the crawl delay to all bots/user agents? I added it like this in my robots.txt: User-agent: * Crawl-delay: 5 Then I specified the directories below I don't want search engines to index at all.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.