Invision Community 4: SEO, prepare for v5 and dormant account notifications Matt November 11, 2024Nov 11
Posted June 27, 200717 yr Since the upgrade to 2.3.1 the amount of guests on my board (mostly bots) have increased a lot!I use the new skin. Is this skin better accessible for bots?
June 27, 200717 yr Quite a few people have noticed in increase in bots lately (myself included and im on 2.2 still):)
June 27, 200717 yr It is amazing, we broke our online record again this morning Most users ever online was 803 on Today, 10:19 AM soon we will have more guests than members, currently on 247 guests :huh:
June 27, 200717 yr Community Expert Management I'm assuming it's because the bots have noted that the site has changed as are therefore spidering it again heavily.
June 27, 200717 yr I've read, also, that Yahoo changed recently how they index a site. They're sending multiple requests from different IP addresses instead of from one. Since we use the IP to match a single session for spiders, this causes each request from Yahoo to be seen as a separate session (or guest).
June 27, 200717 yr It is amazing, we broke our online record again this morning Most users ever online was 803 on Today, 10:19 AM soon we will have more guests than members, currently on 247 guests :huh:Pretty normal for us to have more "most online" than actual members :lol: :( :unsure:I've read, also, that Yahoo changed recently how they index a site. They're sending multiple requests from different IP addresses instead of from one. Since we use the IP to match a single session for spiders, this causes each request from Yahoo to be seen as a separate session (or guest).I`ve heard this too yesterday :)
June 27, 200717 yr Then you would need to add the spiders to your spiders list so they appear as spiders and not guests.Where do you find all these spiders?>> http://ipbots.info/Then I'm pretty sure they don't count as online users.
June 27, 200717 yr Do NOT install the "Biggest Bot List", 90% of the listed spiders would visit only one or two times your forums per month, and you'll slow down your board with no benefit.
June 27, 200717 yr The "most popular" list is recommended (as he ^ suggested). Its also been vastly shortened in 5.0 (the full list).Its that new feature that lets multiple bots be recognized as one identity no matter which IP. So with how Yahoo has multiple ip spiders, before each would have to be specified. Now only one entry matches them all.
June 27, 200717 yr No, no, that's not entirely correct. ;)When we generate the spider session id we use the IP address. It doesn't matter what you enter in the bot list in the ACP - multiple IP addresses will count as multiple sessions. It has always worked this way (this is not something different in 2.2 or 2.3).One question though - why should spider visits not count against users online? In fact, these users ARE online on your site. :blink: Spider or not.
June 27, 200717 yr I find it rather amusing lol, on a certain rival site it has made them wonder why we are so popular all of a sudden :P I just don't like how it messes up my online list, but can live with it ..
June 27, 200717 yr When we generate the spider session id we use the IP address. It doesn't matter what you enter in the bot list in the ACP - multiple IP addresses will count as multiple sessions. It has always worked this way (this is not something different in 2.2 or 2.3).Well, and loading a HUGE array of spiders list - which contains 90% of quite unused bots - is a great thing? I think that only spiders that really comes to your board should be listed, the others, that usually represent quite nothing should not be listed in the ACP.
June 27, 200717 yr I've read, also, that Yahoo changed recently how they index a site. They're sending multiple requests from different IP addresses instead of from one. Since we use the IP to match a single session for spiders, this causes each request from Yahoo to be seen as a separate session (or guest).Yep - had about 500 yahoo sessions yesterday, at about this time.
June 27, 200717 yr Do NOT install the "Biggest Bot List", 90% of the listed spiders would visit only one or two times your forums per month, and you'll slow down your board with no benefit.Oh? Could you post your spider list here? Thanks.
June 27, 200717 yr i noticed it too. Does anybody know how to show only "most registered visiters at one time"?
June 27, 200717 yr Oh? Could you post your spider list here? Thanks.Of courseGooglebot=Google.comFeedFetcher=Google FeedfetcherTeoma=Teomawhatuseek=What You Seekia_archiver=Alexamediapartners=Google AdsenseVoilaBot=VoilaSlurp=Yahoomsnbot=MSNgigabot=GigablastZyBorg=WiseNutFAST-WebCrawler=FastDeepIndex=DeepIndexHenriLeRobotMirago=MiragoOpenbot=OpenFindantibot=Antibotpsbot=PicSearchSzukacz=SzukaczNaverBot=Naverappie=WhalhelloExabot=ExaleadmozDex=mozDexSnapbot=Snapvoyager=KosmixNetMonitor=ExpertMonitorIRLbot=IRLbotAccoona-AI-Agent=Accoonabbot=BBotOMGILI=OmgiliJyxobot=JyxoBaiduspider=Baiduappie=Walhelloarks=ArksEchO=EchoMagpie=MagpieNutch=NutchMJ12bot=MJ12botSpeedy=EntirewebSynooBot=SynooThis is my list. And this is enough.
June 28, 200717 yr One question though - why should spider visits not count against users online? In fact, these users ARE online on your site. :blink: Spider or not.How about because they're not USERS in the computer sense of the term, but automated scripted systems, my thought about Most Users Online, was a pissing contest to show how popular your board was to real people, not to show how many search engines know you exist.
June 28, 200717 yr A suggestion has been submitted to Yahoo to revert what they are doinghttp://suggestions.yahoo.com/detail/?prop=...r&fid=31431http://digg.com/software/Yahoo_Spider_wrec..._guest_trackingI'd urge everyone who sees this to vote for the suggestion. There's no feasible reason Yahoo needs to visit your site from over 100 IP addresses at one moment. I would imagine this would hurt a lot of people on shared hosting with limited resources as well.and we're not the only one reporting this of course ;)A quick web search shows people all over reporting this (including users of other BBs software)
July 2, 200717 yr Can someone explain to me how to block or delay the Yahoo spiders?Just add an entry for them in your robots.txt file. To stop them from indexing your site:User-agent: Slurp Disallow: / To delay intervals between page requests (time in seconds): User-agent: Slurp Crawl-delay: 30http://help.yahoo.com/help/us/ysearch/slurp/slurp-02.html
Archived
This topic is now archived and is closed to further replies.