Mesmer Posted June 27, 2007 Share Posted June 27, 2007 Since the upgrade to 2.3.1 the amount of guests on my board (mostly bots) have increased a lot!I use the new skin. Is this skin better accessible for bots? Link to comment Share on other sites More sharing options...
AndyF Posted June 27, 2007 Share Posted June 27, 2007 Quite a few people have noticed in increase in bots lately (myself included and im on 2.2 still):) Link to comment Share on other sites More sharing options...
zigs Posted June 27, 2007 Share Posted June 27, 2007 It is amazing, we broke our online record again this morning Most users ever online was 803 on Today, 10:19 AM soon we will have more guests than members, currently on 247 guests :huh: Link to comment Share on other sites More sharing options...
Management Matt Posted June 27, 2007 Management Share Posted June 27, 2007 I'm assuming it's because the bots have noted that the site has changed as are therefore spidering it again heavily. Link to comment Share on other sites More sharing options...
bfarber Posted June 27, 2007 Share Posted June 27, 2007 I've read, also, that Yahoo changed recently how they index a site. They're sending multiple requests from different IP addresses instead of from one. Since we use the IP to match a single session for spiders, this causes each request from Yahoo to be seen as a separate session (or guest). Link to comment Share on other sites More sharing options...
AndyF Posted June 27, 2007 Share Posted June 27, 2007 It is amazing, we broke our online record again this morning Most users ever online was 803 on Today, 10:19 AM soon we will have more guests than members, currently on 247 guests :huh:Pretty normal for us to have more "most online" than actual members :lol: :( :unsure:I've read, also, that Yahoo changed recently how they index a site. They're sending multiple requests from different IP addresses instead of from one. Since we use the IP to match a single session for spiders, this causes each request from Yahoo to be seen as a separate session (or guest).I`ve heard this too yesterday :) Link to comment Share on other sites More sharing options...
Stuart Elliott Posted June 27, 2007 Share Posted June 27, 2007 How about not having bots count against "users online" ? Link to comment Share on other sites More sharing options...
Cybertimber2009 Posted June 27, 2007 Share Posted June 27, 2007 Then you would need to add the spiders to your spiders list so they appear as spiders and not guests.Where do you find all these spiders?>> http://ipbots.info/Then I'm pretty sure they don't count as online users. Link to comment Share on other sites More sharing options...
ErwinB Posted June 27, 2007 Share Posted June 27, 2007 Do NOT install the "Biggest Bot List", 90% of the listed spiders would visit only one or two times your forums per month, and you'll slow down your board with no benefit. Link to comment Share on other sites More sharing options...
Cybertimber2009 Posted June 27, 2007 Share Posted June 27, 2007 The "most popular" list is recommended (as he ^ suggested). Its also been vastly shortened in 5.0 (the full list).Its that new feature that lets multiple bots be recognized as one identity no matter which IP. So with how Yahoo has multiple ip spiders, before each would have to be specified. Now only one entry matches them all. Link to comment Share on other sites More sharing options...
bfarber Posted June 27, 2007 Share Posted June 27, 2007 No, no, that's not entirely correct. ;)When we generate the spider session id we use the IP address. It doesn't matter what you enter in the bot list in the ACP - multiple IP addresses will count as multiple sessions. It has always worked this way (this is not something different in 2.2 or 2.3).One question though - why should spider visits not count against users online? In fact, these users ARE online on your site. :blink: Spider or not. Link to comment Share on other sites More sharing options...
zigs Posted June 27, 2007 Share Posted June 27, 2007 I find it rather amusing lol, on a certain rival site it has made them wonder why we are so popular all of a sudden :P I just don't like how it messes up my online list, but can live with it .. Link to comment Share on other sites More sharing options...
bfarber Posted June 27, 2007 Share Posted June 27, 2007 rofl Link to comment Share on other sites More sharing options...
ErwinB Posted June 27, 2007 Share Posted June 27, 2007 When we generate the spider session id we use the IP address. It doesn't matter what you enter in the bot list in the ACP - multiple IP addresses will count as multiple sessions. It has always worked this way (this is not something different in 2.2 or 2.3).Well, and loading a HUGE array of spiders list - which contains 90% of quite unused bots - is a great thing? I think that only spiders that really comes to your board should be listed, the others, that usually represent quite nothing should not be listed in the ACP. Link to comment Share on other sites More sharing options...
Nathan Explosion Posted June 27, 2007 Share Posted June 27, 2007 I've read, also, that Yahoo changed recently how they index a site. They're sending multiple requests from different IP addresses instead of from one. Since we use the IP to match a single session for spiders, this causes each request from Yahoo to be seen as a separate session (or guest).Yep - had about 500 yahoo sessions yesterday, at about this time. Link to comment Share on other sites More sharing options...
TCWT Posted June 27, 2007 Share Posted June 27, 2007 Do NOT install the "Biggest Bot List", 90% of the listed spiders would visit only one or two times your forums per month, and you'll slow down your board with no benefit.Oh? Could you post your spider list here? Thanks. Link to comment Share on other sites More sharing options...
Fabian169 Posted June 27, 2007 Share Posted June 27, 2007 i noticed it too. Does anybody know how to show only "most registered visiters at one time"? Link to comment Share on other sites More sharing options...
Michael Posted June 27, 2007 Share Posted June 27, 2007 http://www.ipsbeyond.com/forums/index.php?...;showarticle=67 Link to comment Share on other sites More sharing options...
ErwinB Posted June 27, 2007 Share Posted June 27, 2007 Oh? Could you post your spider list here? Thanks.Of courseGooglebot=Google.comFeedFetcher=Google FeedfetcherTeoma=Teomawhatuseek=What You Seekia_archiver=Alexamediapartners=Google AdsenseVoilaBot=VoilaSlurp=Yahoomsnbot=MSNgigabot=GigablastZyBorg=WiseNutFAST-WebCrawler=FastDeepIndex=DeepIndexHenriLeRobotMirago=MiragoOpenbot=OpenFindantibot=Antibotpsbot=PicSearchSzukacz=SzukaczNaverBot=Naverappie=WhalhelloExabot=ExaleadmozDex=mozDexSnapbot=Snapvoyager=KosmixNetMonitor=ExpertMonitorIRLbot=IRLbotAccoona-AI-Agent=Accoonabbot=BBotOMGILI=OmgiliJyxobot=JyxoBaiduspider=Baiduappie=Walhelloarks=ArksEchO=EchoMagpie=MagpieNutch=NutchMJ12bot=MJ12botSpeedy=EntirewebSynooBot=SynooThis is my list. And this is enough. Link to comment Share on other sites More sharing options...
Stuart Elliott Posted June 28, 2007 Share Posted June 28, 2007 One question though - why should spider visits not count against users online? In fact, these users ARE online on your site. :blink: Spider or not.How about because they're not USERS in the computer sense of the term, but automated scripted systems, my thought about Most Users Online, was a pissing contest to show how popular your board was to real people, not to show how many search engines know you exist. Link to comment Share on other sites More sharing options...
bfarber Posted June 28, 2007 Share Posted June 28, 2007 A suggestion has been submitted to Yahoo to revert what they are doinghttp://suggestions.yahoo.com/detail/?prop=...r&fid=31431http://digg.com/software/Yahoo_Spider_wrec..._guest_trackingI'd urge everyone who sees this to vote for the suggestion. There's no feasible reason Yahoo needs to visit your site from over 100 IP addresses at one moment. I would imagine this would hurt a lot of people on shared hosting with limited resources as well.and we're not the only one reporting this of course ;)A quick web search shows people all over reporting this (including users of other BBs software) Link to comment Share on other sites More sharing options...
Mesmer Posted June 28, 2007 Share Posted June 28, 2007 Thanks for the update!I informed my members about this. Link to comment Share on other sites More sharing options...
Strike X Posted June 28, 2007 Share Posted June 28, 2007 Forum online counting script needs updating.. very dodgy now. Link to comment Share on other sites More sharing options...
Mesmer Posted July 2, 2007 Share Posted July 2, 2007 Can someone explain to me how to block or delay the Yahoo spiders? Link to comment Share on other sites More sharing options...
ColdinCbus Posted July 2, 2007 Share Posted July 2, 2007 Can someone explain to me how to block or delay the Yahoo spiders?Just add an entry for them in your robots.txt file. To stop them from indexing your site:User-agent: Slurp Disallow: / To delay intervals between page requests (time in seconds): User-agent: Slurp Crawl-delay: 30http://help.yahoo.com/help/us/ysearch/slurp/slurp-02.html Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.