Spider Bot User-Agent

[fernis]

"msnbot=MSN Bot" should be included in the default list of spiders.

[Katsuma]

Thanks! ^_^ I will add it right now. Are there anymore that I can add?

[Andrew S.]

There's a whole list of search engine spiders that you can find here. :)

[WoLeRiNe`]

There's a whole list of search engine spiders that you can find here. :)

The topic has created on Nov 4 2004, 10:47 PM.
In this 2,5years, the news spiders wasn't born :D ?

[Andrew S.]

No, the spiders are updated, thats when the topic was officially created though is back in 2004, but Dan updated it recently.

[WoLeRiNe`]

update date is February 2006
and new bots not updated yet
as;

ypn-rss.overture.com=Yahoo Publisher Network
YahooYSMcm/1.0.0=Yahoo Publisher Network
Yahoo-MMCrawler/3.x=Yahoo Publisher Network
YahooYSMcm/2.0.0=Yahoo Publisher Network

and

eBot=elchron.cz
Intelix=Microton.cz

[Axel Wers]

eBot=elchron.cz

Intelix=Microton.cz

Yes, they are visiting my forum sometimes :D

[WoLeRiNe`]

ypn-rss.overture.com=Yahoo Publisher Network
YahooYSMcm/1.0.0=Yahoo Publisher Network
Yahoo-MMCrawler/3.x=Yahoo Publisher Network
YahooYSMcm/2.0.0=Yahoo Publisher Network

These bots also haven't in the files :)

[bfarber]

You only need to put

yahoo=Yahoo Publisher Network

There's no need to list every individual user agent - the code above would match all instances of "Yahoo" and list it as "Yahoo Publisher Network"

[WoLeRiNe`]

There aren't an official site for a list of bots actual :unsure: ?

[Katsuma]

There aren't an official site for a list of bots actual :unsure: ?

I hope not. :) I got the idea a few days ago and I don't want to have any competition. :P I just need to get some more scratch for another IPB License. ;)

[Jason H]

Please be careful with the bot list.. If you add ALL the bots in the online lists, you run the risk of causing problems with your forum.

There's a KB article about it, but you'll get an error in class_sessions.php on some screens if the bot list is too big. I believe the limit is about 64k.

Best to pick the ones you know and add them.. If you add everything, you're heading for trouble.

[WoLeRiNe`]

Thanks for this info Mr. Jason H

[cooldude7273]

You only need to put

yahoo=Yahoo Publisher Network

There's no need to list every individual user agent - the code above would match all instances of "Yahoo" and list it as "Yahoo Publisher Network"

so yahoo=YahooWhatever would work for yahoodkjdkejdkhbot1, yahoo---d jdj454 bot2, and such like that?

That'd make my day so much easier when I revamp my list! :)

[WoLeRiNe`]

so

yahoo=YahooWhatever

would work for

yahoodkjdkejdkhbot1

,

yahoo---d jdj454 bot2

, and such like that?

That'd make my day so much easier when I revamp my list! :)

Euh, yes i have tested and

YahooYSMcm/1.0.0=Yahoo Publisher Network

Yahoo-MMCrawler/3.x=Yahoo Publisher Network

YahooYSMcm/2.0.0=Yahoo Publisher Network

for these bot, you can just added "yahoo=Yahoo Publisher Network". You can seen 3bots in 1 :)


Thanks bfarber

[bfarber]

It's no problem - it's a common misunderstanding I see.

IPB tries to match what is on the left to the user agent - if it matches, it displays what is on the right, and stops parsing right then.

If 3 entries are put in, and all 3 have the same text on the right side of the equal sign, they will be grouped in the spider logs anyways, so you have no benefit to 3 entries that are meant to display the same.