Lunars Posted May 11, 2015 Posted May 11, 2015 Can anyone share their robots.txt file if they have one for IPB4?
Rhett Posted May 11, 2015 Posted May 11, 2015 In the past this was needed to restrict certain areas from bots etc, with IPS4 it should be needed any longer and has not been included with IPS4. If you or anyone can find a use for one with IPS4, share your details and we can review it though.
Octavian Dima Posted May 11, 2015 Posted May 11, 2015 @Rhett funny because right now I generate sitemap for my forum. This is what I have in robots.txt.User-agent: * Disallow: /applications Disallow: /*admin* Disallow: /datastore Disallow: /system Disallow: /index.php? Disallow: *app=* Disallow: /uploads Disallow: /lostpassword Disallow: /uploads Disallow: /calendar Disallow: /login Disallow: /register Disallow: /activity Disallow: /online Disallow: /statuses Disallow: /privacy Disallow: /contact Disallow: /terms Disallow: /messenger Disallow: /activity Disallow: /search Disallow: *?do=* Disallow: *sortby=* Disallow: *sortdirection=desc Disallow: *page=1 Disallow: *tab=* Disallow: *.xml Disallow: *page=0 Disallow: *type=* Disallow: *change_section=* Disallow: */reputation/* Disallow: */content/*P.S: Friendly URL and rewrite URLs are enabled.
Lunars Posted May 12, 2015 Author Posted May 12, 2015 So, @Dima Octavian, is this a good robots.txt to use?
Octavian Dima Posted May 12, 2015 Posted May 12, 2015 @Lunars thisUser-agent: * Disallow: /applications Disallow: /admin Disallow: /datastore Disallow: /system Disallow: /index.php? Disallow: *app=* Disallow: /uploads Disallow: /lostpassword Disallow: /uploads Disallow: /login Disallow: /register Disallow: /activity Disallow: /online Disallow: /statuses Disallow: /privacy Disallow: /contact Disallow: /terms Disallow: /messenger Disallow: /activity Disallow: /search Disallow: *?do=* Disallow: *sortby=* Disallow: *sortdirection=desc Disallow: *page=1 Disallow: *tab=* Disallow: *.xml Disallow: *page=0 Disallow: *type=* Disallow: *change_section=* Disallow: */reputation/* Disallow: */content/* Allow: *type=status Disallow: *&type=status&do=*should be OK. But to be 100% sure, someone from IP. Board team should confirm this. I will test it again tonight.
opentype Posted May 12, 2015 Posted May 12, 2015 So, @Dima Octavian, is this a good robots.txt to use?It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines?
Octavian Dima Posted May 12, 2015 Posted May 12, 2015 It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? In my second post it will allow it But you're right about personal choice I didn't realize this in my first post, sorry for that.
Lunars Posted May 12, 2015 Author Posted May 12, 2015 It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? True. But I have a question for this part: Disallow: */content/*What exact content isn't being indexed?
Octavian Dima Posted May 13, 2015 Posted May 13, 2015 True. But I have a question for this part: Disallow: */content/*What exact content isn't being indexed? User activity, for example http://community.invisionpower.com/profile/300690-dima-octavian/content/
Octavian Dima Posted May 14, 2015 Posted May 14, 2015 this robots.txt good for seo?thanks,I use it for googlebot, so I think yes.
opentype Posted May 14, 2015 Posted May 14, 2015 this robots.txt good for seo?Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it.
Marcher Technologies Posted May 14, 2015 Posted May 14, 2015 If one allows guests to search, then one needs a robots.txt without argument. Allowing a search engine to access the internal site search is extremely damaging SEO-wise, google and others consider it all duplicate content due to sorts and filters....
eden buganim Posted May 15, 2015 Posted May 15, 2015 Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it.Right.At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -http://www.animes.co.il/lofiversion/***What then should I put in robots.txt that no effect of seo But only to help.thanks,
opentype Posted May 15, 2015 Posted May 15, 2015 Right.At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -http://www.animes.co.il/lofiversion/***What then should I put in robots.txt that no effect of seo But only to help.thanks, I don’t know your site. Why do you have thousands of missing pages? Did you move your installation somehow?
Recommended Posts
Archived
This topic is now archived and is closed to further replies.