Lunars Posted May 11, 2015 Share Posted May 11, 2015 Can anyone share their robots.txt file if they have one for IPB4? Link to comment Share on other sites More sharing options...
Rhett Posted May 11, 2015 Share Posted May 11, 2015 In the past this was needed to restrict certain areas from bots etc, with IPS4 it should be needed any longer and has not been included with IPS4. If you or anyone can find a use for one with IPS4, share your details and we can review it though. Link to comment Share on other sites More sharing options...
Octavian Dima Posted May 11, 2015 Share Posted May 11, 2015 @Rhett funny because right now I generate sitemap for my forum. This is what I have in robots.txt.User-agent: * Disallow: /applications Disallow: /*admin* Disallow: /datastore Disallow: /system Disallow: /index.php? Disallow: *app=* Disallow: /uploads Disallow: /lostpassword Disallow: /uploads Disallow: /calendar Disallow: /login Disallow: /register Disallow: /activity Disallow: /online Disallow: /statuses Disallow: /privacy Disallow: /contact Disallow: /terms Disallow: /messenger Disallow: /activity Disallow: /search Disallow: *?do=* Disallow: *sortby=* Disallow: *sortdirection=desc Disallow: *page=1 Disallow: *tab=* Disallow: *.xml Disallow: *page=0 Disallow: *type=* Disallow: *change_section=* Disallow: */reputation/* Disallow: */content/*P.S: Friendly URL and rewrite URLs are enabled. Link to comment Share on other sites More sharing options...
Lunars Posted May 12, 2015 Author Share Posted May 12, 2015 So, @Dima Octavian, is this a good robots.txt to use? Link to comment Share on other sites More sharing options...
Octavian Dima Posted May 12, 2015 Share Posted May 12, 2015 @Lunars thisUser-agent: * Disallow: /applications Disallow: /admin Disallow: /datastore Disallow: /system Disallow: /index.php? Disallow: *app=* Disallow: /uploads Disallow: /lostpassword Disallow: /uploads Disallow: /login Disallow: /register Disallow: /activity Disallow: /online Disallow: /statuses Disallow: /privacy Disallow: /contact Disallow: /terms Disallow: /messenger Disallow: /activity Disallow: /search Disallow: *?do=* Disallow: *sortby=* Disallow: *sortdirection=desc Disallow: *page=1 Disallow: *tab=* Disallow: *.xml Disallow: *page=0 Disallow: *type=* Disallow: *change_section=* Disallow: */reputation/* Disallow: */content/* Allow: *type=status Disallow: *&type=status&do=*should be OK. But to be 100% sure, someone from IP. Board team should confirm this. I will test it again tonight. Link to comment Share on other sites More sharing options...
opentype Posted May 12, 2015 Share Posted May 12, 2015 So, @Dima Octavian, is this a good robots.txt to use?It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? Link to comment Share on other sites More sharing options...
Octavian Dima Posted May 12, 2015 Share Posted May 12, 2015 It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? In my second post it will allow it But you're right about personal choice I didn't realize this in my first post, sorry for that. Link to comment Share on other sites More sharing options...
Lunars Posted May 12, 2015 Author Share Posted May 12, 2015 It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? True. But I have a question for this part: Disallow: */content/*What exact content isn't being indexed? Link to comment Share on other sites More sharing options...
Octavian Dima Posted May 13, 2015 Share Posted May 13, 2015 True. But I have a question for this part: Disallow: */content/*What exact content isn't being indexed? User activity, for example http://community.invisionpower.com/profile/300690-dima-octavian/content/ Link to comment Share on other sites More sharing options...
Lunars Posted May 13, 2015 Author Share Posted May 13, 2015 Okay thanks a lot! Link to comment Share on other sites More sharing options...
Octavian Dima Posted May 13, 2015 Share Posted May 13, 2015 Okay thanks a lot! You're welcome Link to comment Share on other sites More sharing options...
eden buganim Posted May 14, 2015 Share Posted May 14, 2015 this robots.txt good for seo?thanks, Link to comment Share on other sites More sharing options...
Octavian Dima Posted May 14, 2015 Share Posted May 14, 2015 this robots.txt good for seo?thanks,I use it for googlebot, so I think yes. Link to comment Share on other sites More sharing options...
opentype Posted May 14, 2015 Share Posted May 14, 2015 this robots.txt good for seo?Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it. Link to comment Share on other sites More sharing options...
Marcher Technologies Posted May 14, 2015 Share Posted May 14, 2015 If one allows guests to search, then one needs a robots.txt without argument. Allowing a search engine to access the internal site search is extremely damaging SEO-wise, google and others consider it all duplicate content due to sorts and filters.... Link to comment Share on other sites More sharing options...
eden buganim Posted May 15, 2015 Share Posted May 15, 2015 Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it.Right.At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -http://www.animes.co.il/lofiversion/***What then should I put in robots.txt that no effect of seo But only to help.thanks, Link to comment Share on other sites More sharing options...
opentype Posted May 15, 2015 Share Posted May 15, 2015 Right.At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -http://www.animes.co.il/lofiversion/***What then should I put in robots.txt that no effect of seo But only to help.thanks, I don’t know your site. Why do you have thousands of missing pages? Did you move your installation somehow? Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.