Introducing Quests: Tailored gamification & bridging in-person events with your community Mike Gitkos 22 hours ago22 hr
May 11, 201510 yr In the past this was needed to restrict certain areas from bots etc, with IPS4 it should be needed any longer and has not been included with IPS4. If you or anyone can find a use for one with IPS4, share your details and we can review it though.
May 11, 201510 yr @Rhett funny because right now I generate sitemap for my forum. This is what I have in robots.txt.User-agent: * Disallow: /applications Disallow: /*admin* Disallow: /datastore Disallow: /system Disallow: /index.php? Disallow: *app=* Disallow: /uploads Disallow: /lostpassword Disallow: /uploads Disallow: /calendar Disallow: /login Disallow: /register Disallow: /activity Disallow: /online Disallow: /statuses Disallow: /privacy Disallow: /contact Disallow: /terms Disallow: /messenger Disallow: /activity Disallow: /search Disallow: *?do=* Disallow: *sortby=* Disallow: *sortdirection=desc Disallow: *page=1 Disallow: *tab=* Disallow: *.xml Disallow: *page=0 Disallow: *type=* Disallow: *change_section=* Disallow: */reputation/* Disallow: */content/*P.S: Friendly URL and rewrite URLs are enabled. Edited May 11, 201510 yr by Dima Octavian
May 12, 201510 yr @Lunars thisUser-agent: * Disallow: /applications Disallow: /admin Disallow: /datastore Disallow: /system Disallow: /index.php? Disallow: *app=* Disallow: /uploads Disallow: /lostpassword Disallow: /uploads Disallow: /login Disallow: /register Disallow: /activity Disallow: /online Disallow: /statuses Disallow: /privacy Disallow: /contact Disallow: /terms Disallow: /messenger Disallow: /activity Disallow: /search Disallow: *?do=* Disallow: *sortby=* Disallow: *sortdirection=desc Disallow: *page=1 Disallow: *tab=* Disallow: *.xml Disallow: *page=0 Disallow: *type=* Disallow: *change_section=* Disallow: */reputation/* Disallow: */content/* Allow: *type=status Disallow: *&type=status&do=*should be OK. But to be 100% sure, someone from IP. Board team should confirm this. I will test it again tonight. Edited May 12, 201510 yr by Dima Octavian
May 12, 201510 yr Community Expert So, @Dima Octavian, is this a good robots.txt to use?It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines?
May 12, 201510 yr It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? In my second post it will allow it But you're right about personal choice I didn't realize this in my first post, sorry for that. Edited May 12, 201510 yr by Dima Octavian
May 12, 201510 yr Author It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? True. But I have a question for this part: Disallow: */content/*What exact content isn't being indexed?
May 13, 201510 yr True. But I have a question for this part: Disallow: */content/*What exact content isn't being indexed? User activity, for example http://community.invisionpower.com/profile/300690-dima-octavian/content/
May 14, 201510 yr Community Expert this robots.txt good for seo?Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it.
May 14, 201510 yr If one allows guests to search, then one needs a robots.txt without argument. Allowing a search engine to access the internal site search is extremely damaging SEO-wise, google and others consider it all duplicate content due to sorts and filters....
May 15, 201510 yr Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it.Right.At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -http://www.animes.co.il/lofiversion/***What then should I put in robots.txt that no effect of seo But only to help.thanks,
May 15, 201510 yr Community Expert Right.At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -http://www.animes.co.il/lofiversion/***What then should I put in robots.txt that no effect of seo But only to help.thanks, I don’t know your site. Why do you have thousands of missing pages? Did you move your installation somehow?
Archived
This topic is now archived and is closed to further replies.