Jump to content

Robots.txt File for IPB4

Featured Replies

Posted

Can anyone share their robots.txt file if they have one for IPB4?

In the past this was needed to restrict certain areas from bots etc, with IPS4 it should be needed any longer and has not been included with IPS4. 

If you or anyone can find a use for one with IPS4, share your details and we can review it though.

 

@Rhett funny because right now I generate sitemap for my forum. This is what I have in robots.txt.

User-agent: *
Disallow: /applications
Disallow: /*admin*
Disallow: /datastore
Disallow: /system
Disallow: /index.php?
Disallow: *app=*
Disallow: /uploads
Disallow: /lostpassword
Disallow: /uploads
Disallow: /calendar
Disallow: /login
Disallow: /register
Disallow: /activity
Disallow: /online
Disallow: /statuses
Disallow: /privacy
Disallow: /contact
Disallow: /terms
Disallow: /messenger
Disallow: /activity
Disallow: /search
Disallow: *?do=*
Disallow: *sortby=*
Disallow: *sortdirection=desc
Disallow: *page=1
Disallow: *tab=*
Disallow: *.xml
Disallow: *page=0
Disallow: *type=*
Disallow: *change_section=*
Disallow: */reputation/*
Disallow: */content/*

P.S: Friendly URL and rewrite URLs are enabled.

Edited by Dima Octavian

  • Author

So, @Dima Octavian, is this a good robots.txt to use?

@Lunars this

User-agent: *
Disallow: /applications
Disallow: /admin
Disallow: /datastore
Disallow: /system
Disallow: /index.php?
Disallow: *app=*
Disallow: /uploads
Disallow: /lostpassword
Disallow: /uploads
Disallow: /login
Disallow: /register
Disallow: /activity
Disallow: /online
Disallow: /statuses
Disallow: /privacy
Disallow: /contact
Disallow: /terms
Disallow: /messenger
Disallow: /activity
Disallow: /search
Disallow: *?do=*
Disallow: *sortby=*
Disallow: *sortdirection=desc
Disallow: *page=1
Disallow: *tab=*
Disallow: *.xml
Disallow: *page=0
Disallow: *type=*
Disallow: *change_section=*
Disallow: */reputation/*
Disallow: */content/*
Allow: *type=status
Disallow: *&type=status&do=*

should be OK. But to be 100% sure, someone from IP. Board team should confirm this. I will test it again tonight.

Edited by Dima Octavian

  • Community Expert

So, @Dima Octavian, is this a good robots.txt to use?

​It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? 

​It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? 

​In my second post it will allow it :D

But you're right about personal choice :)

I didn't realize this in my first post, sorry for that.

Edited by Dima Octavian

  • Author

​It’s his personal choice. I wouldn’t just take that over. For example: it disallows Calendar indexing. Do you even have that app? Do you really want calendar entries NOT to appear in search engines? 

​True. But I have a question for this part: 

Disallow: */content/*

What exact content isn't being indexed? 

  • Author

Okay thanks a lot!

Okay thanks a lot!

​You're welcome :)

this robots.txt good for seo?

thanks,

this robots.txt good for seo?

thanks,

​I use it for googlebot, so I think yes.

  • Community Expert

this robots.txt good for seo?

​Depends on what you mean by that. 
As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it. 

If one allows guests to search, then one needs a robots.txt without argument. Allowing a search engine to access the internal site search is extremely damaging SEO-wise, google and others consider it all duplicate content due to sorts and filters....

​Depends on what you mean by that. As I said before: the example robots.txt disallows search engines from crawling certain areas of the site. So certain content will never show up and this can therefore HURT your visibility and ranking. On the other hand, you might want to hide certain areas and simply control what search engines see and do on your site. This is of course some sort of “search engine optimization”, i.e. SEO. But maybe not in the way you meant it.

​Right.

At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -

http://www.animes.co.il/lofiversion/***

What then should I put in robots.txt that no effect of seo But only to help.

thanks,

 

  • Community Expert

​Right.

At the moment I do not have robots.txt, So that search engines go for everything. in webmastertools i got a lots of errors[13,500 pages dowst exist], like -

http://www.animes.co.il/lofiversion/***

What then should I put in robots.txt that no effect of seo But only to help.

thanks,

 

​I don’t know your site. Why do you have thousands of missing pages? Did you move your installation somehow?

Archived

This topic is now archived and is closed to further replies.

Recently Browsing 0

  • No registered users viewing this page.