Jump to content

My robots.txt fpr ipb4


Recommended Posts

Posted

Hi...

I want share my robots.txt for ipb4.

I've maked it with a googlequery "site:mysite.xx" and view all pages for see what is ok to maintain and what is best to remove.

atm this is my result: https://laltroweb.it/robots.txt

admin and modcp isn't present in google (i think already excluded with metatag from ipb... so, is best to not insert in robots).

if someone (user or staff) want improve....

regards :)

Posted

Thanks for this. I was surprised to see IPS stop using robots.txt with IPS4. They must think it's unnecessary. Having been plagued by resource hogging bots, I started using one again and it's certainly helped.

Removing permission for guests and search engines to access Activity Stream has also made a difference, I'd totally missed this.

  • 4 weeks later...
Posted

I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F

it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case :)

 

-

i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch) :p

 

Posted
18 hours ago, laltroweb.it said:

I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F

it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case :)

 

-

i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch) :p

 

I realised the problem when I have seen an increase in  Pages with issues in Mobile Usability in Webmasters Tools . Now I am trying to determine what should really be left within robots.txt.

Posted
15 minutes ago, laltroweb.it said:

i've make some edit and now My test seems to be valid :)

:thumbsup: I am looking for a way to limit Search and Activity to guests rather than bots now ;)

  • 4 months later...
Posted
On 11/29/2015 at 10:55 AM, ABGenc said:

:thumbsup: I am looking for a way to limit Search and Activity to guests rather than bots now ;)

You can remove search permissions for the guest group here if needed.

Applications_2016-04-26_13-22-09.jpg

 

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...