ipbfuck Posted October 29, 2015 Posted October 29, 2015 Hi... I want share my robots.txt for ipb4. I've maked it with a googlequery "site:mysite.xx" and view all pages for see what is ok to maintain and what is best to remove. atm this is my result: https://laltroweb.it/robots.txt admin and modcp isn't present in google (i think already excluded with metatag from ipb... so, is best to not insert in robots). if someone (user or staff) want improve.... regards
The Old Man Posted October 29, 2015 Posted October 29, 2015 Thanks for this. I was surprised to see IPS stop using robots.txt with IPS4. They must think it's unnecessary. Having been plagued by resource hogging bots, I started using one again and it's certainly helped. Removing permission for guests and search engines to access Activity Stream has also made a difference, I'd totally missed this.
ipbfuck Posted October 30, 2015 Author Posted October 30, 2015 in past I've also used long htaccess code to block bad bot via htaccess, but atm I think isn't necessary for my website :P
ipbfuck Posted October 30, 2015 Author Posted October 30, 2015 i've make more changes in robots.txt. now also more badbot exclusion
ABGenc Posted November 29, 2015 Posted November 29, 2015 Hi @laltroweb.it, I have realised that your robots.txt causes mobile compatibility tests to fail cause it disallows /uploads folder where some css files are fetched. After removing that line my site became compatible using the following page ; https://www.google.com/webmasters/tools/mobile-friendly/ I had seen a suggestion where the tag searchs should be allowed ( Allow: /search/?tags=* ) . What do you think ?
ipbfuck Posted November 30, 2015 Author Posted November 30, 2015 I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case - i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch)
ABGenc Posted November 30, 2015 Posted November 30, 2015 18 hours ago, laltroweb.it said: I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case - i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch) I realised the problem when I have seen an increase in Pages with issues in Mobile Usability in Webmasters Tools . Now I am trying to determine what should really be left within robots.txt.
ipbfuck Posted November 30, 2015 Author Posted November 30, 2015 i've make some edit and now My test seems to be valid
ABGenc Posted November 30, 2015 Posted November 30, 2015 15 minutes ago, laltroweb.it said: i've make some edit and now My test seems to be valid I am looking for a way to limit Search and Activity to guests rather than bots now
Gauravk Posted April 26, 2016 Posted April 26, 2016 On 11/30/2015 at 10:55 PM, ABGenc said: I am looking for a way to limit Search and Activity to guests rather than bots now Me too.
Rhett Posted April 26, 2016 Posted April 26, 2016 On 11/29/2015 at 10:55 AM, ABGenc said: I am looking for a way to limit Search and Activity to guests rather than bots now You can remove search permissions for the guest group here if needed.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.