ipbfuck Posted October 29, 2015 Share Posted October 29, 2015 Hi... I want share my robots.txt for ipb4. I've maked it with a googlequery "site:mysite.xx" and view all pages for see what is ok to maintain and what is best to remove. atm this is my result: https://laltroweb.it/robots.txt admin and modcp isn't present in google (i think already excluded with metatag from ipb... so, is best to not insert in robots). if someone (user or staff) want improve.... regards Link to comment Share on other sites More sharing options...
The Old Man Posted October 29, 2015 Share Posted October 29, 2015 Thanks for this. I was surprised to see IPS stop using robots.txt with IPS4. They must think it's unnecessary. Having been plagued by resource hogging bots, I started using one again and it's certainly helped. Removing permission for guests and search engines to access Activity Stream has also made a difference, I'd totally missed this. Link to comment Share on other sites More sharing options...
ipbfuck Posted October 30, 2015 Author Share Posted October 30, 2015 in past I've also used long htaccess code to block bad bot via htaccess, but atm I think isn't necessary for my website :P Link to comment Share on other sites More sharing options...
ipbfuck Posted October 30, 2015 Author Share Posted October 30, 2015 i've make more changes in robots.txt. now also more badbot exclusion Link to comment Share on other sites More sharing options...
sobrenome Posted November 1, 2015 Share Posted November 1, 2015 Thanks a lot for sharing! Link to comment Share on other sites More sharing options...
tnn Posted November 1, 2015 Share Posted November 1, 2015 Thank you very much @laltroweb.it Link to comment Share on other sites More sharing options...
ABGenc Posted November 29, 2015 Share Posted November 29, 2015 Hi @laltroweb.it, I have realised that your robots.txt causes mobile compatibility tests to fail cause it disallows /uploads folder where some css files are fetched. After removing that line my site became compatible using the following page ; https://www.google.com/webmasters/tools/mobile-friendly/ I had seen a suggestion where the tag searchs should be allowed ( Allow: /search/?tags=* ) . What do you think ? Link to comment Share on other sites More sharing options...
ipbfuck Posted November 30, 2015 Author Share Posted November 30, 2015 I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case - i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch) Link to comment Share on other sites More sharing options...
ABGenc Posted November 30, 2015 Share Posted November 30, 2015 18 hours ago, laltroweb.it said: I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case - i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch) I realised the problem when I have seen an increase in Pages with issues in Mobile Usability in Webmasters Tools . Now I am trying to determine what should really be left within robots.txt. Link to comment Share on other sites More sharing options...
ipbfuck Posted November 30, 2015 Author Share Posted November 30, 2015 i've make some edit and now My test seems to be valid Link to comment Share on other sites More sharing options...
ABGenc Posted November 30, 2015 Share Posted November 30, 2015 15 minutes ago, laltroweb.it said: i've make some edit and now My test seems to be valid I am looking for a way to limit Search and Activity to guests rather than bots now Link to comment Share on other sites More sharing options...
Gauravk Posted April 26, 2016 Share Posted April 26, 2016 On 11/30/2015 at 10:55 PM, ABGenc said: I am looking for a way to limit Search and Activity to guests rather than bots now Me too. Link to comment Share on other sites More sharing options...
Rhett Posted April 26, 2016 Share Posted April 26, 2016 On 11/29/2015 at 10:55 AM, ABGenc said: I am looking for a way to limit Search and Activity to guests rather than bots now You can remove search permissions for the guest group here if needed. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.