Invision Community 4: SEO, prepare for v5 and dormant account notifications Matt November 11, 2024Nov 11
Posted October 29, 20159 yr Hi... I want share my robots.txt for ipb4. I've maked it with a googlequery "site:mysite.xx" and view all pages for see what is ok to maintain and what is best to remove. atm this is my result: https://laltroweb.it/robots.txt admin and modcp isn't present in google (i think already excluded with metatag from ipb... so, is best to not insert in robots). if someone (user or staff) want improve.... regards
October 29, 20159 yr Thanks for this. I was surprised to see IPS stop using robots.txt with IPS4. They must think it's unnecessary. Having been plagued by resource hogging bots, I started using one again and it's certainly helped. Removing permission for guests and search engines to access Activity Stream has also made a difference, I'd totally missed this.
October 30, 20159 yr Author in past I've also used long htaccess code to block bad bot via htaccess, but atm I think isn't necessary for my website :P
November 29, 20159 yr Hi @laltroweb.it, I have realised that your robots.txt causes mobile compatibility tests to fail cause it disallows /uploads folder where some css files are fetched. After removing that line my site became compatible using the following page ; https://www.google.com/webmasters/tools/mobile-friendly/ I had seen a suggestion where the tag searchs should be allowed ( Allow: /search/?tags=* ) . What do you think ?
November 30, 20159 yr Author I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case - i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch)
November 30, 20159 yr 18 hours ago, laltroweb.it said: I've test just now for my website, and seems to be valid: https://www.google.com/webmasters/tools/mobile-friendly/?url=https%3A%2F%2Flaltroweb.it%2F it can disllow crawler in specific directory, but if css is called in source (and file exist, and isn't locked via thaccess or permission etc) this will loaded in any case - i'm sorry, u'have reason. For bot optimization (User can load css without problem, but bot is Limited by robots), seems to be best to permit also uploads folder (i hope google don't start to put all files in googlesearch) I realised the problem when I have seen an increase in Pages with issues in Mobile Usability in Webmasters Tools . Now I am trying to determine what should really be left within robots.txt.
November 30, 20159 yr 15 minutes ago, laltroweb.it said: i've make some edit and now My test seems to be valid I am looking for a way to limit Search and Activity to guests rather than bots now
April 26, 20168 yr On 11/30/2015 at 10:55 PM, ABGenc said: I am looking for a way to limit Search and Activity to guests rather than bots now Me too.
April 26, 20168 yr On 11/29/2015 at 10:55 AM, ABGenc said: I am looking for a way to limit Search and Activity to guests rather than bots now You can remove search permissions for the guest group here if needed.
Archived
This topic is now archived and is closed to further replies.