Ilya Hoilik Posted February 5, 2015 Posted February 5, 2015 Can you include robotstxt.txt file for IPS 4.0 to archive with RC version?
White Miku Posted March 29, 2015 Posted March 29, 2015 +1Can you add an example of robots.txt to archives?
Management Lindy Posted May 3, 2015 Management Posted May 3, 2015 We do not include a robots.txt in IPS4 as it is not needed. Guest links can be removed entirely for inaccessible sections, so there's nothing for search engines to exclude as they won't see it anyway.
pyro888 Posted June 8, 2015 Posted June 8, 2015 We do not include a robots.txt in IPS4 as it is not needed. Guest links can be removed entirely for inaccessible sections, so there's nothing for search engines to exclude as they won't see it anyway. Why is it relevant for Google to index for example the ?do=reportComment pages???
opentype Posted June 8, 2015 Posted June 8, 2015 Why is it relevant for Google to index for example the ?do=reportComment pages??? It isn’t, and that’s why Google won’t usually even see that link, because it gets hidden through the guest permissions – that’s what Lindy explained.
pyro888 Posted June 8, 2015 Posted June 8, 2015 Hmm, in guest mode I receive indeed a 404 error (good).Google's index is not updated yet then. The pages are cached around March this year.
Unlucky Posted May 11, 2016 Posted May 11, 2016 can you disallow google from indexing the members profiles (/profile/) from within admin setting guest restrictions? Thanks,
Colonel_mortis Posted May 11, 2016 Posted May 11, 2016 On 03/05/2015 at 8:24 AM, Lindy said: We do not include a robots.txt in IPS4 as it is not needed. Guest links can be removed entirely for inaccessible sections, so there's nothing for search engines to exclude as they won't see it anyway. Yes there is - anything with a CSRF key in the URL will cause issues.
recifbox Posted May 12, 2016 Posted May 12, 2016 on the "Use recommended settings?" i'm on "manual" settings but when we take a look at the xml generated by the sitemap.php we can see clearly that the option as little as no effect and yes we have open support incident on this and other bug on the sitemap, url issue etc etc...
tnn Posted May 12, 2016 Posted May 12, 2016 3 hours ago, recifbox said: on the "Use recommended settings?" i'm on "manual" settings but when we take a look at the xml generated by the sitemap.php we can see clearly that the option as little as no effect and yes we have open support incident on this and other bug on the sitemap, url issue etc etc... I think you should have created a new topic on your sitemap issue. This topic is about robots.txt started by Ilya Hoilik.
MeMaBlue Posted May 18, 2017 Posted May 18, 2017 hello, @Colonel_mortis is there a workaround on this? what issues do you mean ? can you please elaborate? @Lindy hello, is there any documentatiion specifically that will explain this to a newbie " Guest links can be removed entirely for inaccessible sections"
Recommended Posts
Archived
This topic is now archived and is closed to further replies.