Simon Harvey Posted April 14, 2017 Posted April 14, 2017 In a few posts on this community I have ready about renaming robotstxt.txt to use as a robots.txt template file however I am not able to find robotstxt.txt on my installation. Is this not used anymore? Is it advised to create a robots.txt file? Thanks
opentype Posted April 14, 2017 Posted April 14, 2017 There is no need for a robots.txt. You can create one, if there is public content you don’t want indexed.
MeMaBlue Posted May 17, 2017 Posted May 17, 2017 Hello @opentype. what would you do with links like these appearing on top of results page, (as the only links of your site, on that search) http://parents.org.gr/forum/topic/15147-institouto-gonimotitas-loytradis/?do=email&comment=1048539 http://parents.org.gr/forum/notifications/?do=followers&follow_app=forums&follow_area=topic&follow_id=75243 Currently if i search for my domain name + something else in greek, parents.org.gr ρωτήστε την ειδικό or δοκιμή parents.org.gr those links appear exactly on top of search results, instead of a normal, (with content) link that should appear, or just the domain name. what is your recommendation ? I have seen a 5% increase in bounce traffic since the migration. I am discussing it right now with support, they recommend adding a specific robot.txt to hide those. /*?do=email&* Doing a search now, to see what others are doing, ...is it just me who has this problem?.....??
opentype Posted May 17, 2017 Posted May 17, 2017 You seem to allow a lot of activity for guests. That’s why these links are even crawled. In a typical setup, they don’t appear for guests. Allowing emailing for guests is pretty problematic. Every spammer could use your page as a spam distribution center. But anyway. As long as you want to keep your current settings and permissions, then yes, you would want to set up robots.txt rules to filter out those URLs.
MeMaBlue Posted May 17, 2017 Posted May 17, 2017 Thank you very much for your reply. no i didnt get that those links were the result of that. following and emailing.... ofcourse i dont want that for guests. these things are very new for us because of the huuuuge difference between vbulletin 3,8 and ips....! I will go and change that asap! . But i think those things should be something like a default setting when you migrate? ... no ?
Recommended Posts
Archived
This topic is now archived and is closed to further replies.