Jump to content

Use of robots.txt


Simon Harvey

Recommended Posts

Posted

In a few posts on this community I have ready about renaming robotstxt.txt to use as a robots.txt template file however I am not able to find robotstxt.txt on my installation.

Is this not used anymore? Is it advised to create a robots.txt file?

Thanks

  • 1 month later...
Posted

Hello @opentype

what would you do with links like these appearing on top of results page,    (as the only links of your site, on that search)

http://parents.org.gr/forum/topic/15147-institouto-gonimotitas-loytradis/?do=email&comment=1048539

http://parents.org.gr/forum/notifications/?do=followers&follow_app=forums&follow_area=topic&follow_id=75243

Currently  if i search for my domain name + something else in greek, 

parents.org.gr ρωτήστε την ειδικό   

or 

δοκιμή parents.org.gr 

those links appear  exactly on top of search results, instead of a normal, (with content) link that should appear, or just the domain name. 

what is your recommendation ?    I have seen a 5% increase in bounce traffic since the migration. 

I am discussing it right now with support, they recommend adding a specific  robot.txt to hide those.    

/*?do=email&*

Doing a search now, to see what others are doing, ...is it just me who has this problem?.....??

 

Posted

You seem to allow a lot of activity for guests. That’s why these links are even crawled. In a typical setup, they don’t appear for guests. Allowing emailing for guests is pretty problematic. Every spammer could use your page as a spam distribution center. 

But anyway. As long as you want to keep your current settings and permissions, then yes, you would want to set up robots.txt rules to filter out those URLs. 

Posted

Thank you very much for your reply. 

no i didnt get that those links were the result of that.     

following and emailing.... ofcourse i dont want that for guests.      these things are very new for us  because of the huuuuge  difference  between   vbulletin 3,8  and  ips....! 

I will go and change that asap! . 

But i think those things should be something like a default setting when you migrate? ... no ? 

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...