Jump to content

Robots.txt suggestions


Gabriel Torres
 Share

Go to solution Solved by Matt,

Recommended Posts

@Matt Following up our conversation about the excelente upcoming Robots.txt feature, the new, updated tutorial on Robots.txt is excelent, as it has several lines that we didn't have in our robots.txt. However, we ended up with a few lines that we have added in the past that we want you opinion/feedback on:

Disallow: /notifications/
Disallow: /tags/
Disallow: /checkout/
Disallow: /subscriptions/*/?do=purchase
Disallow: /index.php?*
Disallow: /*?app=*
Disallow: /*/?do=download
Disallow: /*?do=email

Note: we have the tags feature disabled.

Should we keep them? Should we remove them? What is your opinion?

Thank you in advance! 🙂

 

Link to comment
Share on other sites

  • Management
  • Solution

I think if you have tags disabled, it's fine to disallow those.

Notifications, checkout, subs purchase are all fine to omit.
I'm unclear what google will index with 'index.php?*' but this is likely fine as long as you have rewritten friendly URLs on.

I'd need to audit to see what 'app=*' would disallow. Generally it should be fine but there may be some functionality that doesn't have a friendly URL.

do=download/do=email both fine to disallow. I'll tweak the robots.txt file to include some of those.

Link to comment
Share on other sites

  • 3 weeks later...
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...