Robots.txt suggestions

[Gabriel Torres]

@Matt Following up our conversation about the excelente upcoming Robots.txt feature, the new, updated tutorial on Robots.txt is excelent, as it has several lines that we didn't have in our robots.txt. However, we ended up with a few lines that we have added in the past that we want you opinion/feedback on:

Disallow: /notifications/
Disallow: /tags/
Disallow: /checkout/
Disallow: /subscriptions/*/?do=purchase
Disallow: /index.php?*
Disallow: /*?app=*
Disallow: /*/?do=download
Disallow: /*?do=email

Note: we have the tags feature disabled.

Should we keep them? Should we remove them? What is your opinion?

Thank you in advance! 🙂

 

[Matt]

I think if you have tags disabled, it's fine to disallow those.

Notifications, checkout, subs purchase are all fine to omit.
I'm unclear what google will index with 'index.php?*' but this is likely fine as long as you have rewritten friendly URLs on.

I'd need to audit to see what 'app=*' would disallow. Generally it should be fine but there may be some functionality that doesn't have a friendly URL.

do=download/do=email both fine to disallow. I'll tweak the robots.txt file to include some of those.

[SeNioR-]

I wrote about the robots.txt file back in 2015 but nobody wanted to listen 😛 

My topic: I advise you to remove "share by email"

Other topics:

• Robots.txt request!
• Seo error on /?do=email
• Robots.txt - IP.Board 4.x
• IPS4 ~ robots.txt
• Use of robots.txt
• Broken links due to ?do=email&comment=47180, etc

[Gabriel Torres]

@Matt

One new suggestion:

Disallow: /discover/content-started/

 

[Yamamura]

4 hours ago, Gabriel Torres said:

@Matt

One new suggestion:

Disallow: /discover/content-started/

 

This url is already blocked in the current version robots.txt