Improving invision robots.txt

[Daddy]

/checkout and potentially /search should be disallowed by default. Seeing a lot of IndexNow requests for those pages. I've blocked it on my end but thought it could be useful to add to the default

[Matt]

I agree with checkout, but I'll need to think about /search as this would exclude the tag search results which have some SEO merit.

[Claudia999]

4 hours ago, Matt said:

but I'll need to think about /search as this would exclude the tag search results which have some SEO merit.

Hi @Matt

could you explain please how tag search results can have SEO merit? 

Maybe I've been doing it wrong for years, but not a single url on my site with /tags/ is indexed by Google.
For years I have been suggesting to improve the tag pages also for SEO reasons. I would like to have a decent intro and an image on the main tag pages. A dream would be tag pages with "sections" - first Pages database posts with the tag, then forum posts, then blog posts, then photos and so on. But that‘s a dream 😉

As they are now, they are just search results pages in another way. Or am I really doing something wrong there for years?

[SeNioR-]

On 6/10/2022 at 6:41 PM, Daddy said:

Seeing a lot of IndexNow requests for those pages.

+ /modcp/

[The Old Man]

Surprised to see /contact/ isn't in the robots.txt! 😬