Robots.txt Management via ACP

[Dreadknux]

I'm looking to add some new rules to my robots.txt, and currently have it set so that Invision Community manages optimized rules for me via the setting:

AdminCP > Search Engine Optimisation > Crawl Management > Invision Community optimized

I'm not interested in losing any of the optimized rules provided by this setting. I quite like Invision Community managing the bulk of robots.txt for me. But I'd also not like to confuse the suite by having my own robots.txt file situated in my website's root folder, with the new rules I want to establish. So I'd like to add the new rules to the existing optimized ones.

With that in mind, here's my question. If I select "Custom" in the Crawl Management section of the AdminCP (which appears to be the only method in which I can add custom rules to my robots.txt), will it also maintain optimized rules provided by Invision Community, or will I need to copy paste the optimized rules into the Custom field alongside my new rules to keep things consistent (I'm aware there's a list of the Invision Community optimized ruleset here, but not sure if it is up to date, can anyone confirm)?

[Jim M]

You would want to grab the existing rules at your website.com/robots.txt and then paste them into the custom area in the ACP. The custom area is designed to overwrite our rules so if you want to include them, you'll need to copy/paste them.

[Dreadknux]

Thanks @Jim M - at present I am exclusively using the Invision Community "optimised" feature as my robots.txt (I did have a /robots.txt file a long time ago, but removed it in favour of letting Invision Community handle that for me)... so in that scenario, would selecting "Custom" in AdminCP and adding new rules mean I'd need to copy/paste the rules laid out in the "Invision Community optimized" set, or would all of those optimised rules still be active and my Custom rules would just cascade on top of them?

[Jim M]

On 9/9/2024 at 4:30 PM, Jim M said:

The custom area is designed to overwrite our rules so if you want to include them, you'll need to copy/paste them.

As mentioned, you’ll need to copy and paste them 😉 

[Dreadknux]

1 hour ago, Jim M said:

As mentioned, you’ll need to copy and paste them 😉 

Okay, just making sure. Thanks 🙂 Just to confirm, the rules listed on this page is the current Invision Community optimized set of rules, right? This isn't out of date? I don't know where to check (either in AdminCP or FTP filemanager) to find out what my Invision Community is using myself.

[Jim M]

You can just go to your-base-url/robots.txt to view it 😉 

[Dreadknux]

And er, what do I do if there isn't a robots.txt file in my base url folder..? I assumed that it wasn't there because I had the "Invision Community optimised" setting active, and that information was being stored somewhere else.

[Nathan Explosion]

4 minutes ago, Dreadknux said:

And er, what do I do if there isn't a robots.txt file in my base url folder..?

There isn't a file.

But there will be if you just go to it in a browser - that's what the Invision solution provides, a pseudo robots.txt

 

See? https://www.ynwa.tv/robots.txt

 

[Dreadknux]

53 minutes ago, Nathan Explosion said:

There isn't a file.

But there will be if you just go to it in a browser - that's what the Invision solution provides, a pseudo robots.txt

Awesome, many thanks for the help there! All sorted now. 🙂