Dreadknux Posted September 9 Share Posted September 9 I'm looking to add some new rules to my robots.txt, and currently have it set so that Invision Community manages optimized rules for me via the setting: AdminCP > Search Engine Optimisation > Crawl Management > Invision Community optimized I'm not interested in losing any of the optimized rules provided by this setting. I quite like Invision Community managing the bulk of robots.txt for me. But I'd also not like to confuse the suite by having my own robots.txt file situated in my website's root folder, with the new rules I want to establish. So I'd like to add the new rules to the existing optimized ones. With that in mind, here's my question. If I select "Custom" in the Crawl Management section of the AdminCP (which appears to be the only method in which I can add custom rules to my robots.txt), will it also maintain optimized rules provided by Invision Community, or will I need to copy paste the optimized rules into the Custom field alongside my new rules to keep things consistent (I'm aware there's a list of the Invision Community optimized ruleset here, but not sure if it is up to date, can anyone confirm)? Emmanu 1 Link to comment Share on other sites More sharing options...
Jim M Posted September 9 Share Posted September 9 You would want to grab the existing rules at your website.com/robots.txt and then paste them into the custom area in the ACP. The custom area is designed to overwrite our rules so if you want to include them, you'll need to copy/paste them. Link to comment Share on other sites More sharing options...
Dreadknux Posted September 10 Author Share Posted September 10 Thanks @Jim M - at present I am exclusively using the Invision Community "optimised" feature as my robots.txt (I did have a /robots.txt file a long time ago, but removed it in favour of letting Invision Community handle that for me)... so in that scenario, would selecting "Custom" in AdminCP and adding new rules mean I'd need to copy/paste the rules laid out in the "Invision Community optimized" set, or would all of those optimised rules still be active and my Custom rules would just cascade on top of them? Link to comment Share on other sites More sharing options...
Jim M Posted September 10 Share Posted September 10 On 9/9/2024 at 4:30 PM, Jim M said: The custom area is designed to overwrite our rules so if you want to include them, you'll need to copy/paste them. As mentioned, you’ll need to copy and paste them 😉 Link to comment Share on other sites More sharing options...
Dreadknux Posted September 10 Author Share Posted September 10 1 hour ago, Jim M said: As mentioned, you’ll need to copy and paste them 😉 Okay, just making sure. Thanks 🙂 Just to confirm, the rules listed on this page is the current Invision Community optimized set of rules, right? This isn't out of date? I don't know where to check (either in AdminCP or FTP filemanager) to find out what my Invision Community is using myself. Link to comment Share on other sites More sharing options...
Jim M Posted September 10 Share Posted September 10 You can just go to your-base-url/robots.txt to view it 😉 Link to comment Share on other sites More sharing options...
Dreadknux Posted September 11 Author Share Posted September 11 And er, what do I do if there isn't a robots.txt file in my base url folder..? I assumed that it wasn't there because I had the "Invision Community optimised" setting active, and that information was being stored somewhere else. Link to comment Share on other sites More sharing options...
Nathan Explosion Posted September 11 Share Posted September 11 4 minutes ago, Dreadknux said: And er, what do I do if there isn't a robots.txt file in my base url folder..? There isn't a file. But there will be if you just go to it in a browser - that's what the Invision solution provides, a pseudo robots.txt See? https://www.ynwa.tv/robots.txt Marc and Dreadknux 1 1 Link to comment Share on other sites More sharing options...
Dreadknux Posted September 11 Author Share Posted September 11 53 minutes ago, Nathan Explosion said: There isn't a file. But there will be if you just go to it in a browser - that's what the Invision solution provides, a pseudo robots.txt Awesome, many thanks for the help there! All sorted now. 🙂 Link to comment Share on other sites More sharing options...
Recommended Posts