Robots.txt advice

[Rimi]

This was initially going to be a response to this bug report (http://community.invisionpower.com/resources/bugs.html/_/ip-board/search-engines-and-lots-of-403-errors-for-search-r39888) but I think it's more of a suggestion/improvement.

I really don't think you should be fixing things by updating the robots.txt, but I understand the need to do so. However, the problem is that those changes which are extremely beneficial are, in fact, probably never going to be noticed by anyone. Various robots.txt suggestions have been posted throughout this forum, and I know I myself was given a valuable line to add via a ticket for an issue I was having. So I think it'd be good for IPB to add a new part to the seo advice section that analyzes a site's robots.txt file.

1) It should check if the file actually exists in the proper location (remember robots.txt has to be in the site root - I assume most people both forget to rename the robotstxt.txt file and also if they install in a /forums/ directory they forget to move the file)

2) IPB should have suggested lines to add to the file and then subsequently check if those lines are present in the robots.txt.

This section will allow IPS to push SEO fixes by updating the robots.txt and clients can benefit by having a center in their ACP which alerts them to these changes.