I would highly suggest ACP manage the robots.txt file for any IPB install. I've been checking Webmaster Tools on a daily basis trying to make all these adjustments to my robots.txt file in hopes of reducing a majority of the Crawl Errors. Since doing so, I've noticed my traffic start to climb, yet I'm still in the thousands as far as HTTP Errors. If anyone has a pretty solid robots.txt file, please share yours here. Here's my current robots.txt file:

User-agent: *

Crawl-delay: 10

Sitemap: /sitemap.xml

Disallow: /admin/

Disallow: /cache/

Disallow: /converge_local/

Disallow: /hooks/

Disallow: /ips_kernel/

Disallow: /retail/

Disallow: /public/style_captcha/

Disallow: /index.php?&unlockUserAgent=1

Disallow: /index.php?app=core&module=task

Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs

Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic

Disallow: /index.php?app=forums&module=extras&section=forward

Disallow: /index.php?app=members&module=messaging

Disallow: /index.php?app=members&module=chat

Disallow: /index.php?app=members&module=search

Disallow: /index.php?app=members&module=search&do=active

Disallow: /index.php?app=members&section=friends

Disallow: /index.php?app=core&module=reports

Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies

Disallow: /index.php?app=core&module=global&section=reputation

Disallow: /index.php?app=blog&module=actions&section=report

Disallow: /index.php?app=forums&module=extras&section=rating

Disallow: /index.php?app=forums&module=forums&section=markasread

Disallow: /*app=core&module=usercp

Disallow: /*app=core&module=usercp

Disallow: /*app=members&module=messaging

Disallow: /*&p=

Disallow: /*&pid=

Disallow: /*&hl=

Disallow: /*&start=

Disallow: /*view__getnewpost$

Disallow: /*view__getlastpost$

Disallow: /*view__old$

Disallow: /*view__new$

Disallow: /*view__getfirst$

Disallow: /*view__getprevious$

Disallow: /*view__getnext$

Disallow: /*view__getlast$

Disallow: /*&view=getnewpost$

Disallow: /*&view=getlastpost$

Disallow: /*&view=old$

Disallow: /*&view=new$

Disallow: /*&view=getfirst$

Disallow: /*&view=getprevious$

Disallow: /*&view=getnext$

Disallow: /*&view=getlast$

Disallow: /*?s=

Disallow: /*&s=

I would really like to see the IPB development team get behind this. I know they must use GWMT, so they have to see the errors people get with their out of the box robots....

 

I would really like to see the IPB development team get behind this. I know they must use GWMT, so they have to see the errors people get with their out of the box robots....

Less work for the board admin to worry about ;)

Thanks for sharing - I had the same as you minus a few lines, which I have now added. The "reputation" line was one I missed and I have a lot of HTTP errors for that module.