Jump to content

Need help in correcting robot.txt


Recommended Posts

Posted

Wondering why IPS4 doesnt come with robot.txt? In my site I am still having old one and I think it need an update to manage certain stuff that we really dont want to get crawl like empty profile pages, admin area and few testing pages. 

Not sure wht all we have added overtime in 3.X.X so appreciate if someone can help cleaning in below robot.txt please with bare minimum entries, as we are seeing lot of crawl error from GWT, and doubting it may be because of this wrong robot.txt. Thanks.

 

User-agent: *
Disallow: /admin/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /discover/
Disallow: /profile/
Disallow: /latest
Disallow: /updates
Disallow: /classy
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=forums&module=moderate&section=moderate
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*&module=search
Allow:	  /*&search_tags=
Disallow: /*app=core&module=global&section=reputation
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*?s=
Disallow: /*&s=

 

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...