Gauravk Posted April 26, 2016 Share Posted April 26, 2016 Wondering why IPS4 doesnt come with robot.txt? In my site I am still having old one and I think it need an update to manage certain stuff that we really dont want to get crawl like empty profile pages, admin area and few testing pages. Not sure wht all we have added overtime in 3.X.X so appreciate if someone can help cleaning in below robot.txt please with bare minimum entries, as we are seeing lot of crawl error from GWT, and doubting it may be because of this wrong robot.txt. Thanks. User-agent: * Disallow: /admin/ Disallow: /cache/ Disallow: /converge_local/ Disallow: /hooks/ Disallow: /ips_kernel/ Disallow: /retail/ Disallow: /public/style_captcha/ Disallow: /discover/ Disallow: /profile/ Disallow: /latest Disallow: /updates Disallow: /classy Disallow: /index.php?app=core&module=task Disallow: /index.php?app=forums&module=moderate§ion=moderate Disallow: /index.php?app=forums&module=extras§ion=forward Disallow: /index.php?app=members&module=chat Disallow: /index.php?&unlockUserAgent=1 Disallow: /*&module=search Allow: /*&search_tags= Disallow: /*app=core&module=global§ion=reputation Disallow: /*app=core&module=usercp Disallow: /*app=members&module=messaging Disallow: /*&p= Disallow: /*&pid= Disallow: /*&hl= Disallow: /*&start= Disallow: /*view__getnewpost$ Disallow: /*view__getlastpost$ Disallow: /*view__old$ Disallow: /*view__new$ Disallow: /*view__getfirst$ Disallow: /*view__getprevious$ Disallow: /*view__getnext$ Disallow: /*view__getlast$ Disallow: /*&view=getnewpost$ Disallow: /*&view=getlastpost$ Disallow: /*&view=old$ Disallow: /*&view=new$ Disallow: /*&view=getfirst$ Disallow: /*&view=getprevious$ Disallow: /*&view=getnext$ Disallow: /*&view=getlast$ Disallow: /*?s= Disallow: /*&s= Link to comment Share on other sites More sharing options...
tnn Posted April 26, 2016 Share Posted April 26, 2016 Here is some info laltroweb.it posted that may be of interest to you.My robots.txt fpr ipb4 Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.