Jump to content

Robots.txt - Customized Crawl Management


Recommended Posts

My seo expert suggested to me that I should exclude any URL's which contain either " php " or " = " from getting crawled by google. My website contains like 25000 url's and as he said it should not exeed the google crawl budget.

From these suggestions I altered the robots.txt file like this. The directory where the the community is located is /members/

Is that correct ? Sorry I have no experience in doint robots.txt files.

--------------

User-agent: *
Disallow: /members/admin/
Disallow: /members/cache/
Disallow: /members/converge_local/
Disallow: /members/hooks/
Disallow: /members/ips_kernel/
Disallow: /members/retail/
Disallow: /members/public/style_captcha/
Disallow: /members/index.php?app=core&module=task
Disallow: /members/index.php?app=forums&module=moderate&section=moderate
Disallow: /members/index.php?app=forums&module=extras&section=forward
Disallow: /members/index.php?app=members&module=chat
Disallow: /members/index.php?&unlockUserAgent=1
Disallow: /members/*&module=search
Allow:      /members/*&search_tags=
Disallow: /members/*app=core&module=global&section=reputation
Disallow: /members/*app=core&module=usercp
Disallow: /members/*app=members&module=messaging
Disallow: /members/*&p=
Disallow: /members/*&pid=
Disallow: /members/*&hl=
Disallow: /members/*&start=
Disallow: /members/*view__getnewpost$
Disallow: /members/*view__getlastpost$
Disallow: /members/*view__old$
Disallow: /members/*view__new$
Disallow: /members/*view__getfirst$
Disallow: /members/*view__getprevious$
Disallow: /members/*view__getnext$
Disallow: /members/*view__getlast$
Disallow: /members/*&view=getnewpost$
Disallow: /members/*&view=getlastpost$
Disallow: /members/*&view=old$
Disallow: /members/*&view=new$
Disallow: /members/*&view=getfirst$
Disallow: /members/*&view=getprevious$
Disallow: /members/*&view=getnext$
Disallow: /members/*&view=getlast$
Disallow: /members/*?s=
Disallow: /members/*&s=
Disallow: /members/index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /members/index.php?app=forums&module=extras&section=rating
Disallow: /members/index.php?app=forums&module=forums&section=markasread
Disallow: /members/*&do=who
Disallow: /members/*&section=dname
Disallow: /members/index.php?
Disallow: /members/profile/
Disallow: /members/search/?
Disallow: /*?do=followers
Disallow: /*?do=find
Disallow: /*?_nodeSelectName
Disallow: /*?do=reportComment

Edited by hpcrazy
Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...