hpcrazy Posted December 26, 2021 Share Posted December 26, 2021 (edited) My seo expert suggested to me that I should exclude any URL's which contain either " php " or " = " from getting crawled by google. My website contains like 25000 url's and as he said it should not exeed the google crawl budget. From these suggestions I altered the robots.txt file like this. The directory where the the community is located is /members/ Is that correct ? Sorry I have no experience in doint robots.txt files. -------------- User-agent: * Disallow: /members/admin/ Disallow: /members/cache/ Disallow: /members/converge_local/ Disallow: /members/hooks/ Disallow: /members/ips_kernel/ Disallow: /members/retail/ Disallow: /members/public/style_captcha/ Disallow: /members/index.php?app=core&module=task Disallow: /members/index.php?app=forums&module=moderate§ion=moderate Disallow: /members/index.php?app=forums&module=extras§ion=forward Disallow: /members/index.php?app=members&module=chat Disallow: /members/index.php?&unlockUserAgent=1 Disallow: /members/*&module=search Allow: /members/*&search_tags= Disallow: /members/*app=core&module=global§ion=reputation Disallow: /members/*app=core&module=usercp Disallow: /members/*app=members&module=messaging Disallow: /members/*&p= Disallow: /members/*&pid= Disallow: /members/*&hl= Disallow: /members/*&start= Disallow: /members/*view__getnewpost$ Disallow: /members/*view__getlastpost$ Disallow: /members/*view__old$ Disallow: /members/*view__new$ Disallow: /members/*view__getfirst$ Disallow: /members/*view__getprevious$ Disallow: /members/*view__getnext$ Disallow: /members/*view__getlast$ Disallow: /members/*&view=getnewpost$ Disallow: /members/*&view=getlastpost$ Disallow: /members/*&view=old$ Disallow: /members/*&view=new$ Disallow: /members/*&view=getfirst$ Disallow: /members/*&view=getprevious$ Disallow: /members/*&view=getnext$ Disallow: /members/*&view=getlast$ Disallow: /members/*?s= Disallow: /members/*&s= Disallow: /members/index.php?app=core&module=global§ion=login&do=deleteCookies Disallow: /members/index.php?app=forums&module=extras§ion=rating Disallow: /members/index.php?app=forums&module=forums§ion=markasread Disallow: /members/*&do=who Disallow: /members/*§ion=dname Disallow: /members/index.php? Disallow: /members/profile/ Disallow: /members/search/? Disallow: /*?do=followers Disallow: /*?do=find Disallow: /*?_nodeSelectName Disallow: /*?do=reportComment Edited December 27, 2021 by hpcrazy Link to comment Share on other sites More sharing options...
Recommended Posts