ASTRAPI Posted March 12, 2010 Posted March 12, 2010 Hello Can anyone post here a robots.txt to add in my public.html to slow down the search bots as it seems taht they are killing my server? Thank you
bfarber Posted March 12, 2010 Posted March 12, 2010 Crawl-delay: 20 That's it. You need to go to Google webmaster tools to slow googlebot down though. I hear it doesn't honor the crawl-delay setting.
3DKiwi Posted March 13, 2010 Posted March 13, 2010 It should start like the following. Here's the first part of mine where I've got a delay plus restricted where they can go (that's the ones that are well behaved and obey the file:User-agent: * Crawl-Delay: 10 Disallow: /ipb/admin/ Disallow: /ipb/cache/ Disallow: /ipb/converge_local/ Disallow: /ipb/hooks/ Disallow: /ipb/ips_kernel/ Disallow: /ipb/retail/ Disallow: /ipb/public/js/ Disallow: /ipb/public/style_captcha/ Disallow: /ipb/public/style_css/ 3DKiwi
Panupat Posted March 13, 2010 Posted March 13, 2010 where to slow down the bot from google webmaster page?
SecondSight Posted March 13, 2010 Posted March 13, 2010 It can be difficult to find sometimes... :) Go to Google Webmaster Tools : https://www.google.c...ebmasters/tools You normally should see you web site url in the "Sites" section (if you added your site before). Clic on it and you'll get a new page. In the dashboard, clic on the first link which is "Site configuration". In the menu appearing, click on "Settings". Then you get a page with a "Crawl rate" section.
SecondSight Posted March 14, 2010 Posted March 14, 2010 If it's your site you don't have there, simply cilc on "Add a site" and follow the instructions for the rest. :)
Recommended Posts
Archived
This topic is now archived and is closed to further replies.