robots.txt to slow down bots

[ASTRAPI]

Hello

Can anyone post here a robots.txt to add in my public.html to slow down the search bots as it seems taht they are killing my server?

Thank you

[bfarber]

Crawl-delay: 20

That's it. You need to go to Google webmaster tools to slow googlebot down though. I hear it doesn't honor the crawl-delay setting.

[ASTRAPI]

Only one line inside the robots.txt file?

[3DKiwi]

It should start like the following. Here's the first part of mine where I've got a delay plus restricted where they can go (that's the ones that are well behaved and obey the file:

User-agent: *

Crawl-Delay: 10

Disallow: /ipb/admin/

Disallow: /ipb/cache/

Disallow: /ipb/converge_local/

Disallow: /ipb/hooks/

Disallow: /ipb/ips_kernel/

Disallow: /ipb/retail/

Disallow: /ipb/public/js/

Disallow: /ipb/public/style_captcha/

Disallow: /ipb/public/style_css/

3DKiwi

[Panupat]

where to slow down the bot from google webmaster page?

[SecondSight]

It can be difficult to find sometimes... :)

Go to Google Webmaster Tools : https://www.google.c...ebmasters/tools
You normally should see you web site url in the "Sites" section (if you added your site before).

Clic on it and you'll get a new page.
In the dashboard, clic on the first link which is "Site configuration".
In the menu appearing, click on "Settings".
Then you get a page with a "Crawl rate" section.

[ASTRAPI]

I don't have it there....

[SecondSight]

If it's your site you don't have there, simply cilc on "Add a site" and follow the instructions for the rest. :)