Jump to content

robots.txt to slow down bots


Recommended Posts

It should start like the following. Here's the first part of mine where I've got a delay plus restricted where they can go (that's the ones that are well behaved and obey the file:

User-agent: *

Crawl-Delay: 10

Disallow: /ipb/admin/

Disallow: /ipb/cache/

Disallow: /ipb/converge_local/

Disallow: /ipb/hooks/

Disallow: /ipb/ips_kernel/

Disallow: /ipb/retail/

Disallow: /ipb/public/js/

Disallow: /ipb/public/style_captcha/

Disallow: /ipb/public/style_css/



3DKiwi

Link to comment
Share on other sites

It can be difficult to find sometimes... :)

Go to Google Webmaster Tools : https://www.google.c...ebmasters/tools
You normally should see you web site url in the "Sites" section (if you added your site before).
post-16027-126845779203_thumb.jpg
Clic on it and you'll get a new page.
In the dashboard, clic on the first link which is "Site configuration".
In the menu appearing, click on "Settings".
Then you get a page with a "Crawl rate" section.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...