Googlebot is slurping up my bandwidth like crazy.

I nearly went over my bandwidth limit last month, so I've been watching the online users periodically through the day, and there are always dozens of instances of the googlebot agent all over the board, all with the same IP (66.249.67.55), and they're there 24/7.

I checked the stats for the site over at Google's webmaster tools, and the reported crawl rate was WAY off, saying that the high end of the average was only 17 pages per day. I just counted 35 google-guests.

Obviously I don't want to disallow the thing, and I did put a delay in the robot.txt, but it seems weird to me that the bot just literally moved into my site permanently!

One thing I did notice was that the robots.txt reads:


Disallow: /index.php?act=Post&CODE=02
Disallow: /index.php?act=Print&client=printer
Disallow: /index.php?app=calendar
Disallow: /index.php?app=calendar$ ..... etc.

My board, however is not in the root directory, but is at mysite.com/forums/index.php

Should I prepend the /forums/ directory to all of those urls?

You just need to throttle Googlebot.

I have it limited to 2.5 seconds between requests. This is a screen shot of my current online users that I took just a few minutes ago. I have 7 pages like this (this is just the top 1/2 of the page), but only six actual users are on line right now and 12 legitimate guests.

You just need to throttle Google.

I gladly fixed this for you :)

Ain't it the truth.