IP. SEO is amazing. I have found lots of mad spiders that crawl non-stop, and been forced to slow them down.


(put some commas in that output for me :P )

One bot called MJ12 crawled 8000 pages in almost 6 hours. For my small site that was unacceptable, I just banned the useragent because that bot had some bad reviews online. Next come Google, the Google Bot respects my robots.txt and crawls at a perfect rate. The same cannot be said about the Bing Bot though. It ignores any crawl-delay I've set and continue to crawl 44,000 pages in 5 days. I'll be having a word with Bing about this. As for the SEO logging. I will log for a continuous 7 days, and then I will turn it off. I know mySQL is not liking all this logging.

The Search Activity has helped me find random topics that have a large amount of views. The strangest topics sometimes attract the largest crowd. Its helped me to expand my forum where I need to, and decrease it where activity lacks.

Thanks for the add-on. Its great.

One bot called MJ12 crawled 8000 pages in almost 6 hours.

I am not sure if it thanks to IP.SEO. Bot MJ12 (in my case Majestic12.co.uk) was very active in last time, here is my log, that bot has more than 100 thousand entries.

The same cannot be said about the Bing Bot though. It ignores any crawl-delay I've set and continue to crawl 44,000 pages in 5 days.

That's because you're misunderstanding what crawl-delay means to some crawlers. In contrast to other crawlers, Bing interprets that number as a relative weighting, not as a cap in seconds. Referring to the Bing team blog, they indicate that if you set it to 1 like you are, you're telling the crawler to go slow (not telling it to wait 1 second between hits), and if you set it to 10, you're telling the crawler to go extremely slow.

How come I don't have the "Last hit xxxminutes ago" column in my IP.SEO? (1.1.)

Thanks Mat, I should of read more clearly.



@Breadfan,

I haven't updated to 1.1 yet, but there is a good possibility that its not there in 1.1 I'll confirm later tonight.

How come I don't have the "Last hit xxxminutes ago" column in my IP.SEO? (1.1.)

That's a screenshot from the Spider Logs, not from IP.SEO.