Jump to content
Announcing Achievements! ×

Community

IP. SEO is a great product.


Connor T
 Share

Recommended Posts

IP. SEO is amazing. I have found lots of mad spiders that crawl non-stop, and been forced to slow them down.

%7Boption%7D
(put some commas in that output for me :P )

One bot called MJ12 crawled 8000 pages in almost 6 hours. For my small site that was unacceptable, I just banned the useragent because that bot had some bad reviews online. Next come Google, the Google Bot respects my robots.txt and crawls at a perfect rate. The same cannot be said about the Bing Bot though. It ignores any crawl-delay I've set and continue to crawl 44,000 pages in 5 days. I'll be having a word with Bing about this. As for the SEO logging. I will log for a continuous 7 days, and then I will turn it off. I know mySQL is not liking all this logging.

The Search Activity has helped me find random topics that have a large amount of views. The strangest topics sometimes attract the largest crowd. Its helped me to expand my forum where I need to, and decrease it where activity lacks.

Thanks for the add-on. Its great.

Link to comment
Share on other sites


The same cannot be said about the Bing Bot though. It ignores any crawl-delay I've set and continue to crawl 44,000 pages in 5 days.




That's because you're misunderstanding what crawl-delay means to some crawlers. In contrast to other crawlers, Bing interprets that number as a relative weighting, not as a cap in seconds. Referring to the Bing team blog, they indicate that if you set it to 1 like you are, you're telling the crawler to go slow (not telling it to wait 1 second between hits), and if you set it to 10, you're telling the crawler to go extremely slow.
Link to comment
Share on other sites

 Share

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

We use technologies, such as cookies, to customise content and advertising, to provide social media features and to analyse traffic to the site. We also share information about your use of our site with our trusted social media, advertising and analytics partners. See more about cookies and our Privacy Policy