Interesting Robot

[KevinMc]

Has anyone noticed this Robot heavily targeting their site lately? I'm not quite sure what it is, and if I should block it.

It looks like it's from an indexing search engine called Ahrefs.

[Grumpy]

If any robot heavily targets your site, feel free to block it. Though, the definition of heavy would be of importance.

[Dmacleo]

yeah ukraine bot from ahrefs.com

this one didn't really do much on my end.

see it every now and then for a few moments

[SocialBrands]

Block it. You don't want your site showing up in their index as a place for spammers to get dofollow backlinks.

[KevinMc]

Thanks for all the help so far, everyone. I've blocked it for now but I have a question about that. They crawl for their search engine, as shown from the second link I provided in the original post. So, why would blocking this robot be a bad thing? Am I not understanding their intent?

[Grumpy]

Thanks for all the help so far, everyone. I've blocked it for now but I have a question about that. They crawl for their search engine, as shown from the second link I provided in the original post. So, why would blocking this robot be a bad thing? Am I not understanding their intent?

Did you mean... why would it be a good thing to block a search engine?

Even if they're a real search engine, would a trustworthy company go around bashing random people's servers? By the very act, it rather proves that they're problematic, a less favorable option. Search engines aren't exactly valuable nor does it need catering to if no one uses it. Conversely, if it gets blocked, it has poor results, thus no one will use them. Their failure to understand the symbiotic nature of publishers and search engines is only their own downfall.

If you really did mean to ask, why would it be a bad thing?

It's not...?

Well, search engines not indexing you means no one from that search engine can come to you. But if no one (or relatively none) uses it, who cares!

Their intent is clear, same with any other heavy crawlers.

Ignoring robots.txt gives an advantage of access to more content. Although it can have potential downfalls, like weird results, unwanted results and honeypots.

Hard hitting crawlers allow them to visit more in a small span of time, thus able to play the catch up game. Google had a chance to crawl for over a decade now. How long did they have? Though, in the end, you could just load balance better to multiple sites as there are virtually infinite sites to visit on the Internet.

[Dmacleo]

Their failure to understand the symbiotic nature of publishers and search engines is only their own downfall.

this .

if their crawler acts politely there are no issues. if its rude its blocked.

this one has been polite to me.