Jump to content

Blocking sppam bots


Recommended Posts

Yeah ditto to Aiwa.., StopForumSpam is a great way. I've used SFS even on sites that are not forums it works great, and one thing I normally do if the board is busy is new users have to get atleast they're first post moderated before they can post freely. I find this just tends to keep the odd human spammer away as they can't be bothered, and the other ways prevent the non human ones. :smile:

It's always best to use 3rd party tools, although there is lists of ip's that are known offenders this take up space in iptables which slows down web performance. It's better to just check these users on registration.

robots.txt is basically useless, as most of these robots don't obey that file at all. It's best to have a proper firewall in place to weed out bad user agents. I like having multiple lines of defense, and even if the registration will fail it's just a waste of server resources. A lot of these bots are using either spoofed headers, or errors in the user-agent line. So an easy way to null those connections completely.

Link to comment
Share on other sites

robots.txt is not useless...

Legit crawlers DO obey it... And it can be used to SLOW the crawlers down so they don't eat resources on your board...

Yes, there are crawlers that don't obey it... Baidu for example. But they are the exception and DO need to be handled via .htaccess if you don't want them crawling your board like mad...

Link to comment
Share on other sites

robots.txt is not useless...

Legit crawlers DO obey it... And it can be used to SLOW the crawlers down so they don't eat resources on your board...

Yes, there are crawlers that don't obey it... Baidu for example. But they are the exception and DO need to be handled via .htaccess if you don't want them crawling your board like mad...

I didn't mean in general. I meant that robots that don't obey, obviously a robot.txt is useless since they don't obey it.

But I see now OP is asking about crawlers anyway so kind of irrelevant now. :smile:

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...