Jump to content

tags giving 429 errors codes


Recommended Posts

2 hours ago, loccom said:

Seems strange how this only happens on a tag page when the page instructs google not to index. I am looking in to the rate limit for this, perhaps i need to slow the crawl down slighty?

Simply having a noindex on the page doesn't necessarily mean they won't crawl it - it just means they won't store it in their index.


The 429 response is likely coming from the Search Flood control, which is hard coded to 30 seconds for spiders.

Link to comment
Share on other sites

1 minute ago, loccom said:

@Ryan Ashbrook My guest group has 20 second search flood control. You mention hard coded for spiders, does that mean search engines are treated differently to guests?

Technically, yes - if you truly want to override this, though, you can do so using constants.php with the following:


Changing the thirty to whatever you wish. I should note, though, that some bots may crawl at an alarming rate (we've had to block some, like Ahrefs, entirely on Community in the Cloud) so you may see an uptick in server resource usage if you decrease this.

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...