known issue Badge stats regeneration uses excessive database resource

[NexusMods]

Hi,

We've been having issues every 6 hours when the badge status cache expires and they regenerate. This causes our (rather large) DB to hit 100% CPU and the forums become unresponsive.

Looking into this, the issue is that multiple threads will try to get the badge stats at once when they expire: \IPS\core\Achievements\_Badge::getBadgeStats()

We have approximately 40 concurrent requests trying to regenerate these stats every 6 hours.

As a work around, we are temporarily updating the stats expiration while we re-generate the statistics, this results in it only running once per 6 hours (instead of 40 times)

 

if( !isset( $stats['expiration'] ) OR time() > $stats['expiration'] OR !isset( $stats['badgeCount'][ $this->id ] ) )
{
  // Write the old stats back to the store while we regenerate them
  $stats['expiration'] = time() + 60;
  \IPS\Data\Store::i()->badgeStats = json_encode( $stats );

  $stats	= NULL;
}

 

[Jim M]

Thank you for reporting this. I have submitted this internally to be further investigated.

[media]

14 hours ago, NexusMods said:

Hi,

We've been having issues every 6 hours when the badge status cache expires and they regenerate. This causes our (rather large) DB to hit 100% CPU and the forums become unresponsive.

Looking into this, the issue is that multiple threads will try to get the badge stats at once when they expire: \IPS\core\Achievements\_Badge::getBadgeStats()

We have approximately 40 concurrent requests trying to regenerate these stats every 6 hours.

As a work around, we are temporarily updating the stats expiration while we re-generate the statistics, this results in it only running once per 6 hours (instead of 40 times)

 

if( !isset( $stats['expiration'] ) OR time() > $stats['expiration'] OR !isset( $stats['badgeCount'][ $this->id ] ) )
{
  // Write the old stats back to the store while we regenerate them
  $stats['expiration'] = time() + 60;
  \IPS\Data\Store::i()->badgeStats = json_encode( $stats );

  $stats	= NULL;
}

 

Great catch... 🙂