Huge increase in server load after upgrading to 4.4.x

[SebastienG]

Yes, it's for the guests pages. I configure it for 30 min, but the pages are not deleted.

For example, my first entry since the purge 

cache_expire: 1553193974

I think the page will be updated in the next refresh. But when a speeder come and parse thousand pages ...

I've trying to use redis 

\define( 'REDIS_ENABLED', false );
\define( 'STORE_METHOD', 'Redis' );
\define( 'STORE_CONFIG', '[]' );
\define( 'CACHE_METHOD', 'Redis' );
\define( 'REDIS_CONFIG', '{"server":"127.0.0.1","port":6379,"password":""}' );
\define( 'CACHE_PAGE_TIMEOUT', 1800 );

With this parameters, guests pages are stored in database

I can't configure Redis in global because i'm using arrowchat and it's not working when the sessions are stored in redis.

If anybody have a script or a way for doing an asynchronous transfert for the sessions Redis -> database, i can enable it. 

For the moment I have to truncate the database dans optimize it for reduce disk size.

In 2 hours 7.2Go of data (without binary logs of mysql )

[ExiledVip3r]

On 3/20/2019 at 6:09 AM, bfarber said:

Please upgrade to 4.4.2 and see if this resolves your issue. We identified one or two slow queries fixed in 4.4.2 (one was specific to the "Unread content" stream, which queries against the core_search_index table, so that might explain what you're seeing).

For what it's worth, though, we're noticing loads dropping pretty significantly upon upgrading from 4.3 to 4.4 on our network.

Load is still absurdly high for me on 4.4.2. Lots of idle sql connections and php-fpm is still eating all of the cpu.

[bfarber]

19 hours ago, SebastienG said:

Yes, it's for the guests pages. I configure it for 30 min, but the pages are not deleted.

For example, my first entry since the purge 

cache_expire: 1553193974

I think the page will be updated in the next refresh. But when a speeder come and parse thousand pages ...

I've trying to use redis 

\define( 'REDIS_ENABLED', false );
\define( 'STORE_METHOD', 'Redis' );
\define( 'STORE_CONFIG', '[]' );
\define( 'CACHE_METHOD', 'Redis' );
\define( 'REDIS_CONFIG', '{"server":"127.0.0.1","port":6379,"password":""}' );
\define( 'CACHE_PAGE_TIMEOUT', 1800 );

With this parameters, guests pages are stored in database

I can't configure Redis in global because i'm using arrowchat and it's not working when the sessions are stored in redis.

If anybody have a script or a way for doing an asynchronous transfert for the sessions Redis -> database, i can enable it. 

For the moment I have to truncate the database dans optimize it for reduce disk size.

In 2 hours 7.2Go of data (without binary logs of mysql )

If you want to store guest page caches in Redis, you will need to define OUTPUT_CACHE_METHOD in your constants.php as "Redis".

[SebastienG]

6 hours ago, bfarber said:

If you want to store guest page caches in Redis, you will need to define OUTPUT_CACHE_METHOD in your constants.php as "Redis".

Thank you, I don't find this option

And for the REDIS_ENABLED, their is option to store everything except sessions in Redis ?

 

[Gabriel Torres]

21 minutes ago, SebastienG said:

Thank you, I don't find this option

You need to add it yourself editing the constants.php file:

\define( 'OUTPUT_CACHE_METHOD', 'Redis' );

 

[SebastienG]

yes i add it, it's work find.

I understand why i've lost more than 70% of my google visitors ...

Since 15 years, i'm using url with https://domain/sujetXXX-name.html

Since 4.4.1 the url was moved on https://domain/forums/sujetXXX-name.html 200000 google scrawl per day... i'm very disappointed 

I understand why, i would like to view if i can add the pageX to my board.

If you're using rules like than /xxx the rule is updated in /forums/xxx automatically 

[ExiledVip3r]

On 3/18/2019 at 8:15 AM, CheersnGears said:

What I've noticed is the php-fpm process seems to be taking up a lot more processor than it use to.  Ram usage is still very light. 

 

On 3/19/2019 at 3:55 AM, ExiledVip3r said:

I've also been experiencing this since updating to 4.4. During peak traffic hours php-fpm now eats up all my servers CPU and the site comes to a crawl and is basically unusable. Off peak hours, the site is usable but php-fpm is still using a TON of the CPU, much more than it used to before the update.

I've been trying to optimize what I can around it to keep things running well, but have had no luck so far. I Tried disabling all customization and plugins for awhile, but saw no improvement there either. My users are getting pretty upset at this point with how poorly the site has been performing since updating to 4.4.

 

Update: After enabling php-fpm's slow log and examining it, almost every entry was related to /applications/core/sources/AdminNotification/AdminNotification.php.

Commenting out every instance of \IPS\core\AdminNotification in init.php and truncating the core_acp_notifications table (which had 67,000+ rows) instantly returned my site from unusable slow to loading instantly. CPU Usage has gone from 98% to 16%. It's only been 30 minutes or so however and it's the middle of the night. Hopefully it lasts through the rest of the day, but I'm actually hopeful for the first time since 4.4's release 😕

[mark007]

7 hours ago, ExiledVip3r said:

Update: After enabling php-fpm's slow log and examining it, almost every entry was related to /applications/core/sources/AdminNotification/AdminNotification.php.

 Commenting out every instance of \IPS\core\AdminNotification in init.php and truncating the core_acp_notifications table (which had 67,000+ rows) instantly returned my site from unusable slow to loading instantly. CPU Usage has gone from 98% to 16%. It's only been 30 minutes or so however and it's the middle of the night. Hopefully it lasts through the rest of the day, but I'm actually hopeful for the first time since 4.4's release 

But that can't be the solution, only version 4.4.(3?) which solves the problem.

[CheersnGears]

On 3/20/2019 at 10:10 AM, CheersnGears said:

The load on my server is a bit lower now and the site is a lot snappier.  Will update as the day goes on. 

 

On 3/20/2019 at 9:09 AM, bfarber said:

Please upgrade to 4.4.2 and see if this resolves your issue. We identified one or two slow queries fixed in 4.4.2 (one was specific to the "Unread content" stream, which queries against the core_search_index table, so that might explain what you're seeing).

For what it's worth, though, we're noticing loads dropping pretty significantly upon upgrading from 4.3 to 4.4 on our network.

I'm back to having heavy loads and slower response times. (it started back up again the same day I updated, just now getting around to reporting it)

10 hours ago, ExiledVip3r said:

 

 

Update: After enabling php-fpm's slow log and examining it, almost every entry was related to /applications/core/sources/AdminNotification/AdminNotification.php.

Commenting out every instance of \IPS\core\AdminNotification in init.php and truncating the core_acp_notifications table (which had 67,000+ rows) instantly returned my site from unusable slow to loading instantly. CPU Usage has gone from 98% to 16%. It's only been 30 minutes or so however and it's the middle of the night. Hopefully it lasts through the rest of the day, but I'm actually hopeful for the first time since 4.4's release 😕

Hopefully this points IPS towards a solution. 

Anyone noticing slowness on this site?  It just took a while for my last post to save. 

[bfarber]

On 3/22/2019 at 5:08 PM, SebastienG said:

Thank you, I don't find this option

And for the REDIS_ENABLED, their is option to store everything except sessions in Redis ?

 

REDIS_ENABLED is only used to store sessions and temporary topic views in redis. If you don't wish to use redis for sessions, don't enable this constant.

You can still use redis for data store, regular caching and output caching with the other constants.

[Gabriel Torres]

On 3/23/2019 at 9:10 AM, ExiledVip3r said:

Update: After enabling php-fpm's slow log and examining it, almost every entry was related to /applications/core/sources/AdminNotification/AdminNotification.php.

Great pointer. Will enable it here at my end and let you know.

@ExiledVip3r How many seconds did you configure at request_slowlog_timeout?

[ExiledVip3r]

On 3/25/2019 at 6:58 PM, Gabriel Torres said:

@ExiledVip3r How many seconds did you configure at request_slowlog_timeout?

40s

[Gabriel Torres]

@ExiledVip3r Cool. I'm seen here the Admin Notifications that you mentioned and a lot of slow processes regarding the Pages application. We are investigating this and will post our findings here, but it seems that Pages need some tweaking.

[ASTRAPI]

Yes pages are a bit slow to load compared to the previous IPB version........

[ExiledVip3r]

10 hours ago, Gabriel Torres said:

@ExiledVip3r Cool. I'm seen here the Admin Notifications that you mentioned and a lot of slow processes regarding the Pages application. We are investigating this and will post our findings here, but it seems that Pages need some tweaking.

If you're seeing the same thing in the fpm slow log that I saw, literally all I did to fix it was comment out every line where "\IPS\core\AdminNotification" occurred in the init.php file. I also emptied the acp notifications table in the database, though I don't think that was necessary. My site went back to loading fast like it did before 4.4 after that.

Fairly certain nothing critical is broken by doing this, just the new notification system in the admin panel that was added in 4.4 no longer works (unsurprisingly) after commenting those lines out.

[sadams101]

My Pages app is also painfully slow after the udpate, even on version 4.4.2.

I am trying to set up Redis. I have it installed correctly on my server, but need some help configuring things. I tried the settings below but I got 500 server errors. I guess I really don't  know much about it and have a few questions:

1) Will my site still use its mysql database with Redis? In the admin for cache settings it sounds like it will no longer use my current mysql database.

2) Is 6379 a standard port for Redis, or should I be using something different?

3) I assume my server is 127.0.0.1, but is the password unique only to Redis, or does this password need to match something else, for example mysql?

Thanks in advance for any help.

\define( 'REDIS_ENABLED', false );
\define( 'STORE_METHOD', 'Redis' );
\define( 'STORE_CONFIG', '[]' );
\define( 'CACHE_METHOD', 'Redis' );
\define( 'REDIS_CONFIG', '{"server":"127.0.0.1","port":6379,"password":""}' );
\define( 'CACHE_PAGE_TIMEOUT', 1800 );

[bfarber]

If you're experiencing a specific and reproducible slow area, I would encourage you to submit a ticket. At the least, it would be beneficial to have us double check there's not an issue with something on the page (with Pages, especially, things are generally built very dynamically and there's always a possibility some feed block or something is causing a slow database query...I can't really guess what the issue might be without seeing though).

[Gabriel Torres]

We are investigating both the php-fpm and MySQL slow queries to see if there is any optimization adjustments that we can suggest regarding the core application. I will keep you guys posted and will open a support ticket if we see that the issue is reproducible.

[b416]

14 hours ago, sadams101 said:

My Pages app is also painfully slow after the udpate, even on version 4.4.2.

I am trying to set up Redis. I have it installed correctly on my server, but need some help configuring things. I tried the settings below but I got 500 server errors. I guess I really don't  know much about it and have a few questions:

1) Will my site still use its mysql database with Redis? In the admin for cache settings it sounds like it will no longer use my current mysql database.

2) Is 6379 a standard port for Redis, or should I be using something different?

3) I assume my server is 127.0.0.1, but is the password unique only to Redis, or does this password need to match something else, for example mysql?

Thanks in advance for any help.

\define( 'REDIS_ENABLED', false );
\define( 'STORE_METHOD', 'Redis' );
\define( 'STORE_CONFIG', '[]' );
\define( 'CACHE_METHOD', 'Redis' );
\define( 'REDIS_CONFIG', '{"server":"127.0.0.1","port":6379,"password":""}' );
\define( 'CACHE_PAGE_TIMEOUT', 1800 );

Shouldn't REDIS_ENABLED be set to true ? 

The password is the one configured in your redis config (requirepass option in /etc/redis/redis.conf). You can set it to whatever you want.

 

[sadams101]

Thank you @b416 for the info...this is helpful. Does anyone know more about 

1) Will my site still use its mysql database with Redis? In the admin for cache settings it sounds like it will no longer use my current mysql database.

[b416]

Yes, the site still uses mysql to store users, forums, posts etc. There is just one table (core_output_cache) that is not used when redis is enabled and the option OUTPUT_CACHE_METHOD is set to redis..

[bfarber]

REDIS_ENABLED controls whether sessions and temporary topic view counters are stored in Redis, and nothing else. Everything else is controlled by CACHE_METHOD and OUTPUT_CACHE_METHOD.

[sadams101]

I have seen several references here to "OUTPUT_CACHE_METHOD", yet I do not see this in my contants.php when I run Redis. Where is that an option? I did select to have Redis handle by counters, which was the only option I saw when setting it up.

\define( 'REDIS_ENABLED', true );
\define( 'STORE_METHOD', 'Redis' );
\define( 'STORE_CONFIG', '[]' );
\define( 'CACHE_METHOD', 'Redis' );
\define( 'REDIS_CONFIG', '{"server":"127.0.0.1","port":6379,"password":""}' );
\define( 'CACHE_PAGE_TIMEOUT', 360 );

 

[sadams101]

I also want to point out that even with Redis running I am seeing very little improvement in site speed in my Pages app. There is clearly something wrong with the app in 4.4, but it wasn't exactly fast in 4.3 either. If you are serious about having an SEO focus, I recommend that you focus on the speed of all of your apps, not just fixing the current severe speed issue with Pages.

[bfarber]

We work on performance and "improving speed" all the time, across the board, and individually for each app. If you have a specific area that is particularly slow please feel free to let us know.

Obviously with the constants.php code you copied, you have not defined OUTPUT_CACHE_METHOD. Unless you manually edit your constants.php to add it, that constant isn't going to be present. None of the AdminCP actions set it for you (the same goes for ~50 other constants defined in init.php by the way).