Huge increase in server load after upgrading to 4.4.x

[sadams101]

Thank you @bfarber for your reply, but I did just mention that in 4.4 PAGES is very slow, and I did not have this issue in 4.3. I did define Redis in Constants.php, but the default file downloaded from the ACP does not include "OUTPUT_CACHE_METHOD". It includes "CACHE_METHOD," so I assume this is what you mean here, although there is no "OUTPUT_" at the beginning as you list it. It looks like I am running Redis correctly.

I think if you are serious about speed you would fix the render blocking CSS and JS issues. These are not very hard to fix. I noticed that your forums are now blocked from performing any speed tests to demonstrate this, but here is one that I know is on the latest IPB:

https://developers.google.com/speed/pagespeed/insights/?utm_source=analytics&url=https%3A%2F%2Fwww.christforums.com

notice that mine does not have the same issues in my forum app, as I've run some custom stuff to fix it:

https://developers.google.com/speed/pagespeed/insights/?utm_source=analytics&url=https%3A%2F%2Fwww.celiac.com%2Fforums%2Fforum%2F12-post-diagnosis-recovery-treatment-of-celiac-disease%2F

 

[bfarber]

10 minutes ago, sadams101 said:

Thank you @bfarber for your reply, but I did just mention that in 4.4 PAGES is very slow, and I did not have this issue in 4.3.

Have you submitted a ticket (as I suggested earlier)? We are not noticing any inherent problems and overall performance has improved dramatically in 4.4 compared to 4.3, so if you are experiencing something different I can only recommend submitting a ticket. You can post about it here instead if you prefer, but it'll be unlikely to get the issue resolved long term as we'll need to access your community to diagnose any issues. 🙂

 

10 minutes ago, sadams101 said:

I did define Redis in Constants.php, but the default file downloaded from the ACP does not include "OUTPUT_CACHE_METHOD". It includes "CACHE_METHOD," so I assume this is what you mean here, although there is no "OUTPUT_" at the beginning as you list it. It looks like I am running Redis correctly.

No, it is not what I meant.

				// Data storage / caching settings
				// To change any of the below, go to [AdminCP > System > Advanced Configuration >
				// Data storage] and follow the instructions - it will give you a new constants.php
				// file to use without messing with anything else you've changed.
				'STORE_METHOD' 				=> $storeMethod,			// Data storage method (Database, Redis or FileSystem)
				'STORE_CONFIG'				=> $storeConfig,			// JSON-encoded settings specific to the Data storage method
				'CACHE_METHOD' 				=> $cacheMethod,			// Caching Method (Redis, Memcache, etc. "None" to disable)
				'CACHE_CONFIG' 				=> $cacheConfig,			// JSON-encoded settings specific to the caching storage method
				'CACHE_PAGE_TIMEOUT' 			=> $guestTimeout,		// Guest page caching timeout in seconds
				'REDIS_ENABLED'				=> $redisEnabled,		// Use Redis for sessions and to topic view counters in addition to normal caching?
				'REDIS_CONFIG' 				=> $redisConfig,			// JSON-encoded settings specific to Redis
				// These ones can't be changed in the AdminCP, but it's the same idea - allows Redis to be used for caching *except* guest page caching:
				'OUTPUT_CACHE_METHOD'			=> $outputCache, 		// Caching Method for guest page caching (Redis or Database)
				'OUTPUT_CACHE_METHOD_CONFIG'	=> $outputCacheConfig	,	// JSON-encoded settings specific to the guest page caching method

(We've improved comments for the constants in 4.4.3, which is why the formatting here won't match your init.php)

STORE_METHOD controls where data-storage items are placed. These are oft-requested mini-caches and this data can optionally be stored in Redis. You can enable this in the AdminCP.

CACHE_METHOD controls where true cache items are placed. If you enable Redis for your data-storage, it is automatically enabled for the caching method as well.

REDIS_ENABLED controls whether sessions and temporary topic views are stored in Redis (instead of handled normally). This can be enabled in the AdminCP.

AND FINALLY, there is a new constant OUTPUT_CACHE_METHOD which controls where guest page caches are stored. By default this is the database, but it can also be set to Redis (just note that this can consume several GB of space depending upon how long your guest page caching CACHE_PAGE_TIMEOUT value is set for and how busy your site is, and so if you use Redis you should ensure you have enough memory available for all of your cached guest pages). This cannot be changed via the AdminCP, but you can manually edit constants.php to adjust this option.

10 minutes ago, sadams101 said:

I think if you are serious about speed you would fix the render blocking CSS and JS issues. These are not very hard to fix. I noticed that your forums are now blocked from performing any speed tests to demonstrate this, but here is one that I know is on the latest IPB:

Actually, that's not really totally accurate - it is actually very hard to fix this. And while we absolutely could "fix" this out of the box for the default theme...that would do you absolutely no good when you then switch over to a custom theme unless the author of that custom theme fixes the issue for their specific theme too.

Javascript isn't an issue here. You can already defer all javascript to the end of page load (i.e. load all JS just inside the </body> tag at the end of the page), so I assume you're just talking about render blocking CSS. The problem ultimately is that which CSS is needed to render the "top fold" is very dependent upon the specific theme you are using.

That said...just because there is a bullet point here that an automated testing tool says can be improved does not at all mean performance improvements haven't been delivered with 4.4. We've done many many things to improve performance in 4.4, including (but not limited to):

• Namespacing certain functions to perform better in the underlying PHP engine
• Added dozens of database indexes
• Restructured queries to perform better, and in some cases split slow queries into separate queries to perform better
• Adjusted background tasks so they can run for longer in some cases (and thus complete quicker overall)
• Implemented lazy-loading
• Changed "letter" profile photos to inline SVG images to avoid network/bandwidth transfer
• Essentially removed calls to call_user_func and call_user_func_array which perform slowly
• Adjusted how our IP address geolookups that call back to our servers function to group requests (i.e. to make one request instead of one request per IP address) in some areas
• Implemented support for brotli compression (if the server supports it)
• Vastly improved guest page caching, allowing pages to be truly cached in the browser and supporting the if-modified-since header for guest pages
• And much more..

And this isn't to say we will never address the "remove render blocking CSS" bullet point mentioned, it's just that the risk vs reward, and time it will take to do properly, has caused it not to end up on the roadmap just yet. That could change in a future release of course.

 

Hopefully this shows how much effort has been placed into performance with 4.4, and that we've taken a global approach to performance improvement and have not simply focused on better scores in one or two specific test suites.

[ExiledVip3r]

I don't doubt a lot of things have been optimized and improved. That doesn't 100% mean performance is going to be better for every scenario though, especially when you release it at the same time as other changes such as new features and other unrelated changes. It's not uncommon for a seemingly useful bugfix to break or slow down other unforeseen things. In my case, the admin notification system added by 4.4 brings my site to a complete crawl for all users, removing much of it from the source fixes my slowdown problems 100%. 

 

---

 

Sidenote on the Redis cache stuff though... The output cache setting in admincp specifically says this:

Quote

If you have a caching method configured that will be used, otherwise the cache is stored in the database.

This is very misleading, as it still uses the database regardless of what cache method you have configured from the admincp. Unless you add OUTPUT_CACHE_METHOD to the constants, something that is mentioned absolutely nowhere and I only know about from reading this thread. I get though that it probably specifically means the output cache method and not the overall cache method, but this should really be made more clear. The constant should really be included in the generated constants.php or given an option to pick from in the cache config. My output cache table when using the db btw, is constantly fragmented while set to cache output for as long as 30 minutes. 

[sadams101]

@bfarber thanks for the clarification. It would definitely be nice to add the checkbox for this to the cache settings in the Admin CP so that this line:

\define( 'OUTPUT_CACHE_METHOD', 'Redis' );

would be automatically added to the constants.php, as I can see myself overwriting this change and forgetting about it.

Since most people probably use the default skin it would, as you said, not be a big deal to correct the defer CSS issue. Basically you just extract only the critical CSS and run it before the /head, then move the rest of the CSS down to the bottom of Global template before the IncludeJS line. Anyone interested contact @Adlago as he's the speedmaster for IPB.

[bfarber]

9 hours ago, sadams101 said:

@bfarber thanks for the clarification. It would definitely be nice to add the checkbox for this to the cache settings in the Admin CP so that this line:

\define( 'OUTPUT_CACHE_METHOD', 'Redis' );

would be automatically added to the constants.php, as I can see myself overwriting this change and forgetting about it.

Since most people probably use the default skin it would, as you said, not be a big deal to correct the defer CSS issue. Basically you just extract only the critical CSS and run it before the /head, then move the rest of the CSS down to the bottom of Global template before the IncludeJS line. Anyone interested contact @Adlago as he's the speedmaster for IPB.

You shouldn't override the constant - if you rebuild the constants.php by adjusting settings in the AdminCP, any that are not being changed will still be included.

And, while it would certainly make our jobs easier - the reality is most clients do not use the default theme. 😉 

[ASTRAPI]

Hello

I would like to say that IPB stuff put a huge effort to make latest IPB easy to use and i would like to remind you that other scripts are a lot harder to configure.

For performance also they did a great job by adding many good stuff as the above @bfarber already mention and i don't think is fear to say that they didn't do a good job.

It is true that it seems a performance issue exists for pages as i can see it also and in the previous version of IPB was ok and i think we must focus on that last fix and all will be good ....

What i see is that on first visit of a topic (PAGES) it needs around 3 sec. The repeated visit on the same topic is instant, just a few ms.

I had the same issue before in 4.3 when i was using as cache the database or Redis. When i was selecting the Filesystem it was faster.

It seems that same scenario exists on 4.4... It looks like there is something that delays the execution of a Query or a difficulty to use/write to Redis.... Don't know. Don't have the time at the moment to troubleshoot that 😞

Let's try to provide more specific feedback and create tickets and in general try to solve it than complaining ....

Thank you 🙂

[sadams101]

2 hours ago, ASTRAPI said:

Let's try to provide more specific feedback and create tickets and in general try to solve it than complaining ....

Someone complaining about the speed issue with Pages is how I found this thread--you agree that this is an issue that causes a ~3 second delay in Pages.

My complaining about IPB not deferring the CSS, which also causes an ~2-3 second delay, will hopefully lead to IPB doing something about it, although the reply here from @bfarber was not very promising (there are other ways to deal with it than what I mentioned).

At the very least my bringing it up here was on the topic of site speed, and what I've posted might help others--if IPB doesn't solve the issue it is important to know that there are solutions out there.

So, everyone, please keep complaining about speed issues so that IPB will take them more seriously! And yes, IPB gets plenty of support tickets from me on this topic! 😉

[sadams101]

BTW, what type of times do some of you use to cache guest output? For me memory should not be an issue, but my concern is that, from a social site perspective, the time of the replies and posts might seem long if I were to have an hour delay. My current setting is 6 minutes.

[Adlago]

Delay when using Pages is not due to the APP Pages  but to the use of multiple images. Also, using div class - only desktop, or phone, or tablet does not improve loading - because images are not displayed on the screen (by the rules on div class), but they are downloaded by browsers and provide enough rendering delay.

I think IPS needs to introduce a device detection and with that to determine a type to download - mobile or desktop.
Now this does not happen and the browser downloads everything - only the media rules determines what to display ...

[ASTRAPI]

@sadams101

Quote

you agree that this is an issue that causes a ~3 second delay in Pages.

The page needs around a sec to load so i can see a delay of about 2 secs. It doesn't matter i think if it is 2 or 3 or 4 as it exists and it should be fixed....

Quote

My complaining about IPB not deferring the CSS, which also causes an ~2-3 second delay

I think that a delay of 2-3 secs for that is too big but ok i agree that your proposal seems to be ok but as we don't know all the technical delays of IPB we must wait from the coders there opinion in general about your proposal and if that can be done and when.....

Quote

So, everyone, please keep complaining about speed issues so that IPB will take them more seriously!

I don't agree on that way of adding or fixing something. Posting something here as an example or prove of better performance is the best way and i am sure that IPB coders will be happy to add. Why to not agree on something that will make there platform better?

@Adlago

I agree.

[David..]

Wanted to chime in on this topic and say that I am also experiencing a high server load on 4.4.2 coming from 4.3.6.

The CPU & RAM jumps up and down causing the site to load really slowly, but never down.

[ASTRAPI]

Coming to this topic to inform everyone that there is no more performance issues for me...

I just optimize from scratch the server and now everything is loaded under 1 sec as expected !

It seems that the performance issues may be not 100% related to IPB code or something.... but we may just need to reconfigure and optimize our servers a bit better 🙂

But that’s apply to my case and people that still have performance issues related to IPB they should keep posting ....

[AtariAge]

37 minutes ago, ASTRAPI said:

Coming to this topic to inform everyone that there is no more performance issues for me...

I just optimize from scratch the server and now everything is loaded under 1 sec as expected !

It seems that the performance issues may be not 100% related to IPB code or something.... but we may just need to reconfigure and optimize our servers a bit better 🙂

But that’s apply to my case and people that still have performance issues related to IPB they should keep posting ....

In what ways did you optimize the server? 

[jair101]

12 hours ago, AtariAge said:

In what ways did you optimize the server? 

He obviously won't share details as his post is not created to be of any value to readers, its just an advertising for his services. 

[Adlago]

13 hours ago, ASTRAPI said:

I just optimize from scratch the server and now everything is loaded under 1 sec as expected !

I guess it's not hard for you to make an illustration with a few photos from different test servers.

More specifically, for me is a curious time loading notification.mp3 and whether you have overcome a 400-600 millisecond delay on this resource.

thanks

[RevengeFNF]

2 hours ago, jair101 said:

He obviously won't share details as his post is not created to be of any value to readers, its just an advertising for his services. 

Yes, this is not a case of optimization. 

I just applied @ExiledVip3r fix to a server that was having load above 40, and it went to below 2, which is the normal.

After checking php slow logs i also noticed that \IPS\core\AdminNotification was showing many times, so it was easy to pinpoint what was the cause. 

So, to anyone that can replicate the same issue, just edit init.php and search for the lines with \IPS\core\AdminNotification and comment all of them. 

This is just a workaround fix and we need Ips team to fully fix it. 

[ASTRAPI]

I am really sorry that my posts has no value for the users and generate doubts....

[bfarber]

We are actively researching the AdminNotification concern that has been raised, as a ticket was submitted with details that were were able to investigate based upon. I anticipate we will have some adjustments for this concern coming in 4.4.3, however the ticket is still open and under review at this time.

[Gabriel Torres]

Hi guys,

We are facing another issue that refers to our MySQL server.

Our MySQL server has 64 GiB of RAM. This used to be enough with IPS 4.3.x. Now, with 4.4.x, once a day (more or less), MySQL is requiring more than that, and disk swapping starts happening. With this, load on the MySQL server goes through the roof (20+ when normal is below 1), to the point that our website becomes unusable (pages won't load).

When this happens, I simply kill the MySQL process, restart MySQL and everything goes back to normal. Until it happens again.

One hour ago this happened, and right now, our MySQL has only 34 GiB of RAM used. So, from the best of my knowledge, we may be facing with a specific query or set of queries that are trigging this behavior (making MySQL to use all available RAM plus swap).

We analysed the slow logs and saw that most slow queries were related to the search engine.

At this point, we are investigating to see if this is a core issue or a plugin/app issue, so we can proceed to fix this.

Will keep you guys posted as soon as we have more information on this.

Cheers.

[Morgin]

1 hour ago, Gabriel Torres said:

Hi guys,

We are facing another issue that refers to our MySQL server.

Our MySQL server has 64 GiB of RAM. This used to be enough with IPS 4.3.x. Now, with 4.4.x, once a day (more or less), MySQL is requiring more than that, and disk swapping starts happening. With this, load on the MySQL server goes through the roof (20+ when normal is below 1), to the point that our website becomes unusable (pages won't load).

When this happens, I simply kill the MySQL process, restart MySQL and everything goes back to normal. Until it happens again.

One hour ago this happened, and right now, our MySQL has only 34 GiB of RAM used. So, from the best of my knowledge, we may be facing with a specific query or set of queries that are trigging this behavior (making MySQL to use all available RAM plus swap).

We analysed the slow logs and saw that most slow queries were related to the search engine.

At this point, we are investigating to see if this is a core issue or a plugin/app issue, so we can proceed to fix this.

Will keep you guys posted as soon as we have more information on this.

Cheers.

Out of curiosity, what’s your average online users and forum stats?

[bfarber]

I can't fathom what kind of database query could even be ran that would consume GB of memory. You would definitely need to use the tools in MySQL (such as logging slow queries or queries not using indexes) to gather more information if you believe this is an issue with the software rather than the server (perhaps a bug in the version of MySQL currently running for instance).

[Gabriel Torres]

20 hours ago, Morgin said:

Out of curiosity, what’s your average online users and forum stats?

Check at https://www.clubedohardware.com.br/forums

@bfarber, yep, we are running some diagnostics here and will post the results once we have an answer. @newbie LAC is helping me with this process.

[sound]

On 4/16/2019 at 4:01 AM, Gabriel Torres said:

 

 

 

2

On 4/16/2019 at 4:01 AM, Gabriel Torres said:

Hi guys,

We are facing another issue that refers to our MySQL server.

Our MySQL server has 64 GiB of RAM. This used to be enough with IPS 4.3.x. Now, with 4.4.x, once a day (more or less), MySQL is requiring more than that, and disk swapping starts happening. With this, load on the MySQL server goes through the roof (20+ when normal is below 1), to the point that our website becomes unusable (pages won't load).

When this happens, I simply kill the MySQL process, restart MySQL and everything goes back to normal. Until it happens again.

One hour ago this happened, and right now, our MySQL has only 34 GiB of RAM used. So, from the best of my knowledge, we may be facing with a specific query or set of queries that are trigging this behavior (making MySQL to use all available RAM plus swap).

We analysed the slow logs and saw that most slow queries were related to the search engine.

At this point, we are investigating to see if this is a core issue or a plugin/app issue, so we can proceed to fix this.

Will keep you guys posted as soon as we have more information on this.

Cheers.

On 4/16/2019 at 5:31 AM, Morgin said:

Out of curiosity, what’s your average online users and forum stats?

 

just seen the amount of posts on your forum

I had a once a day database slow query due to the  way the ip pruning task works

rather than updating where it was needed, it was actually  updating every actual post  each time it run

reported it but switched it off so dont know if was ever fixed

 

 

[Mercury Forever]

Also I am experiencing slowness that takes the site to timeout, I have identified some queries in the mysql that delay in responding.

MySQL

| 44464 | xxxxxx | localhost | xxxxxx | Prepare | 511  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::filterForProfile:240*/ SELECT COUNT(*) FROM `ibf_core_search_inde |

| 44481 | xxxxxx | localhost | xxxxxx | Prepare | 509  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::filterForProfile:240*/ SELECT COUNT(*) FROM `ibf_core_search_inde |

| 44494 | xxxxxx | localhost | xxxxxx | Prepare | 504  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::search:516*/ SELECT main.* FROM `ibf_core_search_index` AS `main` |

| 44505 | xxxxxx | localhost | xxxxxx | Prepare | 502  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::search:350*/ SELECT main.* FROM `ibf_core_search_index` AS `main` |

| 44531 | xxxxxx | localhost | xxxxxx | Prepare | 491  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::search:350*/ SELECT main.* FROM `ibf_core_search_index` AS `main` |

| 44533 | xxxxxx | localhost | xxxxxx | Prepare | 491  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::search:350*/ SELECT main.* FROM `ibf_core_search_index` AS `main` |

| 44642 | xxxxxx | localhost | xxxxxx | Prepare | 463  | Waiting for table metadata lock | /*IPS\Content\Search\Mysql\_Query::search:350*/ SELECT main.* FROM `ibf_core_search_index` AS `main` |

 

The configuration of my server is 32 GB, 24 CPU cores, 4 solid state disks (two RAID 1, one for fórum and other for mysql), it is a dedicated server.

[Gabriel Torres]

@Fred Krugger The queries here that are causing the load issue that made me open this topic are also related to the IPS search engine. However, in our case here, it seems that these queries are being launched by apps and plugins and not by the core software itself. We are still investigating this further, but we narrowed it down to two plugins that seem to be the root of the issues we are seeing here. I will post detailed information once we are 100% sure of the cause and the fixes we did here, but meanwhile I suggest you to debug these queries to see if they are being lauched by plugins/apps or by the core software.