High load after switching to 4.7.7

[SJ77]

It's been a long time since I can post here and avoid the dreaded "We can't help you because you need to upgrade to the newest supported version"

Now that I am officially on 4.7.7 I suppose I will give this a shot.

This graphic explains it all ...
There are no php errors or system errors coming up. Site is nearly useless.

I do see some slow queries that look like this

IPS\Content\Search\Mysql\_Index::removeClassFromSearchIndex:273*/ DELETE FROM `core_search_index_tags` WHERE index_id IN( SELECT index_id FROM `core_search_index` WHERE index_class=? AND index_author=? 

Any ideas?

Thank you 🙂

[Randy Calvert]

Out of curiosity, what does the load look like when third party resources and/or custom widgets/blocks are all disabled?  I’m wondering if there are any that are not playing nice in PHP8. 

[SJ77]

22 minutes ago, Randy Calvert said:

Out of curiosity, what does the load look like when third party resources and/or custom widgets/blocks are all disabled?  I’m wondering if there are any that are not playing nice in PHP8. 

It's not really a fair comparison. The reason people use our site goes away without 3rd party resources. (it's basically like having a car without wheels)

With that said, I tried it and the load did go down, but still higher than on 4.4

All of my 3rd party resources were inspected (and or improved) by a really incredible developer prior to upgrade.
At this time, as aforementioned I don't see any errors in the logs.

The main consumer of cpu is mysql so I started checking slow query logs and monitoring process list
The main things are all long running default IPS stuff that aren't anything to do with 3rd party plugins. (such as the example I gave above that was running for over 10 seconds)

I also followed directions on setting up cloudflare for guest caching. I think it might have helped a little but wasn't a golden ticket.

Thank you for your response 🙂

Edited February 26, 2023 by SJ77

[Randy Calvert]

9 minutes ago, SJ77 said:

It's not really a fair comparison. The reason people use our site goes away without 3rd party resources. (it's basically like having a car without wheels)

But it does help you know if certain areas need further investigation.  Remember...  with the move to the latest version also involved an upgrade of PHP from 7.x to 8.x.  That was a HUGE change and while certain things may not be broken (meaning your site is offline), it could take more resources under PHP8.  

My suggestion was around verifying any third party resources/code is actually optimized under PHP8.  I would also look to make sure have looked at optimizing your Apache/PHP setup itself... for example looking at max client settings, changing the PHP mode (or optimizing PHP-FPM settings which could be very impactful).  

Edited February 26, 2023 by Randy Calvert

[Adlago]

@SJ77

I have some guesses, but if you want, send me your site link so I can do some tests.
You can try this:
open includeMeta template,
at the bottom stop preloading fonts,
or this:

<!--
<link rel="preload" href="{url="applications/core/interface/font/fontawesome-webfont.woff2?v=4.7.0" base="none" noprotocol="true"}" as="font" crossorigin="anonymous">
-->

 

[AlexWebsites]

6 hours ago, SJ77 said:

The main consumer of cpu is mysql so I started checking slow query logs and monitoring process list
The main things are all long running default IPS stuff that aren't anything to do with 3rd party plugins. (such as the example I gave above that was running for over 10 seconds)

Just out of curiosity, are you runing mail bouncer? If so, try disabling it.

If you have other apps/plugins, start disabling one by one to see if anything is causing the hang up.

[SJ77]

3 hours ago, AlexWebsites said:

Just out of curiosity, are you runing mail bouncer? If so, try disabling it.

If you have other apps/plugins, start disabling one by one to see if anything is causing the hang up.

HI, yes I am using this but.. Why specifically did you call out mail bouncer? Have other been having high load with this one?

Edited February 27, 2023 by SJ77

[Marc Stridgen]

I would actually recommend disabling all 3rd party as a first instance, and test without. Until you do this, you are just going to be guessing at what may be the cause. By doing this, at least you can either rule it out, or rule it in. If you then find it is, you can start enabling things one at a time till the issue returns.

One thing I would advise on checking first of all, is that none of the table types changed. Ensure they are all INNODB on your server.

[AlexWebsites]

6 hours ago, SJ77 said:

HI, yes I am using this but.. Why specifically did you call out mail bouncer? Have other been having high load with this one?

That was my issue. Disabled and my phantom msql load gone. Not sure if it’s a conflict or what and have not had time to look deeper. I also enabled slow query logs and nothing really came up. It was also preventing some of my newsletters from going out. I thought it was that app at first. 

Just saying, shut it off and check as well as other apps.

Edited February 27, 2023 by AlexWebsites

[SJ77]

9 hours ago, Marc Stridgen said:

I would actually recommend disabling all 3rd party as a first instance, and test without. Until you do this, you are just going to be guessing at what may be the cause. By doing this, at least you can either rule it out, or rule it in. If you then find it is, you can start enabling things one at a time till the issue returns.

One thing I would advise on checking first of all, is that none of the table types changed. Ensure they are all INNODB on your server.

All tables are INNODB

with all plugins, theme and apps turned off, site load is still 20 to 25, Same server that ran 4.4 with a load of 8 all day long.
Turn on plugins and apps and load just to 25 - 30 range, 5 points doesn't seem like a lot for those things.

[Randy Calvert]

You would have had to move from php7 to php8 to be upgraded. I would highly look at optimizing your PHP as it’s likely not designed for high performance with an out of the box setup. If you’re running php-fpm, it has its own config that needs optimizing. 

That was a lesson I learned the hard way more than a year back. 

[Adlago]

@SJ77

Just a hypothesis - you are updating from 4.4 to 4.7.7
I'm not sure from which version service worker running was introduced, but I think it was after 4.5.
Now in many tests - your landing page lags over 12 seconds because of this service. Your other pages load 10x faster..
It is possible that this new service does not cover your landing page, so this rather long wait for your landing page causes a high level of load on your server.
Try it - shut down your landing page for half an hour and monitor server load...
You might need a landing page edit because most of your users are guests, and that's a lot of high-latency requests...

[SJ77]

After days of experiments and hours of head scratching, I can only conclude that 4.7.7 is more resource intensive than 4.4.9

I suppose I wrongly assumed that it would be more efficient and not less. However, in retrospect, I guess since it is more complex, and doing many more things, it makes sense.

Before upgrading I believe it's important to make sure one has sufficient additional resources available. I was not prepared to go from average load of 8 to average of 25.. shocking!!!  Now my site is really struggling.  There really isn't anything broken per se, thus my next option is to perform emergency migration to a more powerful server.

Again, thank you all for your input. I greatly appreciate it. 🙂

Edited February 28, 2023 by SJ77

[AlexJ]

8 hours ago, SJ77 said:

and doing many more things, it makes sense.

Sorry it doesn't make sense to me and may be because of my rookie experience. Did IPS team confirm that new version is supposed to cause more CPU consumption then previous versions? if not, then something might wrong. 

I am just curious to understand what's causing higher load for your site. 

[SJ77]

3 hours ago, AlexJ said:

Sorry it doesn't make sense to me and may be because of my rookie experience. Did IPS team confirm that new version is supposed to cause more CPU consumption then previous versions? if not, then something might wrong. 

I am just curious to understand what's causing higher load for your site. 

No IPS did not confirm anything. It’s all my conjecture. As far as what specially is causing it, I wish knew.  perhaps my use case is not compatible. 

[marklcfc]

1 hour ago, SJ77 said:

No IPS did not confirm anything. It’s all my conjecture. As far as what specially is causing it, I wish knew.  perhaps my use case is not compatible. 

If you had guest page cache on it really did reduce the load, before that was removed my CPU never went above 50% usage, but after it's gone right upto 100% during busier periods at least 5 times.

I knew it would cause problems and complained about it at the time it was announced it was being removed, likewise with the idiotic move of removing username logins.

Edited March 1, 2023 by marklcfc

[Randy Calvert]

You’re making a lot of speculation with that there. 

The OP has numerous changes in upgrading from a version of IPB that is 4 years old:

- Upgrading multiple versions of PHP.  (The new version is most likely not optimized for the current environment.)

- Most likely running an older version of MySQL (again also most likely not optimized given the age of the server environment).

- Also has third party plugins that had to be upgraded and while they may still technically work today are not optimized for performance  under PHP8 which absolutely have an impact while under load. 

- Was using a CDN to cache until discovering their cache settings were not applied correctly and had to back that change out. 

There is most likely a lot of things that could be done to optimize performance, but my guess is the OP is not a system administrator that is used to monitoring a system and understanding what that output means to optimize their various system configurations. 

For example, when I moved from PHP7 to PHP8… my system load dramatically increased to the point that it would crash a few times a day.  It turns out my PHP-FPM was the culprit and when I optimized it for my server… including the settings around process management, I actually got my system load LOWER than before I moved to PHP8.  

A LOT has changed in 4 years outside of that single change you reference both in terms of technology and to simply chalk it up to one issue that you disagree with is not fair.  Especially without knowing the configuration before and after.  

[SJ77]

  

6 hours ago, Randy Calvert said:

The OP has numerous changes in upgrading from a version of IPB that is 4 years old:

4.4.9 is just slightly over 3 years old.

  

6 hours ago, Randy Calvert said:

- Upgrading multiple versions of PHP.  (The new version is most likely not optimized for the current environment.)

Previous was 7.2 now 8.1

  

6 hours ago, Randy Calvert said:

- Most likely running an older version of MySQL (again also most likely not optimized given the age of the server environment).

Nope was running a version of Maria DB that is only a year old. Still using the same Maria DB

  

6 hours ago, Randy Calvert said:

- Also has third party plugins that had to be upgraded and while they may still technically work today are not optimized for performance  under PHP8 which absolutely have an impact while under load.

Spent $$$$ (a lot) working with top IPB developers to optimize every single bit of code for both PHP 8 and IPS 4.7.7 (one by one), It very expensive and took months, I am confident I am working with the best people in this regard. 

  

6 hours ago, Randy Calvert said:

- Was using a CDN to cache until discovering their cache settings were not applied correctly and had to back that change out.

I am 100% confident that I applied the settings correctly.  I did preciously what everyone else is doing. The problem is that users who don't know how to clear their browser cache end up being locked out of the site. I believe others don't notice because only .001% complain. Since I have millions of visitors, I get hit with lots of feedback.

  

6 hours ago, Randy Calvert said:

but my guess is the OP is not a system administrator that is used to monitoring a system and understanding what that output means to optimize their various system configurations.

I have been a system administrator and developer since the early 2000's. But I also work with other very skilled system administrators and developers. You should know I am not a hobbyist. We are operating a registered corporation and have been in business many years.

  

6 hours ago, Randy Calvert said:

A LOT has changed in 4 years outside of that single change you reference both in terms of technology and to simply chalk it up to one issue that you disagree with is not fair.  Especially without knowing the configuration before and after.  

I agree, there is no point passing blame around. My thread here is not intended for that. The goal is for solutions only. I want you to know that you have been very generous and kind with your assistance to both me and many others around this community. I really appreciate that Randy. Thank you for being so amazing.

While I am upset that the upgrade has caused (and is still causing me) a lot more stress than ideally I would have wanted, I have nothing but gratitude for the software, the entire IPB team and this community. I feel very blessed to have a business that runs on IPS software

[SJ77]

To be honest, I kind of wish this thread wasn't here with debate starting etc. I basically just wanted a support ticket with IPS. I thought this is how it worked now days.

[Driven 2 Services]

Since you upgraded PHP as part of this, the first thing I would look at is to see if the PHP opcache extension is installed, enabled and has some memory set aside for it to use.   The kind of performance difference is exactly the type I see between turning opcache on and off.