Gabriel Torres Posted March 15, 2019 Share Posted March 15, 2019 Hi all, Load on my webserver skyrocketed after upgrading to 4.4.x. With 3.4.x load was always in the 13 range, and after the upgrade it jumped to 40. From my experience, this is either spiders/crawlers reindexing the whole site or spammers running bots to try to create new fake accounts. We've seen this in the past, and this huge spike was gone after a week or so. But it has been about 10 days now since we upgraded. What makes me more suspicious of this is that when we enable the "Under Attack Mode" at Cloudflare (we use them as our CDN), the load drops to the "normal" level of 10 that we used to see with the 3.4.x series. So it is either what I described or a DDoS attack. Update: Moving the security level from "Under Attack" to "High" also solves the issue. The increase in server load was seen under the "Medium" security level: https://support.cloudflare.com/hc/en-us/articles/200170056-What-does-Cloudflare-s-Security-Level-mean- If more seasoned admins could help me to pin-point the origin of this anomaly I'd be very thankful. Cheers, Gabriel. --- Running: Debian Linux with PHP 7.2.x, nginx, opcache, and memcached enabled Two servers, a webserver and a MySQL server. Linked with a private Gigabit Ethernet connection. There is a bunch of RAM available, no disk swapping is happening. Link to comment Share on other sites More sharing options...
CheersnGears Posted March 18, 2019 Share Posted March 18, 2019 My performance has been poor since upgrading to 4.4 (from 4.3.6). My server load is running at more than double what it usually did, but traffic is not up. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted March 18, 2019 Author Share Posted March 18, 2019 @CheersnGears Humm... it seems that you are having the same experience as me. I will keep investigating this further and will keep this topic updated. FYI, moving security to "high" at Cloudflare solved my issue here, but I still want to understand what is happening. Link to comment Share on other sites More sharing options...
CheersnGears Posted March 18, 2019 Share Posted March 18, 2019 39 minutes ago, Gabriel Torres said: @CheersnGears Humm... it seems that you are having the same experience as me. I will keep investigating this further and will keep this topic updated. FYI, moving security to "high" at Cloudflare solved my issue here, but I still want to understand what is happening. What I've noticed is the php-fpm process seems to be taking up a lot more processor than it use to. Ram usage is still very light. Link to comment Share on other sites More sharing options...
ExiledVip3r Posted March 19, 2019 Share Posted March 19, 2019 19 hours ago, CheersnGears said: What I've noticed is the php-fpm process seems to be taking up a lot more processor than it use to. Ram usage is still very light. I've also been experiencing this since updating to 4.4. During peak traffic hours php-fpm now eats up all my servers CPU and the site comes to a crawl and is basically unusable. Off peak hours, the site is usable but php-fpm is still using a TON of the CPU, much more than it used to before the update. I've been trying to optimize what I can around it to keep things running well, but have had no luck so far. I Tried disabling all customization and plugins for awhile, but saw no improvement there either. My users are getting pretty upset at this point with how poorly the site has been performing since updating to 4.4. Link to comment Share on other sites More sharing options...
Robert Angle Posted March 19, 2019 Share Posted March 19, 2019 I had the same or similar issue. I had just switched to a new SSD VPS server the same day, and after making sure everything was working fine, I upgraded from 4.3.6 to 4.4.1 but the following afternoon my memory and cpu was maxed out, and the site was slowed down to a crawl for everyone. The new server was using suphp, and so I had it changed to use the lsapi handler instead, which is considerably more efficient. This appears to have made a difference in server load. Link to comment Share on other sites More sharing options...
CheersnGears Posted March 19, 2019 Share Posted March 19, 2019 4 hours ago, ExiledVip3r said: I've also been experiencing this since updating to 4.4. During peak traffic hours php-fpm now eats up all my servers CPU and the site comes to a crawl and is basically unusable. Off peak hours, the site is usable but php-fpm is still using a TON of the CPU, much more than it used to before the update. I've been trying to optimize what I can around it to keep things running well, but have had no luck so far. I Tried disabling all customization and plugins for awhile, but saw no improvement there either. My users are getting pretty upset at this point with how poorly the site has been performing since updating to 4.4. I've been fiddling with PHP settings... even going back to 7.1 just to check... and I'm not getting any changes. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted March 20, 2019 Author Share Posted March 20, 2019 12 hours ago, CheersnGears said: I've been fiddling with PHP settings... even going back to 7.1 just to check... and I'm not getting any changes. I'm running 7.2.x here and thinking of upgrading to 7.3.x to see if it helps. One thing I noticed is that our memcached has 62.5% of wasted memory and 0% free memory, and 33.5% used. I never understood this "wasted" memory. If anyone here could tell me a little bit more about this and whether or not this needs optimization, I'd appreciate it. Another thing that was happening here was MySQL server load skyrocketing out of the blue, making the web server load go through the roof, to the point that the only solution was to kill the MySQL process and restart it (we use Percona). I re-adjusted the innodb_buffer_pool_size parameter and so far that seems to have fixed the issue. It's been 24 hours without this specific issue, but I will keep monitoring and let you guys know if it happens again. Cheers. Link to comment Share on other sites More sharing options...
ASTRAPI Posted March 20, 2019 Share Posted March 20, 2019 Most of the times that kind of issues (php cpu or ram high usage) are related to an overload of the database and the delay to respond back to php, maybe by just a simple bad qeuery ... But is true as i notice also that issue after the upgrade from latest 4.3 to latest 4.4.... The only change that i did it was for database caching from filesstem to Redis. No option to use filesystem when Redis was selected as a system cache.... In theory that's correct and should work like that as Redis uses ram and is much faster. In practice there is a delay to open topics for example only for the first time and then is instant .. Don't know if the problem is related to that cache change or to a not very well optimized query to the database.... But it is very obvious that is related to the specific 4.4 upgrade ... Don't have the time to look deeper for this... Thank you Link to comment Share on other sites More sharing options...
Lucas James Posted March 20, 2019 Share Posted March 20, 2019 AFAIK, the IPS 4.4 has no server lag or load issue, in fact 4.4 is much more optimized than 4.3. I'm managing a number of cPanel servers and every other is having IPS 4.3 or 4.4 self-hosted variant, while few of them even have 50+ websites hosted on a single server besides IPS; yet haven't felt any performance degradation of the server(s) because of 4.4 upgrade. For instance, one of my client's server has 52 sites hosted on a single server (50 are Wordpress and 2 are IPS forums) - all busy sites and are database driven, and each of them has IPS forums with 4 GB+ databases; whereas the server performance is still excellent. In a nutshell, if your server has some good specs and is setup/installed well (either by yourself, or by a SysAdmin or by the host), and the modules and apps therein are optimized properly - there should not be any reason for the server overload just because of 4.4 upgrade. Link to comment Share on other sites More sharing options...
ASTRAPI Posted March 20, 2019 Share Posted March 20, 2019 My report above is for a dedicated server with more than 32 cores qnd 196GB ram with only ips on it. And the comparison is straight forward... Before and after upgrade... @Lucas James Are they very busy? I have around 1500 users online at the same time.... At low traffic forums i didn't notice anything weird also... Link to comment Share on other sites More sharing options...
Lucas James Posted March 20, 2019 Share Posted March 20, 2019 @ASTRAPI Cloudflare powered 52 sites, including 2 IPS forums - yes, very busy sites, all database driven on OVH dedicated cPanel server (SSD + 32 GB RAM + 8c/16t) without nginx. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted March 20, 2019 Author Share Posted March 20, 2019 Hi guys, I agree that can be something related to database overload. We have a very busy community here, usually 2,000+ people online. It happened again today: MySQL server load went through the roof. This time, though, I was able to wait a bit and log in into MySQL and get the SHOW PROCESSLIST info, and it seems the overload is being caused by several processes doing a SELECT * from the core_search_index table, and this table here has over 700,000 rows. I will open a ticket with the support department to see whether this is normal or not, and will get back to you. I won't post the data publicly because of security concerns. Cheers. Link to comment Share on other sites More sharing options...
bfarber Posted March 20, 2019 Share Posted March 20, 2019 Please upgrade to 4.4.2 and see if this resolves your issue. We identified one or two slow queries fixed in 4.4.2 (one was specific to the "Unread content" stream, which queries against the core_search_index table, so that might explain what you're seeing). For what it's worth, though, we're noticing loads dropping pretty significantly upon upgrading from 4.3 to 4.4 on our network. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted March 20, 2019 Author Share Posted March 20, 2019 31 minutes ago, bfarber said: Please upgrade to 4.4.2 and see if this resolves your issue. We identified one or two slow queries fixed in 4.4.2 (one was specific to the "Unread content" stream, which queries against the core_search_index table, so that might explain what you're seeing). 😍 I literally opened a ticket regarding these specific queries... wow. Thanks for the prompt reply. Will do and will keep you guys posted. Link to comment Share on other sites More sharing options...
CheersnGears Posted March 20, 2019 Share Posted March 20, 2019 The load on my server is a bit lower now and the site is a lot snappier. Will update as the day goes on. Link to comment Share on other sites More sharing options...
AtariAge Posted March 20, 2019 Share Posted March 20, 2019 1 minute ago, CheersnGears said: The load on my server is a bit lower now and the site is a lot snappier. Will update as the day goes on. With 4.4.2? Link to comment Share on other sites More sharing options...
CheersnGears Posted March 20, 2019 Share Posted March 20, 2019 Just now, AtariAge said: With 4.4.2? Yes Link to comment Share on other sites More sharing options...
AtariAge Posted March 20, 2019 Share Posted March 20, 2019 Excellent. 🙂 Link to comment Share on other sites More sharing options...
ASTRAPI Posted March 20, 2019 Share Posted March 20, 2019 Yes 4.4.2 seems to be perfect for me also 🙂 Good job IPB !!!! Link to comment Share on other sites More sharing options...
Silnei L Andrade Posted March 20, 2019 Share Posted March 20, 2019 @Gabriel Torres What made a difference in performance for us was switching memchached to Redis. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted March 21, 2019 Author Share Posted March 21, 2019 @Silnei L Andrade Great tip. Will look into that and let you know! Link to comment Share on other sites More sharing options...
ASTRAPI Posted March 21, 2019 Share Posted March 21, 2019 Yes Redis is performing better on latest IPB 4.4.x Link to comment Share on other sites More sharing options...
SebastienG Posted March 21, 2019 Share Posted March 21, 2019 I will try the 4.4.2 version, but in my website, i have some problem with catching visitor content The table size always increase it size, and i have some downtime for disk out of space. The last size... 290Go. I have truncate it The data are not compressed, the space are not removed, ... It's not possible to store this data on a file datastore ? Link to comment Share on other sites More sharing options...
bfarber Posted March 21, 2019 Share Posted March 21, 2019 47 minutes ago, SebastienG said: I will try the 4.4.2 version, but in my website, i have some problem with catching visitor content The table size always increase it size, and i have some downtime for disk out of space. The last size... 290Go. I have truncate it The data are not compressed, the space are not removed, ... It's not possible to store this data on a file datastore ? Are you referring to caching guest pages? If so, you can store in redis, a remote database or the local (normal community) database, but there is no (built in) option to store on the filesystem. That said, the local database ultimately just pushes the data to the file system behind the scenes hence the out of disk space concern, so it wouldn't be any different. 290GB seems quite extreme, however, if you are talking about guest page caches. How long are you caching guest pages for? I believe the default is only 30 seconds. Are your tasks running correctly to keep various areas pruned as expected? Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.