High webserver load (hitting 100)

[Gabriel Torres]

Hi guys,

Here I am with server load issues again. There are times that load is above 100. Right now it is 47. Definetely not right. Please see screenshots. Usually it stays below 10. During peak times, it gets around 13-14, but not more than that.

BTW. Our forum is specific to a given country (Brazil), so traffic follows the business time in this country. What is unusual is that we are seeing this high load "afterhours". These screenshots were taken around 10:00 PM Brasilia time. Which makes everything even more intriguing, as if this was caused by normal traffic, we would see this high load during the day, and not at night.

As you can see, there is plenty of available RAM.

I've already checked IP addresses with logs and netstat, and there is nothing strange there, all seems legit. Followed my old topic on this subject: http://community.invisionpower.com/topic/401758-high-loads/#comment-2493418

I've already restarted nginx, php-fpm, and even rebooted the server, at no avail.

Please take a look at the screenshot and you will see that php-fpm is hitting 100% load, which is unusual for the configuration we have (see below). 

I wonder if there is a way for me to "look inside" the php-fpm processes that are hitting 100% to see what is causing this...

I've attached ngninx.conf and php.ini for your reference as well.

A little bit more about this server configuration:

Dell PowerEdge R520, 2x Xeon E5-2430 (12 cores, 24 threads), 32 GiB RAM, 2x 120 GiB SSDs in RAID0

DB server is another machine, connected to this one through a dedicated Gigabit connection. Percona, with XtraDB, same configuration as above but with 64 GiB. Load there is stable at 0.20, so it is not a DB server issue.

We use CloudFlare Business. Security is now set at "high," and we have Railgun enabled and running fine.

PHP  5.3.29, nginx 1.6.0

Zend OPcache v7.0.4 (for PHP code cache)

XCache v3.2.0 (for variable cache)

Any advice would be welcome.

 

nginx.conf.txt

php.ini.txt

[RevengeFNF]

Have you considered moving to php 5.4? Its much better in high load environments.

[Gabriel Torres]

It is on our to-do list, but I don't think the PHP version is the culprit right now.

[Ryan H.]

I've seen a case where misbehaving cache (APC) was compounding a poor nginx configuration to similar results. Disabling APC went a long way toward fixing it in that case.

Seems to me you need more operational data. Is New Relic an option? Extremely helpful performance monitoring and metrics; basic version is free. The pro version (and trial) also includes application traces, which might help you isolate any problem code.

[ASTRAPI]

Hi Gabriel Torres

It will be better if you use Memcache instead of xcache but this is not your problem.....

How many users you have online when you have this problem and how many posts your forum has?

Do you use Sphinx?

Your nginx config is wrong !

You may also have wrong values at phpfpm.conf file or my.cnf file but i have to check them.....

Also you may hit a system/kernel limit....

Do you use Centos?

[Makoto]

First guess would be maybe you don't have enough FPM pools spawned to handle your incoming traffic, but I could sit here and list countless possible causes for server load issues. It's hard to say without looking at it myself.

It's not normal by any means. I would be happy to take a look at it for you and at least try and point you in the right direction if you'd like. Send me a message if so.

[ASTRAPI]

don't have enough FPM pools spawned

I think you mean child processes or max spare servers as having more than one pool for a single forum doesn't make any sense....

[Makoto]

I think you mean child processes or max spare servers as having more than one pool for a single forum doesn't make any sense....

​Yes, I did obviously mean child processes ASTRAPI, please excuse the basic human error, multi-tasking sometimes causes people to mix up simple terminology.

As I said in my previous message please feel free to contact me if you'd like some more one on one help @Gabriel Torres, I'd be happy to at least try and point out the cause of your issues free of charge but I'm not going to play a back and forth guessing match against anyone here for a highly generic and common issue that has countless possible causes.

Best of luck regardless.

[ASTRAPI]

Sure no problem i just note that to avoid user looking for more pools.....

We are all here to help

[Gabriel Torres]

Thanks for all the input.

Here are the information you requested:

We have around 2,000 online users when the issue occurs. Out forum has about 5.8 million posts and about 460,000 users. Our current record for most online users is 3,341.

We run Debian Linux.

We use Sphinx, forgot to mention that. I will take a look at Relic, I didn't know about this tool before.

Your nginx config is wrong !

You may also have wrong values at phpfpm.conf file or my.cnf file but i have to check them.....

​Can you please elaborate on that? Saying that my config file is wrong without telling me exactly what is wrong doesn't help me at all...

There is no file called phpfpm.conf at my system. Please note that I compiled PHP from scratch using --enable-fpm; I guess with that all configurations relative to FPM mode go inside php.ini (which is already attached on my OP). But I may be wrong.

Kirito, thank you for your offer. Let me try fix this by myself, if I am not able to, then I will summon you help. Deal?

I just want to stress out that this issue only occurs from time to time. Most of the time the load is "normal".

my.cnf.txt

[ASTRAPI]

change these:

worker_processes     24;

worker_connections  10240;

 

to

worker_processes     6;

worker_connections  32768;

 

Then run mysqltuner and post the output here so we can adjust the mysql ....

Also you must locate the phpfpm config file that keeps the info for :

pm =
pm.max_children =
pm.start_servers =
pm.min_spare_servers =
pm.max_spare_servers =
pm.max_requests = 

 

 

[Makoto]

change these:

worker_processes     24;

worker_connections  10240;

 

to

worker_processes     6;

worker_connections  32768;

​His configuration is fine. He has a dual CPU server running 12 cores / 24 threads. A setting of 12 might be more optimal but it depends and honestly doesn't matter that much regardless. RTFM. Setting it to auto should also suffice.

There is no file called phpfpm.conf at my system. Please note that I compiled PHP from scratch using --enable-fpm; I guess with that all configurations relative to FPM mode go inside php.ini (which is already attached on my OP). But I may be wrong.

Kirito, thank you for your offer. Let me try fix this by myself, if I am not able to, then I will summon you help. Deal?

​No problem, I'll try and help you out a bit here anyways because I think I may be right based on your answers. Are you sure you don't have a php-fpm config file anywhere on your server? If you don't know regardless, you're probably just running PHP-FPM using default setup. You'll want to create and tune a configuration file yourself and probably specify the path to it when initializing the PHP-FPM daemon.

What you should use for your configuration depends on your specific load requirements. If you need help tuning, I again don't mind offering a bit of personal help, but that should give you something to go on regardless.

I do have some general advice for configuring PHP-FPM for IP.Board in an old thread of mine here;

[ASTRAPI]

I was read what kind of CPU he has before i recommend my values

Maybe the phpfpm config file name is php-fpm.conf

[Gabriel Torres]

First of all, thanks for all your help.

Regarding nginx.conf, I guess my configuration is fine, based on what others have suggested me in the past (they recommend configuring the number of available threads, as stated in the manual). I don't understand the logic of reducing to only six with I have 24 threads available. Furthermore, I don't think with this particular case this is the issue, as I am pretty sure the issue is specifically with PHP (if a configuration issue) or with certain users or with certain users performing a specific task at the same time.

I found the php-fpm.conf, I completely forgot where it was (/usr/local/php/etc, for my own future reference). Please see it attached. I configured this file according to what other have suggested me in the past, and some fine tunings may be required.

Thank you in advance, once again.

 

php-fpm.conf.txt

[Gabriel Torres]

PS: as for MySQL stats, I run MySQLTuner from time to time and I am always making fine tunings. I believe my current configuration is fine, but I am attaching the results just in case. Thank you.

mysqltuner.txt

[Gabriel Torres]

PS2: I will upgrade to PHP 5.6 today, I hope this will help as well.

[ASTRAPI]

Regarding nginx.conf, I guess my configuration is fine, based on what others have suggested me in the past (they recommend configuring the number of available threads, as stated in the manual). I don't understand the logic of reducing to only six with I have 24 threads available. Furthermore, I don't think with this particular case this is the issue, as I am pretty sure the issue is specifically with PHP (if a configuration issue) or with certain users or with certain users performing a specific task at the same time.

This is total wrong as the manuals recommend this value if you have only Nginx in the server then is fine to use 24 but if you have more like Mysql and php they need also some space to breath.

If nginx have huge traffic and use all cores as you set it at 24 then you know what it will happen to the rest.....

And this is one of the main reasons that users get huge loads as yours.....

For phpfpm you can try to edit your values with the below:

pm = dynamic
pm.max_children = 400
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 390
pm.max_requests = 8000

 

Try them and let me know

 

 

[Makoto]

You're using the static process manager, so you can strip these lines from your php-fpm configuration,

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 128

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 128

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 256

These are only used when using a dynamic process manager. 120 children is a very high value by most standards (it depends on how much traffic you actually need to handle, what your average and peak RPM's are and so on), but with 32GB of memory, it should be reasonably safe depending on how much free memory you actually have available.

This is total wrong as the manuals recommend this value if you have only Nginx in the server then is fine to use 24 but if you have more like Mysql and php they need also some space to breath.

If nginx have huge traffic and use all cores as you set it at 24 then you know what it will happen to the rest.....

And this is one of the main reasons that users get huge loads as yours.....

For phpfpm you can try this:

pm = dynamic
pm.max_children = 400
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 390
pm.max_requests = 8000

 

Try them and let me know

 

 

​Please do not follow his destructive advice. He's telling you to change values without fully understanding your needs or hardware capabilities and you don't need to use a dynamic process manager for a single application web server.

He is also completely wrong regarding how to configure Nginx and that is not at all relevant to the server loads you are experiencing.

Seriously, I fully discredit this guy and everything he says, I implore you to ignore his posts as he's more likely to crash your entire server than provide you any helpful advice.

[ASTRAPI]

Add also this my.cnf:

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html

[client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqld]
#default-character-set = utf8
character-set-server = utf8
init_connect='SET collation_connection = utf8_unicode_ci; SET NAMES utf8;'
#init-connect='set character_set_connection = utf8'
#init-connect='SET NAMES utf8'
#init-connect='set collation_connection = utf8_unicode_ci'
#collation_connection = utf8_unicode_ci
#collation-database = utf8_unicode_ci
collation-server = utf8_unicode_ci
skip-character-set-client-handshake

skip_name_resolve
slow_query_log
slow_query_log_file = /var/log/mysql/slow-query.log
log-error = /var/log/mysql/mysql.err
explicit_defaults_for_timestamp

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
innodb_buffer_pool_size = 16G
innodb_buffer_pool_instances = 24
innodb_thread_concurrency = 0
innodb_flush_method=O_DIRECT
#innodb_page_size=4k
max_heap_table_size=128M
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
port = 3306
# server_id = .....
socket = /tmp/mysql.sock

# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
join_buffer_size = 64K
sort_buffer_size = 64K
key_buffer_size = 128M
read_rnd_buffer_size = 256K
#table_open_cache = 256K
query_cache_type = 1
query_cache_size = 128M
query_cache_limit = 1M
#
max_connections = 500
thread_cache_size = 4096
wait_timeout = 120
interactive_timeout = 120
#sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES 

 

If you so all my recommendations Nginx phpfpm and mysql and then restart the mysql and nginx you will be fine 100%

@Gabriel Torres

As i don't want to reply and start a trouble again with kirito if you want send me a pm and i will point you to a forum with a bit higher traffic than yours that is using a similar config with the one that i recommend you so you can check the speed and how it works and then if you want you can try my recommendations.

I will never say to a user don't use another member's recommendations as every one can has his opinion even if it is ok or not

That's why i recommend to users to try both settings as it is easy with a simple restart of Nginx and Mysql

[ASTRAPI]

Seriously, I fully discredit this guy and everything he says, I implore you to ignore his posts as he's more likely to crash your entire server than provide you any helpful advice.

I can bet with you if you want 

[Makoto]

You're utterly ignorant and you're giving people bad advice that you probably just copy and pasted off a blog somewhere. You clearly don't actually understand what most of these settings and configuration directives you're telling him to change do. You don't even understand the basic principles of how a server works.

This isn't about having opinions. This is a support forum. Your "opinions" are wrong and destructive. Please stop pretending like you're an expert in a field you barely understand, or if you wish to continue doing so, please do it in a manner that does not endanger the stability and security of other peoples hardware.

I can bet with you if you want  

​Really?

[RevengeFNF]

PS2: I will upgrade to PHP 5.6 today, I hope this will help as well.

​You should also upgrade to MariaDB 10.0 as it scales a lot better than Mysql 5.6 in busy environments.

PS: You are using mysqltuner 1.2.0 when the most recent version is 1.4.0

[ASTRAPI]

I can say my opinion and i will and i don't care what you say

Users can try recommendations and compare and decide who copy manuals and don't have any idea of servers !

I will not follow you and try to say bad things about you and your recommendations as you do for me.

For me it ends here as the members are here for help and not to read our problems!

[Makoto]

Sure, you can, and I will continue to discredit any bad advice you give others.

My experience actually speaks for itself with all that I've done to contribute to this forum and this community as a whole. You're just some random guy with a rev meter as an avatar and a seriously annoying spammy signature.

Bugger off kid.

@Gabriel Torres I'm deeply sorry this crap has cluttered your thread but I think I've more than made my point. I'm not going to risk continuing this personal feud in here. My above offer still stands if you have continue experiencing issues.

[ASTRAPI]

Sure, you can, and I will continue to discredit any bad advice you give others.

That's good ! You can do this by providing your recommendation and let others try other recommendations.

You're just some guy with a rev monitor as an avatar and a seriously annoying spammy signature.

I hate also your signature

This is my last message for you .

I am also so sorry about this hijacking of your topic for our personal problems...