Lighttpd - 95% Cpu Usage. Apache - All Ram Used, Swap Comes In

[Bobsuch]

Hi,



I've hired rack911 to take care of my server, but it's just getting so frustrating, as it seems they can't fix my server/optimize it correctly.

I just want some input as to what the problem is here.


I got him to install lighttpd along with xcache.

After about15-18 hours, the server's CPU slowly increases to be using 90-90%, giving me the 500 Internal Server Error.
I've asked him to optimize xcache and lighttpd, but I keep getting these responses:

The should be pretty optimized. Xcache has limited optimization features.

Lighttpd itself is pretty optimized, the problem is php. Php runs as its own binary and theres not many parameters that can be tweaked.

Is that even correct? Or is he just getting lazy?

-------------------------

So I got him to enable apache, and disable lighttpd.

This time, it's the other way around.
My ram get's completely used up, forcing the server to use Swap; making the load shoot to 50+. CPU usage is fine the entire way.

-------------------------

This has been going back and forth, enabling/disabling lighttpd and re-enabling apache but a problem always comes up each time.


Am I asking for too much?
Or isn't he doing his job properly?

Server specs;
Intel C2D E6400
4GB Ram


This is what I'm getting when lighttpd is on.

Tasks: 117 total, 17 running, 100 sleeping, 0 stopped, 0 zombie Cpu(s): 95.0%us, 4.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.5%si, 0.0%st Mem: 4147196k total, 1346888k used, 2800308k free, 203144k buffers Swap: 2907756k total, 0k used, 2907756k free, 712584k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2401 mysql 15 0 318m 107m 3868 S 44 2.6 49:02.02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib 3279 root 15 0 27712 15m 2900 S 12 0.4 10:33.81 /usr/local/lighttpd/bin/php-cgi 3272 root 16 0 29752 16m 2908 R 11 0.4 10:15.94 /usr/local/lighttpd/bin/php-cgi 3266 root 15 0 27536 14m 2920 S 11 0.4 10:40.14 /usr/local/lighttpd/bin/php-cgi 3270 root 16 0 27624 14m 2908 R 11 0.4 10:36.71 /usr/local/lighttpd/bin/php-cgi 3267 root 16 0 27796 14m 2972 R 10 0.4 10:35.22 /usr/local/lighttpd/bin/php-cgi 3268 root 15 0 28148 15m 3112 S 10 0.4 10:49.86 /usr/local/lighttpd/bin/php-cgi 3276 root 16 0 27696 15m 2900 R 10 0.4 10:29.73 /usr/local/lighttpd/bin/php-cgi 3269 root 16 0 28312 15m 2916 R 10 0.4 10:23.57 /usr/local/lighttpd/bin/php-cgi 3274 root 16 0 27924 15m 2912 R 10 0.4 10:44.51 /usr/local/lighttpd/bin/php-cgi 3275 root 16 0 29368 16m 2928 R 9 0.4 10:41.84 /usr/local/lighttpd/bin/php-cgi 3278 root 15 0 29432 16m 2876 S 9 0.4 10:28.52 /usr/local/lighttpd/bin/php-cgi 3271 root 16 0 26784 14m 2900 R 9 0.3 10:41.61 /usr/local/lighttpd/bin/php-cgi 3277 root 16 0 27156 14m 2928 R 9 0.4 10:31.07 /usr/local/lighttpd/bin/php-cgi 3265 root 15 0 28408 15m 2880 S 8 0.4 10:46.89 /usr/local/lighttpd/bin/php-cgi 3280 root 16 0 27036 14m 2880 R 8 0.3 10:43.20 /usr/local/lighttpd/bin/php-cgi 3273 root 15 0 24676 12m 2952 R 8 0.3 10:43.57 /usr/local/lighttpd/bin/php-cgi 3262 root 15 0 13356 11m 632 R 3 0.3 3:04.27 /usr/local/lighttpd/sbin/lighttpd -f /etc/lighttpd.conf 1 root 15 0 2032 648 552 S 0 0.0 0:00.36 init [3] 2 root RT 0 0 0 0 S 0 0.0 0:00.00 [migration/0] 3 root 34 19 0 0 0 R 0 0.0 0:00.00 [ksoftirqd/0] 4 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/0] 5 root RT 0 0 0 0 S 0 0.0 0:00.00 [migration/1] 6 root 34 19 0 0 0 S 0 0.0 0:00.00 [ksoftirqd/1] 7 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/1] 8 root 10 -5 0 0 0 S 0 0.0 0:00.00 [events/0] 9 root 10 -5 0 0 0 S 0 0.0 0:00.00 [events/1] 10 root 10 -5 0 0 0 S 0 0.0 0:00.00 [khelper] 11 root 11 -5 0 0 0 S 0 0.0 0:00.00 [kthread] 15 root 10 -5 0 0 0 S 0 0.0 0:00.00 [kblockd/0] 16 root 10 -5 0 0 0 S 0 0.0 0:00.00 [kblockd/1]

top - 22:56:02 up 18:46,  1 user,  load average: 23.38, 22.84, 22.44

[FrostedPopTart]

There's no *apparent* reason; just looking at that small quote above, that your server load should be 20+.

Depending on your lighttpd / Apache setup, PHP can run as FastCGI, which is different than running as a module; so to speak. I know of several hosts that set PHP up to run as FastCGI, but I haven't read too many good reviews about it.

From the looks of what you posted above, MySQL is using 44% of your CPU resources. You may need to fine tune your my.cnf file. Would you mind posting that here for us to look at? :)

(btw there are some nice tweaks posted in this forum for that by bfarber, rct2dotcom, and others. :) )

[bfarber]

Was going to say the same thing - looks like MySQL is using up all the resources according to your top printout.

[rct2·com]

So I got him to enable apache, and disable lighttpd.

This time, it's the other way around.

My ram get's completely used up, forcing the server to use Swap; making the load shoot to 50+. CPU usage is fine the entire way.

Ok well when Apache is running, its configuration is read from httpd.conf. With lighttpd it's lighttpd.conf. So fundemental inconsistencies between those config files may explain the differences in memory/CPU profiles. For example, a setting in httpd.conf is the maximum number of Apache [httpd] instances that are allowed to run concurrently. If it's too high, bang goes all your memory, and paging starts.

Sidenote: Convention is that the swap area should be at least double, and up to 4 times your physical memory size. In your case, it is half. That's not good.

I don't agree with the others about MySQL 'greed' being the main contributing factor. I don't know lighttpd, and how it uses PHP as a separate binary, or whether your Apache has PHP compiled in, or whether it too is using the binary, or why the PHP cgi.bin needs to be running so many times. But at between 9% and 12% CPU per PHP instance that is running, in your lighttpd configuration, that's where you need to focus.

[Bobsuch]

There's no *apparent* reason; just looking at that small quote above, that your server load should be 20+.

Depending on your lighttpd / Apache setup, PHP can run as FastCGI, which is different than running as a module; so to speak. I know of several hosts that set PHP up to run as FastCGI, but I haven't read too many good reviews about it.

From the looks of what you posted above, MySQL is using 44% of your CPU resources. You may need to fine tune your my.cnf file. Would you mind posting that here for us to look at? :)

(btw there are some nice tweaks posted in this forum for that by bfarber, rct2dotcom, and others. :) )

Was going to say the same thing - looks like MySQL is using up all the resources according to your top printout.

Ok, I'll take another look at the Mysql config.

Ok well when Apache is running, its configuration is read from httpd.conf. With lighttpd it's lighttpd.conf. So fundemental inconsistencies between those config files may explain the differences in memory/CPU profiles. For example, a setting in httpd.conf is the maximum number of Apache [httpd] instances that are allowed to run concurrently. If it's too high, bang goes all your memory, and paging starts.

[i]Sidenote:[/i] Convention is that the swap area should be at least double, and up to 4 times your physical memory size. In your case, it is half. That's not good.

I don't agree with the others about MySQL 'greed' being the main contributing factor. I don't know lighttpd, and how it uses PHP as a separate binary, or whether your Apache has PHP compiled in, or whether it too is using the binary, or why the PHP cgi.bin needs to be running so many times. But at between 9% and 12% CPU per PHP instance that is running, in your lighttpd configuration, that's where you need to focus.

I did a "fresh database" restore, and the server was fine for nearly 20 hours up to a few minutes ago.

Ram usage was just over 1.3gb/4gb of ram, and the CPU usage was lower than 4%. Load was never over 0.10


So does this mean that the config files for lighttpd, mysql and xcache need fixing?

When the lighttpd processes start taking up 40-55% of the CPU, it gives me a time out error.
After about 10-15 minutes, I just get an internal 500 server error.


Heres my lighttpd.conf file:

"mod_access", "mod_fastcgi", "mod_accesslog" ) server.document-root = "/var/www/html/" server.errorlog = "/var/log/lighttpd.error.log" index-file.names = ( "index.php", "index.html", "index.htm", "default.htm" ) mimetype.assign = ( ".pdf" => "application/pdf", ".sig" => "application/pgp-signature", ".spl" => "application/futuresplash", ".class" => "application/octet-stream", ".ps" => "application/postscript", ".torrent" => "application/x-bittorrent", ".dvi" => "application/x-dvi", ".gz" => "application/x-gzip", ".pac" => "application/x-ns-proxy-autoconfig", ".swf" => "application/x-shockwave-flash", ".tar.gz" => "application/x-tgz", ".tgz" => "application/x-tgz", ".tar" => "application/x-tar", ".zip" => "application/zip", ".mp3" => "audio/mpeg", ".m3u" => "audio/x-mpegurl", ".wma" => "audio/x-ms-wma", ".wax" => "audio/x-ms-wax", ".ogg" => "application/ogg", ".wav" => "audio/x-wav", ".gif" => "image/gif", ".jpg" => "image/jpeg", ".jpeg" => "image/jpeg", ".png" => "image/png", ".xbm" => "image/x-xbitmap", ".xpm" => "image/x-xpixmap", ".xwd" => "image/x-xwindowdump", ".css" => "text/css", ".html" => "text/html", ".htm" => "text/html", ".js" => "text/javascript", ".asc" => "text/plain", ".c" => "text/plain", ".cpp" => "text/plain", ".log" => "text/plain", ".conf" => "text/plain", ".text" => "text/plain", ".txt" => "text/plain", ".dtd" => "text/xml", ".xml" => "text/xml", ".mpeg" => "video/mpeg", ".mpg" => "video/mpeg", ".mov" => "video/quicktime", ".qt" => "video/quicktime", ".avi" => "video/x-msvideo", ".asf" => "video/x-ms-asf", ".asx" => "video/x-ms-asf", ".wmv" => "video/x-ms-wmv", ".bz2" => "application/x-bzip", ".tbz" => "application/x-bzip-compressed-tar", ".tar.bz2" => "application/x-bzip-compressed-tar", # default mime type "" => "application/octet-stream", ) #accesslog.filename = "/var/log/lighttpd.access.log" url.access-deny = ( "~", ".inc" ) server.port = 80 #server.bind = "127.0.0.1" server.username = "nobody" server.groupname = "nobody" fastcgi.server = ( ".php" => ( "localhost" => ( "socket" => "/tmp/php-fastcgi.socket" + var.PID, "bin-path" => "/usr/local/lighttpd/bin/php-cgi", "max-procs" => 1, "bin-environment" => ( "PHP_FCGI_CHILDREN" => "3", "PHP_FCGI_MAX_REQUESTS" => "10000" ) ) ) )

server.modules              = (

Heres the last few lines of my lighttpd.error.log file:

2007-10-11 04:13:58: (server.c.1312) [note] sockets enabled again 2007-10-11 04:13:58: (server.c.1360) [note] sockets disabled, out-of-fds 2007-10-11 04:13:59: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:13:59: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:00: (server.c.1253) NOTE: a request for /index.php timed out after writing 12312 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:02: (server.c.1253) NOTE: a request for /index.php timed out after writing 18980 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:02: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:02: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:03: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:03: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:03: (server.c.1312) [note] sockets enabled again 2007-10-11 04:14:03: (server.c.1360) [note] sockets disabled, out-of-fds 2007-10-11 04:14:04: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:05: (server.c.1253) NOTE: a request for /index.php timed out after writing 12420 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2007-10-11 04:14:05: (server.c.1253) NOTE: a request for /index.php timed out after writing 13140 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle

Some lines near the first half of the lighttpd.error.log:

2007-10-03 12:29:54: (mod_fastcgi.c.3496) all handlers for /index.php on .php are down. 2007-10-03 12:29:57: (mod_fastcgi.c.2633) fcgi-server re-enabled: 0 /tmp/php-fastcgi.socket 2007-10-03 12:30:05: (mod_fastcgi.c.2855) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 145 2007-10-03 12:30:05: (mod_fastcgi.c.3496) all handlers for /index.php on .php are down. 2007-10-03 12:30:08: (mod_fastcgi.c.2633) fcgi-server re-enabled: 0 /tmp/php-fastcgi.socket 2007-10-03 12:30:14: (mod_fastcgi.c.2855) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 146 2007-10-03 12:30:14: (mod_fastcgi.c.3496) all handlers for /index.php on .php are down. 2007-10-03 12:30:17: (mod_fastcgi.c.2633) fcgi-server re-enabled: 0 /tmp/php-fastcgi.socket 2007-10-03 12:30:23: (mod_fastcgi.c.2855) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 146 2007-10-03 12:30:23: (mod_fastcgi.c.3496) all handlers for /index.php on .php are down. 2007-10-03 12:30:26: (mod_fastcgi.c.2633) fcgi-server re-enabled: 0 /tmp/php-fastcgi.socket 2007-10-03 12:30:30: (mod_fastcgi.c.2855) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 143 2007-10-03 12:30:30: (mod_fastcgi.c.3496) all handlers for /index.php on .php are down. 2007-10-03 12:30:33: (mod_fastcgi.c.2633) fcgi-server re-enabled: 0 /tmp/php-fastcgi.socket 2007-10-03 12:30:38: (mod_fastcgi.c.2855) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 141 2007-10-03 12:30:38: (mod_fastcgi.c.3496) all handlers for /index.php on .php are down. 2007-10-03 12:30:41: (mod_fastcgi.c.2633) fcgi-server re-enabled: 0 /tmp/php-fastcgi.socket

2007-10-03 12:29:54: (mod_fastcgi.c.2855) backend is overloaded; we'll disable it for 2 seconds and send the request to another backend instead: reconnects: 0 load: 144 

My.cnf:

set-variable=local-infile=0 datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock max_allowed_packet=32M max_connections=1200 # Default to using old password format for compatibility with mysql 3.x # clients (those using the mysqlclient10 compatibility package). old_passwords=1 table_cache=1024 key_buffer=256M sort_buffer_size=4M join_buffer=4M read_buffer_size=4M wait_timeout = 30 connect_timeout = 10 skip-innodb myisam_sort_buffer_size=64M thread_cache=256 tmp_table_size=128M max_heap_table_size=150M low-priority-updates skip-name-resolve concurrent_insert=2 [mysql.server] user=mysql basedir=/var/lib [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

[mysqld]

[Darkside_RG_merged]

:unsure: If you have a fairly big database the key_buffer size is way to low and most likely what is pushing the webserver into thrashing

[Bobsuch]

My.cnf update:
Added/Edited:
query_cache_limit=2M
key_buffer=768M
join_buffer=2M
record_buffer=2M
query_cache_size=2M

[mysqld]

set-variable=local-infile=0

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

max_allowed_packet=32M

max_connections=1200

old_passwords=1

table_cache=1024

key_buffer=1024M

sort_buffer_size=4M

join_buffer=2M

read_buffer_size=4M

record_buffer=2M

wait_timeout = 30

connect_timeout = 10

skip-innodb

myisam_sort_buffer_size=64M

thread_cache=256

tmp_table_size=128M

max_heap_table_size=150M

low-priority-updates

skip-name-resolve

concurrent_insert=2

query_cache_limit=2M

query_cache_size=2M



[mysql.server]

user=mysql

basedir=/var/lib


[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

Lighttpd.conf update: Added server.max-write-idle = 720

server.modules			  = (

								"mod_access",

							   "mod_fastcgi",

								"mod_accesslog" )


server.max-write-idle = 720

server.document-root		= "/var/www/html/"

server.errorlog			 = "/var/log/lighttpd.error.log"


index-file.names			= ( "index.php", "index.html",

								"index.htm", "default.htm" )


mimetype.assign			 = (

  ".pdf"		  =>	  "application/pdf",

  ".sig"		  =>	  "application/pgp-signature",

  ".spl"		  =>	  "application/futuresplash",

  ".class"		=>	  "application/octet-stream",

  ".ps"		   =>	  "application/postscript",

  ".torrent"	  =>	  "application/x-bittorrent",

  ".dvi"		  =>	  "application/x-dvi",

  ".gz"		   =>	  "application/x-gzip",

  ".pac"		  =>	  "application/x-ns-proxy-autoconfig",

  ".swf"		  =>	  "application/x-shockwave-flash",

  ".tar.gz"	   =>	  "application/x-tgz",

  ".tgz"		  =>	  "application/x-tgz",

  ".tar"		  =>	  "application/x-tar",

  ".zip"		  =>	  "application/zip",

  ".mp3"		  =>	  "audio/mpeg",

  ".m3u"		  =>	  "audio/x-mpegurl",

  ".wma"		  =>	  "audio/x-ms-wma",

  ".wax"		  =>	  "audio/x-ms-wax",

  ".ogg"		  =>	  "application/ogg",

  ".wav"		  =>	  "audio/x-wav",

  ".gif"		  =>	  "image/gif",

  ".jpg"		  =>	  "image/jpeg",

  ".jpeg"		 =>	  "image/jpeg",

  ".png"		  =>	  "image/png",

  ".xbm"		  =>	  "image/x-xbitmap",

  ".xpm"		  =>	  "image/x-xpixmap",

  ".xwd"		  =>	  "image/x-xwindowdump",

  ".css"		  =>	  "text/css",

  ".html"		 =>	  "text/html",

  ".htm"		  =>	  "text/html",

  ".js"		   =>	  "text/javascript",

  ".asc"		  =>	  "text/plain",

  ".c"			=>	  "text/plain",

  ".cpp"		  =>	  "text/plain",

  ".log"		  =>	  "text/plain",

  ".conf"		 =>	  "text/plain",

  ".text"		 =>	  "text/plain",

  ".txt"		  =>	  "text/plain",

  ".dtd"		  =>	  "text/xml",

  ".xml"		  =>	  "text/xml",

  ".mpeg"		 =>	  "video/mpeg",

  ".mpg"		  =>	  "video/mpeg",

  ".mov"		  =>	  "video/quicktime",

  ".qt"		   =>	  "video/quicktime",

  ".avi"		  =>	  "video/x-msvideo",

  ".asf"		  =>	  "video/x-ms-asf",

  ".asx"		  =>	  "video/x-ms-asf",

  ".wmv"		  =>	  "video/x-ms-wmv",

  ".bz2"		  =>	  "application/x-bzip",

  ".tbz"		  =>	  "application/x-bzip-compressed-tar",

  ".tar.bz2"	  =>	  "application/x-bzip-compressed-tar",

  # default mime type

  ""			  =>	  "application/octet-stream",

 )


#accesslog.filename		  = "/var/log/lighttpd.access.log"


url.access-deny			 = ( "~", ".inc" )


server.port				= 80


#server.bind				= "127.0.0.1"


server.username			= "nobody"

server.groupname		   = "nobody"


fastcgi.server			 = ( ".php" =>

							   ( "localhost" =>

								 (

								   "socket" => "/tmp/php-fastcgi.socket" + var.PID,

								   "bin-path" => "/usr/local/lighttpd/bin/php-cgi",

				   "max-procs" => 1,

					 			"bin-environment" => ( 

							   "PHP_FCGI_CHILDREN" => "3",

							   "PHP_FCGI_MAX_REQUESTS" => "10000"

							 )

								 )

							   )

							)

[bfarber]

See how that does (it's always best to make one or two changes, test them, and then go back to tweak - otherwise you don't get a good sense for how things interact with each other if you make a ton of changes at once).

Looking at the lighttpd (which I'm not overly familiar with) config file, these two settings would seem to control the same thing rc was talking about above with apache

"PHP_FCGI_MAX_REQUESTS" => "10000"

"PHP_FCGI_CHILDREN" => "3",

I'd read up on those in the lighttpd configuration documentation to see if you have these set to good values for your server specs. Maybe even post on their forums. If you do, I'd be interested in hearing what they have to say - especially if they can give you like a generalized formula (if you have 100 simultaneous connections you should set the requests to number of cpus * 4, for example).

[rct2·com]

Only MySQL my.cnf is common to your Apache and lighttpd environments, but the symptoms of failure are completely different. In Apache you burn all the memory, in lighttpd you burn all the CPU. Two entirely different issues. Best to leave MySQL settings unchanged in my opinion while we focus on the differences between Apache and lighttpd. If we can find the 'best of both' we have a solution.

In in your lighttpd settings, over 100% CPU is being grabbed by the PHP cgi when you add all the instances together. As you have posted no Apache httpd.conf settings, I guess that lighttpd is where you want us to concentrate?

[Bobsuch]

See how that does (it's always best to make one or two changes, test them, and then go back to tweak - otherwise you don't get a good sense for how things interact with each other if you make a ton of changes at once).

Looking at the lighttpd (which I'm not overly familiar with) config file, these two settings would seem to control the same thing rc was talking about above with apache

"PHP_FCGI_CHILDREN" => "3",

"PHP_FCGI_MAX_REQUESTS" => "10000"

I'd read up on those in the lighttpd configuration documentation to see if you have these set to good values for your server specs. Maybe even post on their forums. If you do, I'd be interested in hearing what they have to say - especially if they can give you like a generalized formula (if you have 100 simultaneous connections you should set the requests to number of cpus * 4, for example).

Posted, haven't got a relevant reply as of yet, seems to only have one replier so far

Only MySQL my.cnf is common to your Apache and lighttpd environments, but the symptoms of failure are completely different. In Apache you burn all the memory, in lighttpd you burn all the CPU. Two entirely different issues. Best to leave MySQL settings unchanged in my opinion while we focus on the differences between Apache and lighttpd. If we can find the 'best of both' we have a solution.

Ok

In in your lighttpd settings, over 100% CPU is being grabbed by the PHP cgi when you add all the instances together. As you have posted no Apache httpd.conf settings, I guess that lighttpd is where you want us to concentrate?

That's correct,

I've tried to base the settings (The PHP_FCGI settings) on this posted config:
http://trac.lighttpd.net/trac/browser/tags...c/lighttpd.conf


But no matter what combos I do, the php processes are always dominating the CPU.

[Bobsuch]

Just an update:
It seems to be giving a 500 error once every 1-2 page load.


Top -c:

[root@server ~]# top -c

top - 03:24:46 up 20:28, 2 users, load average: 14.45, 14.62, 14.66

Tasks: 113 total, 12 running, 101 sleeping, 0 stopped, 0 zombie

Cpu(s): 90.9%us, 8.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.3%hi, 0.5%si, 0.0%st

Mem: 4147196k total, 1371800k used, 2775396k free, 180440k buffers

Swap: 2907756k total, 0k used, 2907756k free, 750868k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

6376 nobody 16 0 20140 7740 2768 R 26 0.2 1:16.50 /usr/local/lighttpd/bin/php-cgi

6375 nobody 16 0 23648 10m 2452 R 23 0.3 1:18.11 /usr/local/lighttpd/bin/php-cgi

6377 nobody 15 0 23336 10m 2496 R 22 0.3 1:13.92 /usr/local/lighttpd/bin/php-cgi

6370 nobody 15 0 21308 8824 2812 R 20 0.2 1:23.50 /usr/local/lighttpd/bin/php-cgi

6382 nobody 15 0 20156 7468 2476 R 20 0.2 0:58.60 /usr/local/lighttpd/bin/php-cgi

6374 nobody 16 0 23396 10m 2496 R 19 0.3 1:24.49 /usr/local/lighttpd/bin/php-cgi

6371 nobody 16 0 23576 10m 2452 R 18 0.3 1:33.45 /usr/local/lighttpd/bin/php-cgi

6391 nobody 16 0 20140 7420 2448 R 17 0.2 0:26.91 /usr/local/lighttpd/bin/php-cgi

6373 nobody 15 0 20464 7620 2452 R 16 0.2 1:28.85 /usr/local/lighttpd/bin/php-cgi

6372 nobody 15 0 20140 7424 2452 S 14 0.2 1:29.65 /usr/local/lighttpd/bin/php-cgi

6090 nobody 15 0 16712 15m 656 R 4 0.4 4:38.29 /usr/local/lighttpd/sbin/lighttpd -f /etc/lighttpd.conf

2429 mysql 15 0 1072m 250m 3940 S 2 6.2 21:10.22 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib

1 root 15 0 2036 648 552 S 0 0.0 0:00.63 init [3]

2 root RT 0 0 0 0 S 0 0.0 0:00.00 [migration/0]

3 root 34 19 0 0 0 S 0 0.0 0:00.00 [ksoftirqd/0]

4 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/0]

5 root RT 0 0 0 0 S 0 0.0 0:00.00 [migration/1]

6 root 34 19 0 0 0 S 0 0.0 0:00.00 [ksoftirqd/1]

7 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/1]

8 root 10 -5 0 0 0 S 0 0.0 0:00.00 [events/0]

9 root 10 -5 0 0 0 S 0 0.0 0:00.00 [events/1]

10 root 10 -5 0 0 0 S 0 0.0 0:00.00 [khelper]

11 root 11 -5 0 0 0 S 0 0.0 0:00.00 [kthread]

15 root 10 -5 0 0 0 S 0 0.0 0:00.00 [kblockd/0]

16 root 10 -5 0 0 0 S 0 0.0 0:00.04 [kblockd/1]

17 root 14 -5 0 0 0 S 0 0.0 0:00.00 [kacpid]

109 root 14 -5 0 0 0 S 0 0.0 0:00.00 [cqueue/0]

110 root 14 -5 0 0 0 S 0 0.0 0:00.00 [cqueue/1]

113 root 10 -5 0 0 0 S 0 0.0 0:00.00 [khubd]

115 root 10 -5 0 0 0 S 0 0.0 0:00.00 [kseriod]

181 root 15 0 0 0 0 S 0 0.0 0:00.04 [pdflush]

182 root 15 0 0 0 0 S 0 0.0 0:01.25 [pdflush]

183 root 14 -5 0 0 0 S 0 0.0 0:00.00 [kswapd0]

Lighttpd.conf:

server.modules = ( "mod_status",

"mod_access",

"mod_fastcgi",

"mod_accesslog" )

status.status-url = "/server-status"

status.statistics-url = "/server-statistics"

status.config-url = "/server-config"

server.max-write-idle = 720

server.max-fds = 4096

server.max-connections = 1395

server.network-backend = "linux-sendfile"

server.event-handler = "select"

server.document-root = "/var/www/html/"

server.errorlog = "/var/log/lighttpd.error.log"

index-file.names = ( "index.php", "index.html",

"index.htm", "default.htm" )

mimetype.assign = (

".pdf" => "application/pdf",

".sig" => "application/pgp-signature",

".spl" => "application/futuresplash",

".class" => "application/octet-stream",

".ps" => "application/postscript",

".torrent" => "application/x-bittorrent",

".dvi" => "application/x-dvi",

".gz" => "application/x-gzip",

".pac" => "application/x-ns-proxy-autoconfig",

".swf" => "application/x-shockwave-flash",

".tar.gz" => "application/x-tgz",

".tgz" => "application/x-tgz",

".tar" => "application/x-tar",

".zip" => "application/zip",

".mp3" => "audio/mpeg",

".m3u" => "audio/x-mpegurl",

".wma" => "audio/x-ms-wma",

".wax" => "audio/x-ms-wax",

".ogg" => "application/ogg",

".wav" => "audio/x-wav",

".gif" => "image/gif",

".jpg" => "image/jpeg",

".jpeg" => "image/jpeg",

".png" => "image/png",

".xbm" => "image/x-xbitmap",

".xpm" => "image/x-xpixmap",

".xwd" => "image/x-xwindowdump",

".css" => "text/css",

".html" => "text/html",

".htm" => "text/html",

".js" => "text/javascript",

".asc" => "text/plain",

".c" => "text/plain",

".cpp" => "text/plain",

".log" => "text/plain",

".conf" => "text/plain",

".text" => "text/plain",

".txt" => "text/plain",

".dtd" => "text/xml",

".xml" => "text/xml",

".mpeg" => "video/mpeg",

".mpg" => "video/mpeg",

".mov" => "video/quicktime",

".qt" => "video/quicktime",

".avi" => "video/x-msvideo",

".asf" => "video/x-ms-asf",

".asx" => "video/x-ms-asf",

".wmv" => "video/x-ms-wmv",

".bz2" => "application/x-bzip",

".tbz" => "application/x-bzip-compressed-tar",

".tar.bz2" => "application/x-bzip-compressed-tar",

# default mime type

"" => "application/octet-stream",

)

#accesslog.filename = "/var/log/lighttpd.access.log"

url.access-deny = ( "~", ".inc" )

server.port = 80

#server.bind = "127.0.0.1"

server.username = "nobody"

server.groupname = "nobody"

fastcgi.server = ( ".php" =>

( "localhost" =>

(

"socket" => "/tmp/php-fastcgi.socket" + var.PID,

"bin-path" => "/usr/local/lighttpd/bin/php-cgi",

"max-procs" => 1,

"bin-environment" => (

"PHP_FCGI_CHILDREN" => "10",

"PHP_FCGI_MAX_REQUESTS" => "2000"

)

)

)

)

2 Hours after enabling server-status and server-statistics:

Server-status:

legend

. = connect, C = close, E = hard error

r = read, R = read-POST, W = write, h = handle-request

q = request-start, Q = request-end

s = response-start, S = response-end

457 connections

rrhWrrWWWWrWWWWWrhWrWWhrWWrWWrrWhhWWWWhWhWrrWWWhWr

WWWWrWhhrWhWhWrhhrWrWhhWWWrrWWWWWWrhWrWWWWhWWWhWWh

rrrhhrrWWWrhWWrrrWWWrWWhWWWrWhhrrWrWhWWhWWhrWrWWrW

WWWWWhWWrWrhWWrWWWhhhWhWWhhWrWhrWWWhWWWWhWWrWrrWrr

hhhhrhWrWWhrrWWWWWWWWWrhWWhhWWrWWWWrWhWWhhhWWWrrrr

hWhhrWrWhhhhhWrWhhWWhhrWhWWWrWWWhWrWWrhWWWWhrWWWWW

rWWWWrWWhrWWhWWWWWWhWrWWWWWrWWrWWWWWhhWrhWrrWrhrrW

hhrhrrWEWWrrWhhWWWhhWhWrrWhWWhrhrrhWhrWWhhWrhrWrhr

rhrhWWWhrhWrrrrrWWrWrWrhhhrrhrrhrrWWhhrrhhrrhrrrrr

rrrrrrh

Server-statistics:

fastcgi.active-requests: 35

fastcgi.backend.localhost.0.connected: 223712

fastcgi.backend.localhost.0.died: 0

fastcgi.backend.localhost.0.disabled: 0

fastcgi.backend.localhost.0.load: 35

fastcgi.backend.localhost.0.overloaded: 1067

fastcgi.backend.localhost.load: 35

fastcgi.requests: 223712

[rct2·com]

AS I say, I don't know anything about lighthttpd but I see 10 PHP cgi processes in that top listing and they are all consuming huge amounts of CPU and I see "PHP_FCGI_CHILDREN" => "10" in the .conf file, so if it were me I'd be googling around to see what other people are saying about PHP_FCGI_CHILDREN

[Bobsuch]

AS I say, I don't know anything about lighthttpd but I see 10 PHP cgi processes in that top listing and they are all consuming huge amounts of CPU and I see "PHP_FCGI_CHILDREN" => "10" in the .conf file, so if it were me I'd be googling around to see what other people are saying about PHP_FCGI_CHILDREN

Yep, I've tried searching (Spent nearly 4 hours combined) and I keep getting mixed answers.

I've still got apache 2 installed, but it's turned off.

I guess I'll just use Apache seeing as I would receive more help :)


Heres the apache config file.
Would there be any missing parts that would effect performance? If so, tell me so I can post those bits too.

#

# PidFile: The file in which the server should record its process

# identification number when it starts.

#

PidFile run/httpd.pid


#

# Timeout: The number of seconds before receives and sends time out.

#

Timeout 120


#

# KeepAlive: Whether or not to allow persistent connections (more than

# one request per connection). Set to "Off" to deactivate.

#

KeepAlive Off


#

# MaxKeepAliveRequests: The maximum number of requests to allow

# during a persistent connection. Set to 0 to allow an unlimited amount.

# We recommend you leave this number high, for maximum performance.

#

MaxKeepAliveRequests 100


#

# KeepAliveTimeout: Number of seconds to wait for the next request from the

# same client on the same connection.

#

KeepAliveTimeout 15


##

## Server-Pool Size Regulation (MPM specific)

## 


# prefork MPM

# StartServers: number of server processes to start

# MinSpareServers: minimum number of server processes which are kept spare

# MaxSpareServers: maximum number of server processes which are kept spare

# ServerLimit: maximum value for MaxClients for the lifetime of the server

# MaxClients: maximum number of server processes allowed to start

# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>

StartServers	   8

MinSpareServers	5

MaxSpareServers   20

ServerLimit	  256

MaxClients	   256

MaxRequestsPerChild  4000

</IfModule>


# worker MPM

# StartServers: initial number of server processes to start

# MaxClients: maximum number of simultaneous client connections

# MinSpareThreads: minimum number of worker threads which are kept spare

# MaxSpareThreads: maximum number of worker threads which are kept spare

# ThreadsPerChild: constant number of worker threads in each server process

# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule worker.c>

StartServers		 2

MaxClients		 150

MinSpareThreads	 25

MaxSpareThreads	 75 

ThreadsPerChild	 25

MaxRequestsPerChild  0

</IfModule>



LoadModule auth_basic_module modules/mod_auth_basic.so

LoadModule auth_digest_module modules/mod_auth_digest.so

LoadModule authn_file_module modules/mod_authn_file.so

LoadModule authn_alias_module modules/mod_authn_alias.so

LoadModule authn_anon_module modules/mod_authn_anon.so

LoadModule authn_dbm_module modules/mod_authn_dbm.so

LoadModule authn_default_module modules/mod_authn_default.so

LoadModule authz_host_module modules/mod_authz_host.so

LoadModule authz_user_module modules/mod_authz_user.so

LoadModule authz_owner_module modules/mod_authz_owner.so

LoadModule authz_groupfile_module modules/mod_authz_groupfile.so

LoadModule authz_dbm_module modules/mod_authz_dbm.so

LoadModule authz_default_module modules/mod_authz_default.so

LoadModule ldap_module modules/mod_ldap.so

LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

LoadModule include_module modules/mod_include.so

LoadModule log_config_module modules/mod_log_config.so

LoadModule logio_module modules/mod_logio.so

LoadModule env_module modules/mod_env.so

LoadModule ext_filter_module modules/mod_ext_filter.so

LoadModule mime_magic_module modules/mod_mime_magic.so

LoadModule expires_module modules/mod_expires.so

LoadModule deflate_module modules/mod_deflate.so

LoadModule headers_module modules/mod_headers.so

LoadModule usertrack_module modules/mod_usertrack.so

LoadModule setenvif_module modules/mod_setenvif.so

LoadModule mime_module modules/mod_mime.so

LoadModule dav_module modules/mod_dav.so

LoadModule status_module modules/mod_status.so

LoadModule autoindex_module modules/mod_autoindex.so

LoadModule info_module modules/mod_info.so

LoadModule dav_fs_module modules/mod_dav_fs.so

LoadModule vhost_alias_module modules/mod_vhost_alias.so

LoadModule negotiation_module modules/mod_negotiation.so

LoadModule dir_module modules/mod_dir.so

LoadModule actions_module modules/mod_actions.so

LoadModule speling_module modules/mod_speling.so

LoadModule userdir_module modules/mod_userdir.so

LoadModule alias_module modules/mod_alias.so

LoadModule rewrite_module modules/mod_rewrite.so

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

LoadModule proxy_ftp_module modules/mod_proxy_ftp.so

LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule proxy_connect_module modules/mod_proxy_connect.so

LoadModule cache_module modules/mod_cache.so

LoadModule suexec_module modules/mod_suexec.so

LoadModule disk_cache_module modules/mod_disk_cache.so

LoadModule file_cache_module modules/mod_file_cache.so

LoadModule mem_cache_module modules/mod_mem_cache.so

LoadModule cgi_module modules/mod_cgi.so

[rct2·com]

In lighttpd, what happens if you change the PHP_FCGI_CHILDREN value to "1"?

In Apache are you using the prefork or Worker MPM? And what is the memory occupancy of the httpd process in Apache?

[Bobsuch]

In lighttpd, what happens if you change the PHP_FCGI_CHILDREN value to "1"?

In Apache are you using the prefork or Worker MPM? And what is the memory occupancy of the httpd process in Apache?

It seems so odd.
I had left these settings:


my.cnf

set-variable=local-infile=0 datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock max_allowed_packet=32M max_connections=1200 old_passwords=1 table_cache=1024 key_buffer=1024M sort_buffer_size=4M join_buffer=2M read_buffer_size=4M record_buffer=2M wait_timeout = 30 connect_timeout = 10 skip-innodb myisam_sort_buffer_size=64M thread_cache=256 tmp_table_size=128M max_heap_table_size=150M low-priority-updates skip-name-resolve concurrent_insert=2 query_cache_limit=2M query_cache_size=2M [mysql.server] user=mysql basedir=/var/lib [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

[mysqld]

lighttp.conf

server.modules = ( "mod_status", "mod_access", "mod_fastcgi", "mod_accesslog" ) status.status-url = "/server-status" status.statistics-url = "/server-statistics" status.config-url = "/server-config" server.max-write-idle = 720 server.max-fds = 4096 server.max-connections = 1395 server.event-handler = "linux-sysepoll" server.network-backend = "writev" server.document-root = "/var/www/html/" server.errorlog = "/var/log/lighttpd.error.log" index-file.names = ( "index.php", "index.html", "index.htm", "default.htm" ) mimetype.assign = ( ".pdf" => "application/pdf", ".sig" => "application/pgp-signature", ".spl" => "application/futuresplash", ".class" => "application/octet-stream", ".ps" => "application/postscript", ".torrent" => "application/x-bittorrent", ".dvi" => "application/x-dvi", ".gz" => "application/x-gzip", ".pac" => "application/x-ns-proxy-autoconfig", ".swf" => "application/x-shockwave-flash", ".tar.gz" => "application/x-tgz", ".tgz" => "application/x-tgz", ".tar" => "application/x-tar", ".zip" => "application/zip", ".mp3" => "audio/mpeg", ".m3u" => "audio/x-mpegurl", ".wma" => "audio/x-ms-wma", ".wax" => "audio/x-ms-wax", ".ogg" => "application/ogg", ".wav" => "audio/x-wav", ".gif" => "image/gif", ".jpg" => "image/jpeg", ".jpeg" => "image/jpeg", ".png" => "image/png", ".xbm" => "image/x-xbitmap", ".xpm" => "image/x-xpixmap", ".xwd" => "image/x-xwindowdump", ".css" => "text/css", ".html" => "text/html", ".htm" => "text/html", ".js" => "text/javascript", ".asc" => "text/plain", ".c" => "text/plain", ".cpp" => "text/plain", ".log" => "text/plain", ".conf" => "text/plain", ".text" => "text/plain", ".txt" => "text/plain", ".dtd" => "text/xml", ".xml" => "text/xml", ".mpeg" => "video/mpeg", ".mpg" => "video/mpeg", ".mov" => "video/quicktime", ".qt" => "video/quicktime", ".avi" => "video/x-msvideo", ".asf" => "video/x-ms-asf", ".asx" => "video/x-ms-asf", ".wmv" => "video/x-ms-wmv", ".bz2" => "application/x-bzip", ".tbz" => "application/x-bzip-compressed-tar", ".tar.bz2" => "application/x-bzip-compressed-tar", # default mime type "" => "application/octet-stream", ) #accesslog.filename = "/var/log/lighttpd.access.log" url.access-deny = ( "~", ".inc" ) server.port = 80 #server.bind = "127.0.0.1" server.username = "nobody" server.groupname = "nobody" fastcgi.server = ( ".php" => ( "localhost" => ( "socket" => "/tmp/php-fastcgi.socket" + var.PID, "bin-path" => "/usr/local/lighttpd/bin/php-cgi", "max-procs" => 1, "bin-environment" => ( "PHP_FCGI_CHILDREN" => "5", "PHP_FCGI_MAX_REQUESTS" => "2000" ) ) ) )

At the time after I restarted mysql and lighttpd, the lighttp related processes was still consuming 95% of the cpu.
I check back about 8 hours later and everything seems to be fine.

Question, I'm asking now is, should I downgrade to a lower spec machine?

Top -c;

Tasks: 108 total, 1 running, 107 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4147196k total, 1731188k used, 2416008k free, 353272k buffers Swap: 2907756k total, 0k used, 2907756k free, 927508k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10297 root 15 0 2164 1016 788 R 1 0.0 0:01.37 top -c 1 root 15 0 2036 648 552 S 0 0.0 0:01.48 init [3] 2 root RT 0 0 0 0 S 0 0.0 0:00.05 [migration/0] 3 root 34 19 0 0 0 S 0 0.0 0:00.00 [ksoftirqd/0] 4 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/0] 5 root RT 0 0 0 0 S 0 0.0 0:00.01 [migration/1] 6 root 34 19 0 0 0 S 0 0.0 0:00.00 [ksoftirqd/1] 7 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/1] 8 root 10 -5 0 0 0 S 0 0.0 0:00.01 [events/0] 9 root 10 -5 0 0 0 S 0 0.0 0:00.00 [events/1] 10 root 10 -5 0 0 0 S 0 0.0 0:00.00 [khelper] 11 root 11 -5 0 0 0 S 0 0.0 0:00.00 [kthread] 15 root 10 -5 0 0 0 S 0 0.0 0:00.10 [kblockd/0] 16 root 10 -5 0 0 0 S 0 0.0 0:00.19 [kblockd/1] 17 root 14 -5 0 0 0 S 0 0.0 0:00.00 [kacpid] 109 root 14 -5 0 0 0 S 0 0.0 0:00.00 [cqueue/0] 110 root 14 -5 0 0 0 S 0 0.0 0:00.00 [cqueue/1] 113 root 10 -5 0 0 0 S 0 0.0 0:00.00 [khubd] 115 root 10 -5 0 0 0 S 0 0.0 0:00.00 [kseriod] 181 root 15 0 0 0 0 S 0 0.0 0:00.10 [pdflush] 182 root 15 0 0 0 0 S 0 0.0 0:04.15 [pdflush] 183 root 14 -5 0 0 0 S 0 0.0 0:00.00 [kswapd0] 184 root 14 -5 0 0 0 S 0 0.0 0:00.00 [aio/0] 185 root 14 -5 0 0 0 S 0 0.0 0:00.00 [aio/1] 341 root 11 -5 0 0 0 S 0 0.0 0:00.00 [kpsmoused] 372 root 16 -5 0 0 0 S 0 0.0 0:00.00 [ata/0] 373 root 16 -5 0 0 0 S 0 0.0 0:00.00 [ata/1] 374 root 16 -5 0 0 0 S 0 0.0 0:00.00 [ata_aux] 378 root 10 -5 0 0 0 S 0 0.0 0:00.00 [scsi_eh_0] 379 root 10 -5 0 0 0 S 0 0.0 0:00.00 [scsi_eh_1] 380 root 18 -5 0 0 0 S 0 0.0 0:48.09 [kjournald] 407 root 12 -5 0 0 0 S 0 0.0 0:00.17 [kauditd] 441 root 21 -4 2212 600 372 S 0 0.0 0:00.07 /sbin/udevd -d

top - 22:17:59 up 4 days, 15:21,  2 users,  load average: 0.05, 0.02, 0.00

As you can see the CPU is basically idle, and the ram usage is less than half.
Average around 150-350+ within 15 minutes.

I'm currently on an Intel C2D 6400, with 4gb of ram; with this server I'm currently going over my head with the costs for this server each month.

Would a P4 3.2 Ghz HT, with 2gb of ram be OK? (It may include direct admin, and apache as I have no idea how to disable apache/install lighttpd and xcache.)

Gotta say so far, big thanks to rct2dotcom and bfarber, along with Darkside_RG and CMGuy for your help. :)

[bfarber]

I don't often find the processor to make a huge difference. I mean sure if you run a Celeron your site won't perform well, but generally any real server processor these days will be ok in my views. I'm sure some will correct me. ;)

RAM is extremely important, however. If you had the option of downgrading (given that you said cost is an issue - if it weren't, I would not recommend downgrading at all) to the server you mentioned, but adding 2 GB of RAM (to keep it at 4) you could do that ok I'd assume.

Or, if you wanted to get creative - find ways to charge money for your site to make back the costs. Once you have a server like that, you have some options. Ad campaigns. Personal space for paying members. Email accounts for a small charge. There are lots of possibilities. You could likely even offer space as a host to some of your members.