Lighttpd + Fastcgi = 100% Cpu?

[paulobrien]

I've just moved my site (modaco.com) from a rather old redhat 3 machine to a shiny new Ubuntu box, and moved from Apache to Lighttpd in the process - I have the latest lighttpd, mysql and php courtesy of dotdeb (awesome!)

Everything is running OK, but for some reason the php5-cgi processes are munching up all the CPU time!

Here's my current "ps -aux" output...

USER	   PID %CPU %MEM	VSZ   RSS TTY	  STAT START   TIME COMMAND

root		 1  0.0  0.0   5108  1972 ?		Ss   Sep26   0:02 /sbin/init

root		 2  0.0  0.0	  0	 0 ?		S	Sep26   0:00 [migration/0]

root		 3  0.0  0.0	  0	 0 ?		SN   Sep26   0:00 [ksoftirqd/0]

root		 4  0.0  0.0	  0	 0 ?		S	Sep26   0:00 [watchdog/0]

root		 5  0.0  0.0	  0	 0 ?		S	Sep26   0:00 [migration/1]

root		 6  0.0  0.0	  0	 0 ?		SN   Sep26   0:00 [ksoftirqd/1]

root		 7  0.0  0.0	  0	 0 ?		S	Sep26   0:00 [watchdog/1]

root		 8  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [events/0]

root		 9  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [events/1]

root		10  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [khelper]

root		11  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [kthread]

root		34  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [kblockd/0]

root		35  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [kblockd/1]

root		36  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [kseriod]

root	   105  0.0  0.0	  0	 0 ?		S<   Sep26   0:06 [kswapd0]

root	   106  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [aio/0]

root	   107  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [aio/1]

root	   843  0.0  0.0	  0	 0 ?		S	Oct08   0:03 [pdflush]

root	  1022  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [xfslogd/0]

root	  1023  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [xfslogd/1]

root	  1024  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [xfsdatad/0]

root	  1025  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [xfsdatad/1]

root	  1065  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [ata/0]

root	  1066  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [ata/1]

root	  1067  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [ata_aux]

root	  1091  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [scsi_eh_0]

root	  1092  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [scsi_eh_1]

root	  1097  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [scsi_eh_2]

root	  1098  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [scsi_eh_3]

root	  1524  0.0  0.0	  0	 0 ?		S	Oct08   0:01 [pdflush]

root	  2064  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [ksuspend_usbd]

root	  2065  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [khubd]

root	  2280  0.0  0.0	  0	 0 ?		S<   Sep26   3:39 [kjournald]

root	  2437  0.0  0.0  18980   732 ?		S<s  Sep26   0:01 /sbin/udevd --daemon

root	  3301  0.0  0.0	  0	 0 ?		S<   Sep26   0:00 [kpsmoused]

root	  3764  0.0  0.0   3824   576 tty4	 Ss+  Sep26   0:00 /sbin/getty 38400 tty4

root	  3765  0.0  0.0   3828   580 tty5	 Ss+  Sep26   0:00 /sbin/getty 38400 tty5

root	  3767  0.0  0.0   3828   580 tty2	 Ss+  Sep26   0:00 /sbin/getty 38400 tty2

root	  3770  0.0  0.0   3824   576 tty3	 Ss+  Sep26   0:00 /sbin/getty 38400 tty3

root	  3771  0.0  0.0   3824   576 tty1	 Ss+  Sep26   0:00 /sbin/getty 38400 tty1

root	  3775  0.0  0.0   3828   580 tty6	 Ss+  Sep26   0:00 /sbin/getty 38400 tty6

root	  3801  0.0  0.0   5928   616 ?		Ss   Sep26   0:11 /sbin/syslogd

root	  3821  0.0  0.0   8080   572 ?		Ss   Sep26   0:00 /bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg

klog	  3823  0.0  0.0   5136  1828 ?		Ss   Sep26   0:00 /sbin/klogd -P /var/run/klogd/kmsg

root	  4113  0.0  0.0  46588  1072 ?		Ss   Sep26   0:08 /usr/sbin/sshd

root	  4145  0.0  0.0  12372   364 ?		Ss   Sep26   0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog

daemon	4161  0.0  0.0  16384   432 ?		Ss   Sep26   0:00 /usr/sbin/atd

root	  4174  0.0  0.0  20880   936 ?		Ss   Sep26   0:00 /usr/sbin/cron

root	  9883  0.0  0.0   3908   584 ?		S	11:01   0:00 /bin/sh /usr/bin/mysqld_safe

mysql	 9903 27.0 16.6 3245732 662748 ?	  Sl   11:01  49:42 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/mickey.modaco.com.pid --skip-external-locking --

root	  9904  0.0  0.0   4976   728 ?		S	11:01   0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld

www-data 12928  2.4  0.4  68772 16304 ?		S	11:35   3:42 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

www-data 12930  0.0  0.1 225524  7628 ?		Ss   11:35   0:00 /usr/bin/php5-cgi

www-data 12935  0.0  0.1 225524  7628 ?		Ss   11:35   0:00 /usr/bin/php5-cgi

www-data 13498 20.8  1.5 246968 61288 ?		S	12:26  20:33 /usr/bin/php5-cgi

www-data 13855 19.4  1.5 246272 59900 ?		R	12:34  17:35 /usr/bin/php5-cgi

root	 13873  0.0  0.0  68060  2844 ?		Ss   12:37   0:00 sshd: root@pts/0 

root	 13876  0.0  0.0  18920  2120 pts/0	Ss   12:38   0:00 -bash

www-data 13914 18.9  1.4 245304 59692 ?		R	12:41  15:53 /usr/bin/php5-cgi

www-data 14005 20.0  1.4 243244 57396 ?		S	12:55  13:51 /usr/bin/php5-cgi

www-data 14627 22.3  1.3 247160 54404 ?		R	13:37   6:07 /usr/bin/php5-cgi

postfix  14810  0.0  0.0  36256  1972 ?		S	13:42   0:00 pickup -l -t fifo -u -c

www-data 14811 19.2  1.4 246312 57728 ?		R	13:43   4:09 /usr/bin/php5-cgi

postfix  14845  0.0  0.0  40492  2288 ?		S	13:51   0:00 smtp -t unix -u -c

www-data 14892 19.2  1.2 247068 49896 ?		R	13:54   2:02 /usr/bin/php5-cgi

www-data 14934 23.3  1.1 243496 44532 ?		S	13:59   1:18 /usr/bin/php5-cgi

postfix  14945  0.0  0.0  40492  2260 ?		S	14:02   0:00 smtp -t unix -u -c

postfix  14949  0.0  0.0  36336  2056 ?		S	14:03   0:00 cleanup -z -t unix -u -c

root	 14960  0.0  0.0  15008  1056 pts/0	R+   14:05   0:00 ps -aux

root	 30068  0.0  0.0  34204  1964 ?		Ss   Oct03   0:01 /usr/lib/postfix/master

postfix  30070  0.0  0.0  36472  2184 ?		S	Oct03   0:01 qmgr -l -t fifo -u

postfix  30250  0.0  0.0  38360  2352 ?		S	Oct03   0:00 tlsmgr -l -t unix -u -c

root	 31263  0.0  0.2  61348  9600 ?		Ss   Oct07   0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf

As you can see, php5-cgi isn't happy.

I'm running eAccelerator and IonCube Loader (for CommunitySEO), the machine is an Opteron 1218 with 4GB RAM and should be able to cope with the load.

Does anyone have any ideas on what could be causing this high load? You can see my PHP settings at http://mickey.modaco.com/info.php, any help is very much appreciated!

In fact, there could be a v small reward for tracking down this problem ;)

P

[paulobrien]

Bother!

With communitySEO disabled...

www-data 15215  0.2  0.1 225528  7632 ?		Ss   14:17   0:00 /usr/bin/php5-cgi

www-data 15216  5.7  0.6 237916 26392 ?		R	14:17   0:00 /usr/bin/php5-cgi

www-data 15217  1.5  0.4 236252 19460 ?		S	14:17   0:00 /usr/bin/php5-cgi

www-data 15218  8.1  0.6 235836 25208 ?		S	14:17   0:00 /usr/bin/php5-cgi

www-data 15219  3.8  0.5 236952 23584 ?		S	14:17   0:00 /usr/bin/php5-cgi

www-data 15220  0.2  0.1 225524  7628 ?		Ss   14:17   0:00 /usr/bin/php5-cgi

www-data 15221  5.5  0.6 238984 24560 ?		R	14:17   0:00 /usr/bin/php5-cgi

www-data 15222  8.1  0.7 243480 28772 ?		S	14:17   0:00 /usr/bin/php5-cgi

www-data 15223  8.6  0.6 237476 26088 ?		R	14:17   0:00 /usr/bin/php5-cgi

www-data 15224  5.9  0.6 238988 25100 ?		S	14:17   0:00 /usr/bin/php5-cgi

Guess i've found the culprit!

:(

P

[paulobrien]

Ah, that could be a red herring, as commenting out communitySEO breaks a lot of stuff!

/me investigates...

P

[bfarber]

Interesting. I know someone on the communityseo site used lighttpd, but I don't know what their experience was (just that they converted the mod_rewrite syntax for them). You should try posting there to see if anyone else has tested it and how it worked out.

You could try disabling a bunch of stuff for it too, to see if you can narrow it down.

You can also do an strace on processes, maybe which would help. Take the pid and run something like

strace -o output.txt #####

(with ##### being the pid). It's not easy stuff to analyze, but perhaps something would stick out. Let it run for a few seconds and then hit ctrl+c to stop it.

[paulobrien]

That doesn't reallly produce much useful, but I am 100% sure it's communitySEO related now. When I disable it (by giving it a bad 'registration key'), usage falls through the floor.

Doh!

I guess it could be something odd about ioncube, but doubt it.

P

[bfarber]

Hmm, that's possible - I'm not a huge fan of ioncube myself. The real problems start once ioncube/zend/caching program xyz start playing together - they all try to do the same stuff and end up fighting with one another, at least that's the best way I can describe it.

[paulobrien]

I've logged a ticket with communityseo, so we'll see what happens.

I installed apache, same issue!

I'm stumped, I would work through the code but it's encoded, doh!

I'll report back with any progress...

P

[bfarber]

This is a new server too right? Did everything work ok on the old server? Maybe some settings just need to be tweaked?

[paulobrien]

Well, the old server was overworked in so many ways it was kinda hard to tell :D

I'm having a sneaking suspicion now that because communitySEO is encoded, it's the only script that isn't being eAccelerated, and so that is bumping up the load.

To be honest tho, it's all guesswork. All I can see is without communitySEO = speedy speedy, with communitySEO = server load central :D

Watch this space, the communitySEO peeps have acknowledged my ticket, and are investigates.

P (hates code he can't debug himself!)

[bfarber]

Well, the old server was overworked in so many ways it was kinda hard to tell :D

I'm having a sneaking suspicion now that because communitySEO is encoded, it's the only script that isn't being eAccelerated, and so that is bumping up the load.

To be honest tho, it's all guesswork. All I can see is without communitySEO = speedy speedy, with communitySEO = server load central :D

Watch this space, the communitySEO peeps have acknowledged my ticket, and are investigates.

P (hates code he can't debug himself!)

lol

Well, I've seen the code (months ago, at least, though I doubt much has changed) and honestly the code itself was quite good (they took a different approach from the general 4000 preg_replaces you see most SEO mods running, which works much better) - however if it's something like ioncube vs eaccelerator I couldn't really speak about that, since we don't really deal with too many products being encoded. :P

Converge is presently encoded - I should install that with eaccelerator and see how it behaves. Or you could if you wanted to test, but you'd have to drive enough traffic to it to make the test worthwhile, which would probably be difficult.

[.om_merged]

Maybe disabling the eaccelerator "optimizer" could help? I am not too sure but try it out..I too have CSEO+eacc running on my centOS box :D

[paulobrien]

Would you believe it, the problem was IPB couldn't send mail!!!!

An apt-get install postfix resolved the issue, and the server is flying! Switching from eaccelerator to xcache also has given me a speed boost, weirdly. Drop by www.modaco.com and sample the speedy goodness :D

Top stuff tho :D

P

[bfarber]

Sweet B)

I love oddball problems like that. Oh, and personally I like xcache a bit better than eaccelerator myself, but that's mostly preference. I think xcache is designed by the same people that make lighttpd so if you're still using that, they'd probably play better together.