paulobrien Posted October 9, 2007 Posted October 9, 2007 I've just moved my site (modaco.com) from a rather old redhat 3 machine to a shiny new Ubuntu box, and moved from Apache to Lighttpd in the process - I have the latest lighttpd, mysql and php courtesy of dotdeb (awesome!) Everything is running OK, but for some reason the php5-cgi processes are munching up all the CPU time! Here's my current "ps -aux" output...USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 5108 1972 ? Ss Sep26 0:02 /sbin/init root 2 0.0 0.0 0 0 ? S Sep26 0:00 [migration/0] root 3 0.0 0.0 0 0 ? SN Sep26 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S Sep26 0:00 [watchdog/0] root 5 0.0 0.0 0 0 ? S Sep26 0:00 [migration/1] root 6 0.0 0.0 0 0 ? SN Sep26 0:00 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S Sep26 0:00 [watchdog/1] root 8 0.0 0.0 0 0 ? S< Sep26 0:00 [events/0] root 9 0.0 0.0 0 0 ? S< Sep26 0:00 [events/1] root 10 0.0 0.0 0 0 ? S< Sep26 0:00 [khelper] root 11 0.0 0.0 0 0 ? S< Sep26 0:00 [kthread] root 34 0.0 0.0 0 0 ? S< Sep26 0:00 [kblockd/0] root 35 0.0 0.0 0 0 ? S< Sep26 0:00 [kblockd/1] root 36 0.0 0.0 0 0 ? S< Sep26 0:00 [kseriod] root 105 0.0 0.0 0 0 ? S< Sep26 0:06 [kswapd0] root 106 0.0 0.0 0 0 ? S< Sep26 0:00 [aio/0] root 107 0.0 0.0 0 0 ? S< Sep26 0:00 [aio/1] root 843 0.0 0.0 0 0 ? S Oct08 0:03 [pdflush] root 1022 0.0 0.0 0 0 ? S< Sep26 0:00 [xfslogd/0] root 1023 0.0 0.0 0 0 ? S< Sep26 0:00 [xfslogd/1] root 1024 0.0 0.0 0 0 ? S< Sep26 0:00 [xfsdatad/0] root 1025 0.0 0.0 0 0 ? S< Sep26 0:00 [xfsdatad/1] root 1065 0.0 0.0 0 0 ? S< Sep26 0:00 [ata/0] root 1066 0.0 0.0 0 0 ? S< Sep26 0:00 [ata/1] root 1067 0.0 0.0 0 0 ? S< Sep26 0:00 [ata_aux] root 1091 0.0 0.0 0 0 ? S< Sep26 0:00 [scsi_eh_0] root 1092 0.0 0.0 0 0 ? S< Sep26 0:00 [scsi_eh_1] root 1097 0.0 0.0 0 0 ? S< Sep26 0:00 [scsi_eh_2] root 1098 0.0 0.0 0 0 ? S< Sep26 0:00 [scsi_eh_3] root 1524 0.0 0.0 0 0 ? S Oct08 0:01 [pdflush] root 2064 0.0 0.0 0 0 ? S< Sep26 0:00 [ksuspend_usbd] root 2065 0.0 0.0 0 0 ? S< Sep26 0:00 [khubd] root 2280 0.0 0.0 0 0 ? S< Sep26 3:39 [kjournald] root 2437 0.0 0.0 18980 732 ? S<s Sep26 0:01 /sbin/udevd --daemon root 3301 0.0 0.0 0 0 ? S< Sep26 0:00 [kpsmoused] root 3764 0.0 0.0 3824 576 tty4 Ss+ Sep26 0:00 /sbin/getty 38400 tty4 root 3765 0.0 0.0 3828 580 tty5 Ss+ Sep26 0:00 /sbin/getty 38400 tty5 root 3767 0.0 0.0 3828 580 tty2 Ss+ Sep26 0:00 /sbin/getty 38400 tty2 root 3770 0.0 0.0 3824 576 tty3 Ss+ Sep26 0:00 /sbin/getty 38400 tty3 root 3771 0.0 0.0 3824 576 tty1 Ss+ Sep26 0:00 /sbin/getty 38400 tty1 root 3775 0.0 0.0 3828 580 tty6 Ss+ Sep26 0:00 /sbin/getty 38400 tty6 root 3801 0.0 0.0 5928 616 ? Ss Sep26 0:11 /sbin/syslogd root 3821 0.0 0.0 8080 572 ? Ss Sep26 0:00 /bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg klog 3823 0.0 0.0 5136 1828 ? Ss Sep26 0:00 /sbin/klogd -P /var/run/klogd/kmsg root 4113 0.0 0.0 46588 1072 ? Ss Sep26 0:08 /usr/sbin/sshd root 4145 0.0 0.0 12372 364 ? Ss Sep26 0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog daemon 4161 0.0 0.0 16384 432 ? Ss Sep26 0:00 /usr/sbin/atd root 4174 0.0 0.0 20880 936 ? Ss Sep26 0:00 /usr/sbin/cron root 9883 0.0 0.0 3908 584 ? S 11:01 0:00 /bin/sh /usr/bin/mysqld_safe mysql 9903 27.0 16.6 3245732 662748 ? Sl 11:01 49:42 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/mickey.modaco.com.pid --skip-external-locking -- root 9904 0.0 0.0 4976 728 ? S 11:01 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld www-data 12928 2.4 0.4 68772 16304 ? S 11:35 3:42 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf www-data 12930 0.0 0.1 225524 7628 ? Ss 11:35 0:00 /usr/bin/php5-cgi www-data 12935 0.0 0.1 225524 7628 ? Ss 11:35 0:00 /usr/bin/php5-cgi www-data 13498 20.8 1.5 246968 61288 ? S 12:26 20:33 /usr/bin/php5-cgi www-data 13855 19.4 1.5 246272 59900 ? R 12:34 17:35 /usr/bin/php5-cgi root 13873 0.0 0.0 68060 2844 ? Ss 12:37 0:00 sshd: root@pts/0 root 13876 0.0 0.0 18920 2120 pts/0 Ss 12:38 0:00 -bash www-data 13914 18.9 1.4 245304 59692 ? R 12:41 15:53 /usr/bin/php5-cgi www-data 14005 20.0 1.4 243244 57396 ? S 12:55 13:51 /usr/bin/php5-cgi www-data 14627 22.3 1.3 247160 54404 ? R 13:37 6:07 /usr/bin/php5-cgi postfix 14810 0.0 0.0 36256 1972 ? S 13:42 0:00 pickup -l -t fifo -u -c www-data 14811 19.2 1.4 246312 57728 ? R 13:43 4:09 /usr/bin/php5-cgi postfix 14845 0.0 0.0 40492 2288 ? S 13:51 0:00 smtp -t unix -u -c www-data 14892 19.2 1.2 247068 49896 ? R 13:54 2:02 /usr/bin/php5-cgi www-data 14934 23.3 1.1 243496 44532 ? S 13:59 1:18 /usr/bin/php5-cgi postfix 14945 0.0 0.0 40492 2260 ? S 14:02 0:00 smtp -t unix -u -c postfix 14949 0.0 0.0 36336 2056 ? S 14:03 0:00 cleanup -z -t unix -u -c root 14960 0.0 0.0 15008 1056 pts/0 R+ 14:05 0:00 ps -aux root 30068 0.0 0.0 34204 1964 ? Ss Oct03 0:01 /usr/lib/postfix/master postfix 30070 0.0 0.0 36472 2184 ? S Oct03 0:01 qmgr -l -t fifo -u postfix 30250 0.0 0.0 38360 2352 ? S Oct03 0:00 tlsmgr -l -t unix -u -c root 31263 0.0 0.2 61348 9600 ? Ss Oct07 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf As you can see, php5-cgi isn't happy. I'm running eAccelerator and IonCube Loader (for CommunitySEO), the machine is an Opteron 1218 with 4GB RAM and should be able to cope with the load. Does anyone have any ideas on what could be causing this high load? You can see my PHP settings at http://mickey.modaco.com/info.php, any help is very much appreciated!In fact, there could be a v small reward for tracking down this problem ;) P
paulobrien Posted October 9, 2007 Posted October 9, 2007 Bother! With communitySEO disabled...www-data 15215 0.2 0.1 225528 7632 ? Ss 14:17 0:00 /usr/bin/php5-cgi www-data 15216 5.7 0.6 237916 26392 ? R 14:17 0:00 /usr/bin/php5-cgi www-data 15217 1.5 0.4 236252 19460 ? S 14:17 0:00 /usr/bin/php5-cgi www-data 15218 8.1 0.6 235836 25208 ? S 14:17 0:00 /usr/bin/php5-cgi www-data 15219 3.8 0.5 236952 23584 ? S 14:17 0:00 /usr/bin/php5-cgi www-data 15220 0.2 0.1 225524 7628 ? Ss 14:17 0:00 /usr/bin/php5-cgi www-data 15221 5.5 0.6 238984 24560 ? R 14:17 0:00 /usr/bin/php5-cgi www-data 15222 8.1 0.7 243480 28772 ? S 14:17 0:00 /usr/bin/php5-cgi www-data 15223 8.6 0.6 237476 26088 ? R 14:17 0:00 /usr/bin/php5-cgi www-data 15224 5.9 0.6 238988 25100 ? S 14:17 0:00 /usr/bin/php5-cgi Guess i've found the culprit! :( P
paulobrien Posted October 9, 2007 Posted October 9, 2007 Ah, that could be a red herring, as commenting out communitySEO breaks a lot of stuff! /me investigates... P
bfarber Posted October 9, 2007 Posted October 9, 2007 Interesting. I know someone on the communityseo site used lighttpd, but I don't know what their experience was (just that they converted the mod_rewrite syntax for them). You should try posting there to see if anyone else has tested it and how it worked out. You could try disabling a bunch of stuff for it too, to see if you can narrow it down. You can also do an strace on processes, maybe which would help. Take the pid and run something like strace -o output.txt ##### (with ##### being the pid). It's not easy stuff to analyze, but perhaps something would stick out. Let it run for a few seconds and then hit ctrl+c to stop it.
paulobrien Posted October 9, 2007 Posted October 9, 2007 That doesn't reallly produce much useful, but I am 100% sure it's communitySEO related now. When I disable it (by giving it a bad 'registration key'), usage falls through the floor. Doh! I guess it could be something odd about ioncube, but doubt it. P
bfarber Posted October 9, 2007 Posted October 9, 2007 Hmm, that's possible - I'm not a huge fan of ioncube myself. The real problems start once ioncube/zend/caching program xyz start playing together - they all try to do the same stuff and end up fighting with one another, at least that's the best way I can describe it.
paulobrien Posted October 9, 2007 Posted October 9, 2007 I've logged a ticket with communityseo, so we'll see what happens. I installed apache, same issue! I'm stumped, I would work through the code but it's encoded, doh! I'll report back with any progress... P
bfarber Posted October 10, 2007 Posted October 10, 2007 This is a new server too right? Did everything work ok on the old server? Maybe some settings just need to be tweaked?
paulobrien Posted October 10, 2007 Posted October 10, 2007 Well, the old server was overworked in so many ways it was kinda hard to tell :D I'm having a sneaking suspicion now that because communitySEO is encoded, it's the only script that isn't being eAccelerated, and so that is bumping up the load. To be honest tho, it's all guesswork. All I can see is without communitySEO = speedy speedy, with communitySEO = server load central :D Watch this space, the communitySEO peeps have acknowledged my ticket, and are investigates. P (hates code he can't debug himself!)
bfarber Posted October 10, 2007 Posted October 10, 2007 Well, the old server was overworked in so many ways it was kinda hard to tell :D I'm having a sneaking suspicion now that because communitySEO is encoded, it's the only script that isn't being eAccelerated, and so that is bumping up the load. To be honest tho, it's all guesswork. All I can see is without communitySEO = speedy speedy, with communitySEO = server load central :D Watch this space, the communitySEO peeps have acknowledged my ticket, and are investigates. P (hates code he can't debug himself!) lol Well, I've seen the code (months ago, at least, though I doubt much has changed) and honestly the code itself was quite good (they took a different approach from the general 4000 preg_replaces you see most SEO mods running, which works much better) - however if it's something like ioncube vs eaccelerator I couldn't really speak about that, since we don't really deal with too many products being encoded. :P Converge is presently encoded - I should install that with eaccelerator and see how it behaves. Or you could if you wanted to test, but you'd have to drive enough traffic to it to make the test worthwhile, which would probably be difficult.
.om_merged Posted October 12, 2007 Posted October 12, 2007 Maybe disabling the eaccelerator "optimizer" could help? I am not too sure but try it out..I too have CSEO+eacc running on my centOS box :D
paulobrien Posted October 17, 2007 Posted October 17, 2007 Would you believe it, the problem was IPB couldn't send mail!!!! An apt-get install postfix resolved the issue, and the server is flying! Switching from eaccelerator to xcache also has given me a speed boost, weirdly. Drop by www.modaco.com and sample the speedy goodness :D Top stuff tho :D P
bfarber Posted October 17, 2007 Posted October 17, 2007 Sweet B) I love oddball problems like that. Oh, and personally I like xcache a bit better than eaccelerator myself, but that's mostly preference. I think xcache is designed by the same people that make lighttpd so if you're still using that, they'd probably play better together.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.