Jump to content

Erratic server load spikes


Guest MarcusInMd

Recommended Posts

Thanks guys.

We have actually been running 2.2.2 for about a month and a half now.

We also have been using eaccelerator for quite a while now.

Blair, During these crazy load times I have actually shut down the search capability of our forums much to the dismay of our members. So I know its not search related but something else.

I might be willing to give something like httplight a shot, but I will have to research it's setup. Everything I have read so far says to install it on a fresh OS installation.

Link to comment
Share on other sites

  • Replies 154
  • Created
  • Last Reply

I also like hardware, I currently run a fast Opteron dual core with 4GB of RAM and SCSI drives. I rarely see the load over 1 (too much server :P). Generally 600-700 online during the peak of the day. I've had as many as 2,000 online in the past 15 minutes, and the load hovered around 2. Users won't notice any difference until it gets over 4.



For faecess and giggles, here's the current my.cnf (mysqld) and httpd.conf settings:

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

skip-locking

skip-innodb

query_cache_limit=16M

query_cache_size=32M

query_cache_type=1

interactive_timeout=1200

wait_timeout=100

connect_timeout=20

key_buffer=800M

key_cache_division_limit=40

low_priority_updates=1

join_buffer_size=2M

record_buffer=6M

max_allowed_packet=16M

table_cache=1280

tmp_table_size=64M

sort_buffer_size=200M

key_buffer_size=200M

read_buffer_size=2M

thread_cache_size=200

max_user_connections=500

max_connections=500

max_connect_errors=10

thread_concurrency=4

I strongly recommend the tuning-primer.sh script to optimise your SQL settings.

httpd.conf...


Timeout 300


KeepAlive On


MaxKeepAliveRequests 100


KeepAliveTimeout 15


MinSpareServers 5

MaxSpareServers 10


StartServers 5


MaxClients 150


MaxRequestsPerChild 0



BTW, I'm running CentOS 64-bit. I'm NOT running a PHP cache program yet, as I haven't found a 64-bit version I'm confident in.

Link to comment
Share on other sites

Thanks guys.



We have actually been running 2.2.2 for about a month and a half now.



We also have been using eaccelerator for quite a while now.



Blair, During these crazy load times I have actually shut down the search capability of our forums much to the dismay of our members. So I know its not search related but something else.



I might be willing to give something like httplight a shot, but I will have to research it's setup. Everything I have read so far says to install it on a fresh OS installation.


Why a fresh OS setup? I've never read something stating that
Link to comment
Share on other sites

how about you do some good monitoring first, before you point fingers.

Login to mysql and take a look on the current running sql-queries with SHOW PROCESSLIST and make conclusions from that

Besides that, take a look on the webserver(s) and see what they're doing during peak time.

I have 2 webservers myself, taking care of the PHP-codig, one database server and a seperate very cheap VIA-based fileswebserver which host our static files like images, js, css, etc with lighttpd.

One of the webservers is also running a seperate search engine for the board, xapian.

This serves 3000 people online. Granted, 50% are from the other subsites which shares the same user/sessiontable as the forum and the subsites are custombuilt for speed, but still.... the forum itself is still generating 1500 people online constantly without any hard changes.

Link to comment
Share on other sites

Yeah,
I dropped all indexes and recreated them.

I have been monitoring all the processes for 2 years now...This year especially. There are some slow sql queries as I posted above (this was before we got the new DB server)

I am about out of options with regards to pointing fingers at the cause of the problem.

We have been getting a decent amount of traffic because of this late season storm:

375 user(s) active in the past 15 minutes

Server load is 1.15 at the moment.

Here is a quick snap shot now:



top - 13:53:22 up 46 days, 18:07, 1 user, load average: 1.78, 1.34, 1.14
Tasks: 162 total, 2 running, 160 sleeping, 0 stopped, 0 zombie
Cpu(s): 18.1% us, 5.3% sy, 0.0% ni, 75.7% id, 0.0% wa, 0.0% hi, 0.8% si
Mem: 3994580k total, 2993612k used, 1000968k free, 188908k buffers
Swap: 2031608k total, 772k used, 2030836k free, 1413336k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18055 apache 15 0 205m 41m 16m S 18.9 1.1 0:42.17 httpd
17345 apache 15 0 206m 44m 17m S 8.3 1.1 1:10.28 httpd
19258 apache 15 0 206m 38m 12m S 5.6 1.0 0:16.32 httpd
19737 apache 16 0 205m 35m 10m R 4.3 0.9 0:02.40 httpd
17521 apache 15 0 206m 42m 16m S 3.3 1.1 0:56.32 httpd
18267 apache 15 0 205m 42m 17m S 3.3 1.1 0:38.72 httpd
17788 apache 15 0 205m 42m 16m S 1.3 1.1 0:51.96 httpd
18868 apache 15 0 206m 43m 17m S 1.3 1.1 0:24.59 httpd
1815 named 16 0 75940 18m 1944 S 0.3 0.5 65:48.17 named
16483 apache 15 0 208m 45m 17m S 0.3 1.2 1:22.62 httpd
17790 apache 15 0 206m 42m 15m S 0.3 1.1 0:50.66 httpd
19026 apache 15 0 205m 39m 14m S 0.3 1.0 0:16.36 httpd
19135 apache 15 0 205m 36m 11m S 0.3 0.9 0:15.77 httpd
19370 apache 15 0 205m 36m 11m S 0.3 0.9 0:08.45 httpd
19841 root 16 0 7432 1120 808 R 0.3 0.0 0:00.03 top
1 root 16 0 4876 596 500 S 0.0 0.0 0:02.09 init
2 root RT 0 0 0 0 S 0.0 0.0 0:01.96 migration/0




DB Server:




top - 13:53:44 up 46 days, 14:42, 3 users, load average: 0.27, 0.38, 0.44
Tasks: 124 total, 1 running, 122 sleeping, 0 stopped, 1 zombie
Cpu(s): 2.7% us, 0.3% sy, 0.0% ni, 95.3% id, 1.6% wa, 0.0% hi, 0.1% si
Mem: 4040568k total, 4003520k used, 37048k free, 79888k buffers
Swap: 2031608k total, 160k used, 2031448k free, 3046276k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4287 root 16 0 54240 23m 3236 S 21 0.6 11117:52 X
13437 mysql 15 0 823m 513m 4416 S 3 13.0 5658:10 mysqld
466 root 15 0 0 0 0 S 0 0.0 21:08.92 kjournald
4727 root 16 0 286m 48m 20m S 0 1.2 19:18.40 firefox-bin
24656 root 16 0 6148 1032 776 R 0 0.0 0:00.03 top
1 root 16 0 4752 556 460 S 0 0.0 0:01.34 init
2 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.09 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/2
7 root 34 19 0 0 0 S 0 0.0 0:00.01 ksoftirqd/2
8 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/3
9 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3
10 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/4
11 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/4
12 root RT 0 0 0 0 S 0 0.0 0:00.04 migration/5

Link to comment
Share on other sites

When I set the DB box up at the data center I had gnome running on it with firefox to do some stuff. Forgot to shut it down but that server is not even being touched load wise so I have not bothered shutting it down.

I am going to the DC tomorrow to swap out one of the raptor drives that died.

350 users on now




top - 14:14:58 up 46 days, 18:28, 1 user, load average: 0.89, 0.87, 0.84
Tasks: 166 total, 17 running, 149 sleeping, 0 stopped, 0 zombie
Cpu(s): 30.3% us, 8.3% sy, 0.0% ni, 53.6% id, 5.8% wa, 0.3% hi, 1.7% si
Mem: 3994580k total, 3031424k used, 963156k free, 189000k buffers
Swap: 2031608k total, 772k used, 2030836k free, 1450036k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21010 apache 15 0 205m 33m 8916 R 17.2 0.9 0:01.75 httpd
20662 apache 15 0 205m 37m 11m S 9.9 1.0 0:11.21 httpd
19135 apache 15 0 205m 40m 15m R 8.3 1.1 0:48.50 httpd
20706 apache 15 0 205m 35m 10m R 7.6 0.9 0:07.84 httpd
18445 popuser 16 0 122m 47m 2432 R 7.0 1.2 0:51.45 spamd
20711 apache 16 0 206m 36m 10m S 7.0 0.9 0:07.75 httpd
20130 apache 15 0 205m 39m 13m R 4.6 1.0 0:25.56 httpd
18618 apache 15 0 206m 42m 16m R 2.0 1.1 0:54.84 httpd
20665 apache 15 0 205m 37m 12m S 1.7 1.0 0:08.49 httpd
1815 named 16 0 75940 18m 1944 S 1.3 0.5 65:50.35 named
20136 apache 15 0 205m 39m 14m S 1.3 1.0 0:24.48 httpd
20671 apache 15 0 205m 37m 12m R 1.3 1.0 0:10.10 httpd
20876 apache 15 0 205m 36m 11m S 1.3 0.9 0:05.05 httpd
17345 apache 15 0 207m 46m 19m R 1.0 1.2 1:46.26 httpd
19258 apache 15 0 206m 41m 15m S 1.0 1.1 0:47.68 httpd
19370 apache 15 0 205m 41m 15m R 1.0 1.1 0:42.00 httpd
19863 drweb 16 0 19652 13m 812 S 1.0 0.3 0:00.12 drwebd
20579 apache 15 0 205m 36m 11m S 1.0 0.9 0:09.20 httpd
20660 apache 15 0 205m 35m 10m R 0.7 0.9 0:06.90 httpd
17065 apache 15 0 208m 46m 18m S 0.3 1.2 1:47.87 httpd
18868 apache 15 0 206m 44m 18m R 0.3 1.1 0:57.33 httpd
19273 apache 15 0 206m 41m 16m R 0.3 1.1 0:44.29 httpd
20093 apache 15 0 205m 39m 14m R 0.3 1.0 0:26.40 httpd
20666 apache 15 0 205m 37m 12m R 0.3 1.0 0:08.53 httpd
1 root 16 0 4876 596 500 S 0.0 0.0 0:02.09 init
2 root RT 0 0 0 0 S 0.0 0.0 0:01.96 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:01.09 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0.0 0.0 0:25.11 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.55 ksoftirqd/1
7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.64 events/0
9 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/1
10 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
16 root 10 -5 0 0 0 S 0.0 0.0 0:05.10 kblockd/0
17 root 10 -5 0 0 0 S 0.0 0.0 0:00.51 kblockd/1
18 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
110 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd

Link to comment
Share on other sites

437 user(s) active in the past 15 minutes


Anything I should be looking at right now?



top - 15:41:57 up 46 days, 19:55, 1 user, load average: 2.83, 2.21, 1.83
Tasks: 173 total, 3 running, 170 sleeping, 0 stopped, 0 zombie
Cpu(s): 26.8% us, 10.0% sy, 0.0% ni, 62.1% id, 0.0% wa, 0.0% hi, 1.2% si
Mem: 3994580k total, 3291744k used, 702836k free, 189484k buffers
Swap: 2031608k total, 772k used, 2030836k free, 1653952k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24432 apache 16 0 206m 43m 17m S 10.3 1.1 1:01.86 httpd
26119 apache 15 0 205m 39m 14m S 10.0 1.0 0:27.77 httpd
26216 apache 15 0 205m 37m 12m S 9.6 1.0 0:22.81 httpd
25988 apache 15 0 205m 40m 15m S 9.3 1.0 0:22.16 httpd
26537 apache 15 0 205m 37m 12m S 9.3 1.0 0:12.92 httpd
26471 apache 15 0 205m 38m 13m S 7.0 1.0 0:13.69 httpd
27023 apache 16 0 205m 31m 7812 S 3.3 0.8 0:00.10 httpd
26248 apache 15 0 206m 39m 13m S 2.7 1.0 0:22.34 httpd
25942 apache 17 0 205m 39m 14m S 2.0 1.0 0:30.58 httpd
26214 apache 15 0 206m 39m 13m S 2.0 1.0 0:18.61 httpd
26962 apache 15 0 206m 35m 9.8m R 1.3 0.9 0:01.12 httpd
25949 apache 15 0 206m 41m 15m S 1.0 1.1 0:28.07 httpd
25976 apache 15 0 206m 39m 13m S 1.0 1.0 0:22.33 httpd
26843 apache 15 0 205m 35m 9968 S 1.0 0.9 0:04.14 httpd
26757 apache 15 0 205m 38m 13m S 0.7 1.0 0:06.96 httpd
2674 root 15 0 40840 2568 1960 S 0.3 0.1 5:01.42 monit
20050 root 16 0 7428 1144 812 R 0.3 0.0 0:18.07 top
26120 apache 16 0 206m 40m 15m S 0.3 1.0 0:25.90 httpd
26479 apache 15 0 206m 38m 12m S 0.3 1.0 0:16.81 httpd
27021 apache 15 0 205m 31m 7664 S 0.3 0.8 0:00.30 httpd
1 root 16 0 4876 596 500 S 0.0 0.0 0:02.09 init
2 root RT 0 0 0 0 S 0.0 0.0 0:01.96 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:01.09 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0.0 0.0 0:25.28 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.56 ksoftirqd/1
7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.64 events/0


DB Server:


top - 15:42:37 up 46 days, 16:31, 3 users, load average: 0.56, 0.57, 0.57
Tasks: 124 total, 1 running, 122 sleeping, 0 stopped, 1 zombie
Cpu(s): 3.3% us, 0.4% sy, 0.0% ni, 93.3% id, 2.8% wa, 0.0% hi, 0.2% si
Mem: 4040568k total, 4007048k used, 33520k free, 79892k buffers
Swap: 2031608k total, 160k used, 2031448k free, 3046816k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4287 root 16 0 54240 23m 3236 S 21 0.6 11135:33 X
13437 mysql 15 0 823m 514m 4416 S 9 13.0 5695:58 mysqld
466 root 15 0 0 0 0 S 0 0.0 21:16.89 kjournald
4727 root 15 0 286m 48m 20m S 0 1.2 19:20.38 firefox-bin

Link to comment
Share on other sites

Our httpd.conf :



#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 120

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
#StartServers 8
#MinSpareServers 5
#MaxSpareServers 20
#ServerLimit 256
#MaxClients 256
#MaxRequestsPerChild 4000

##StartServers 8
StartServers 5
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 125
#MaxRequestsPerChild 1000
MaxRequestsPerChild 0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
#StartServers 2
#MaxClients 150
#MinSpareThreads 25
#MaxSpareThreads 75
#ThreadsPerChild 25
#MaxRequestsPerChild 0

##StartServers 2
#StartServers 3
#MaxClients 125
#MinSpareThreads 25
#MaxSpareThreads 75
#ThreadsPerChild 25
##MaxRequestsPerChild 1000
#MaxRequestsPerChild 0
</IfModule>

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 80

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
LoadModule access_module modules/mod_access.so
LoadModule auth_module modules/mod_auth.so
LoadModule auth_anon_module modules/mod_auth_anon.so
LoadModule auth_dbm_module modules/mod_auth_dbm.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule auth_ldap_module modules/mod_auth_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule asis_module modules/mod_asis.so
LoadModule info_module modules/mod_info.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule frontpage_module /usr/lib/httpd/modules/mod_frontpage.so



IPB Version 2.2.2 (ID:)
MYSQL Version MYSQL 4.1.20-log
PHP Version 5.0.4 (apache2handler) ( PHP INFO )
Disabled PHP Functions no information
Safe Mode OFF
Server Software Linux 2.6.17-1.2142_FC4smp #1 SMP Tue Jul 11 22:59:20 EDT 2006 x86_64
Current Server Load 1.71
Total Server Memory 3900 MB
Available Physical Memory 643 MB

I am a bit under the weather at the moment, how does one get the apache version? I think its 2.0 something or another.

Link to comment
Share on other sites

Here we go again


423 user(s) active in the past 15 minutes


top - 22:45:58 up 47 days, 2:59, 1 user, load average: 13.55, 6.45, 4.06
Tasks: 202 total, 11 running, 191 sleeping, 0 stopped, 0 zombie
Cpu(s): 58.5% us, 15.1% sy, 0.0% ni, 21.7% id, 0.0% wa, 0.0% hi, 4.7% si
Mem: 3994580k total, 3962244k used, 32336k free, 164900k buffers
Swap: 2031608k total, 6812k used, 2024796k free, 2334996k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19056 apache 16 0 206m 35m 10m R 28.1 0.9 0:05.12 httpd
19308 apache 16 0 206m 34m 8784 R 24.4 0.9 0:00.71 httpd
18208 apache 15 0 206m 39m 13m S 16.9 1.0 0:25.33 httpd
19144 apache 15 0 206m 36m 10m S 15.0 0.9 0:04.02 httpd
19027 apache 16 0 206m 35m 10m R 13.1 0.9 0:04.70 httpd
19239 apache 15 0 207m 35m 9448 S 13.1 0.9 0:01.59 httpd
19133 apache 15 0 205m 34m 9048 S 5.6 0.9 0:03.84 httpd
19207 apache 16 0 205m 33m 8944 R 5.6 0.9 0:02.05 httpd
18632 apache 15 0 206m 38m 12m S 3.8 1.0 0:17.22 httpd
19029 apache 15 0 205m 34m 9636 S 3.8 0.9 0:07.26 httpd
19059 apache 15 0 206m 35m 10m R 3.8 0.9 0:04.07 httpd
19094 apache 15 0 206m 37m 12m S 3.8 1.0 0:05.48 httpd
13181 root 15 0 7428 1156 812 R 1.9 0.0 0:18.97 top
18819 apache 15 0 205m 37m 12m S 1.9 1.0 0:11.93 httpd
18991 apache 15 0 206m 36m 10m R 1.9 0.9 0:05.70 httpd
19268 apache 15 0 205m 33m 8312 S 1.9 0.8 0:01.02 httpd
19272 apache 15 0 201m 22m 1752 S 1.9 0.6 0:00.09 httpd
19273 apache 15 0 203m 27m 5148 S 1.9 0.7 0:00.18 httpd
19274 apache 15 0 205m 32m 7940 S 1.9 0.8 0:00.67 httpd
1 root 15 0 4876 596 500 S 0.0 0.0 0:02.11 init
2 root RT 0 0 0 0 S 0.0 0.0 0:01.97 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:01.09 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0.0 0.0 0:25.92 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.56 ksoftirqd/1
7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.64 events/0
9 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/1
10 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
16 root 10 -5 0 0 0 S 0.0 0.0 0:05.15 kblockd/0
17 root 10 -5 0 0 0 S 0.0 0.0 0:00.52 kblockd/1
18 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
110 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
112 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
178 root 15 0 0 0 0 S 0.0 0.0 0:07.60 kswapd0



DB Server:



top - 22:46:38 up 46 days, 23:35, 3 users, load average: 1.25, 0.96, 0.86
Tasks: 124 total, 2 running, 121 sleeping, 0 stopped, 1 zombie
Cpu(s): 5.6% us, 1.4% sy, 0.0% ni, 90.9% id, 1.8% wa, 0.0% hi, 0.2% si
Mem: 4040568k total, 4015400k used, 25168k free, 79948k buffers
Swap: 2031608k total, 160k used, 2031448k free, 3049072k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13437 mysql 15 0 823m 518m 4416 S 32 13.1 5836:05 mysqld
4287 root 16 0 54372 23m 3236 S 24 0.6 11205:13 X
466 root 15 0 0 0 0 D 0 0.0 21:47.43 kjournald
1 root 16 0 4752 556 460 S 0 0.0 0:01.34 init
2 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.09 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/2
7 root 34 19 0 0 0 S 0 0.0 0:00.01 ksoftirqd/2
8 root RT 0 0 0 0 S 0 0.0 0:00.07 migration/3
9 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3
10 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/4
11 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/4
12 root RT 0 0 0 0 S 0 0.0 0:00.04 migration/5
13 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/5
14 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/6
15 root 34 19 0 0 0 S 0 0.0 0:00.02 ksoftirqd/6
16 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/7
17 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/7
18 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/0
19 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/1
20 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/2
21 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/3
22 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/4
23 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/5
24 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/6
25 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/7
26 root 5 -10 0 0 0 S 0 0.0 0:00.00 khelper
27 root 15 -10 0 0 0 S 0 0.0 0:00.00 kacpid
64 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/0
65 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1
66 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/2
67 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/3
68 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/4

Link to comment
Share on other sites

Here we go again


423 user(s) active in the past 15 minutes


top - 22:45:58 up 47 days, 2:59, 1 user, load average: 13.55, 6.45, 4.06


Tasks: 202 total, 11 running, 191 sleeping, 0 stopped, 0 zombie


Cpu(s): 58.5% us, 15.1% sy, 0.0% ni, 21.7% id, 0.0% wa, 0.0% hi, 4.7% si


Mem: 3994580k total, 3962244k used, 32336k free, 164900k buffers


Swap: 2031608k total, 6812k used, 2024796k free, 2334996k cached



PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND


19056 apache 16 0 206m 35m 10m R 28.1 0.9 0:05.12 httpd


19308 apache 16 0 206m 34m 8784 R 24.4 0.9 0:00.71 httpd


18208 apache 15 0 206m 39m 13m S 16.9 1.0 0:25.33 httpd


19144 apache 15 0 206m 36m 10m S 15.0 0.9 0:04.02 httpd


19027 apache 16 0 206m 35m 10m R 13.1 0.9 0:04.70 httpd


19239 apache 15 0 207m 35m 9448 S 13.1 0.9 0:01.59 httpd


19133 apache 15 0 205m 34m 9048 S 5.6 0.9 0:03.84 httpd


19207 apache 16 0 205m 33m 8944 R 5.6 0.9 0:02.05 httpd


18632 apache 15 0 206m 38m 12m S 3.8 1.0 0:17.22 httpd


19029 apache 15 0 205m 34m 9636 S 3.8 0.9 0:07.26 httpd


19059 apache 15 0 206m 35m 10m R 3.8 0.9 0:04.07 httpd


19094 apache 15 0 206m 37m 12m S 3.8 1.0 0:05.48 httpd


13181 root 15 0 7428 1156 812 R 1.9 0.0 0:18.97 top


18819 apache 15 0 205m 37m 12m S 1.9 1.0 0:11.93 httpd


18991 apache 15 0 206m 36m 10m R 1.9 0.9 0:05.70 httpd


19268 apache 15 0 205m 33m 8312 S 1.9 0.8 0:01.02 httpd


19272 apache 15 0 201m 22m 1752 S 1.9 0.6 0:00.09 httpd


19273 apache 15 0 203m 27m 5148 S 1.9 0.7 0:00.18 httpd


19274 apache 15 0 205m 32m 7940 S 1.9 0.8 0:00.67 httpd


<snip>




DB Server:


top - 22:46:38 up 46 days, 23:35, 3 users, load average: 1.25, 0.96, 0.86


Tasks: 124 total, 2 running, 121 sleeping, 0 stopped, 1 zombie


Cpu(s): 5.6% us, 1.4% sy, 0.0% ni, 90.9% id, 1.8% wa, 0.0% hi, 0.2% si


Mem: 4040568k total, 4015400k used, 25168k free, 79948k buffers


Swap: 2031608k total, 160k used, 2031448k free, 3049072k cached



PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND


13437 mysql 15 0 823m 518m 4416 S 32 13.1 5836:05 mysqld


4287 root 16 0 54372 23m 3236 S 24 0.6 11205:13 X


466 root 15 0 0 0 0 D 0 0.0 21:47.43 kjournald


1 root 16 0 4752 556 460 S 0 0.0 0:01.34 init


<snip>


The first of these tops illustrates that while only half of your CPU cycles are being used, there are processes queuing up to get CPU but not being allocated any. Second, there are some Apache processes that are taking disproportionate amounts of CPU, all are using insignificant amounts of memory.

This suggests that something in Apache (or compiled into Apache) is taking a long time to execute, and not entering a wait state to allow other processes some CPU cycles.

I do believe that the next step is to look at your access logs for a 5 minute window around this top listing, and find out what Apache was being asked to do (i.e. what URLS were being hit).

PS. Hope you're feeling better today. :)
Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.

×
×
  • Create New...