Knowing What's Causing Server Load Rise

[SecondSight_merged]

Hello !

This morning, I got a server load of 8.80 making my board impossible to use for about one minute. It happened twice.

I recorded the mysql processes in a txt.file (attached here). Is it possible to know what is causing the server load ?

I had only 50 users online and my IPB version is 2.2.2. My server is a Dual Xeon 2.4 with 4 GigaBytes RAM, PHP is 4.4.6 (apache), MySQL is MYSQL 4.1.21-standard-log.

Thank you for your help ! :)

[djbob2]

My guess is either you cron'd a bunch of really intensive commands or you got DDOS'd.

[SecondSight_merged]

Thank you.

I will try checking for all this.

I put a system overview here. Does it show something abnormal in your opinion ?

Thanks a lot ! :)

[.CMANNS]

your http load is alittle high

[SecondSight_merged]

Well... There's something wrong. I can no longer access my board (I get a white page) and I have these figures in WHM :
Server load : 1.08
Memory used : 69 %
Swap used : 76.81%

[.CMANNS]

Well... There's something wrong. I can no longer access my board (I get a white page) and I have these figures in WHM :

Server load : 1.08

Memory used : 69 %

Swap used : 76.81%

PHP :-)

Codebox your conf here thats really the most server admin help I can give for free sorry unless you want to paste other conf's but it sounds like php's max memory or max exe times are set low and the load is causing it to time out....try restarting apafaeces

[djbob2]

That can't be it, as if you set PHP's max allocated memory or execution time too low for a script the script will encounter a fatal error and relate that to the user. Try restarting httpd and if that doesn't work try recompiling PHP. Make sure PHP is configured correctly in httpd.conf.

[SecondSight_merged]

Here is the my.cnf file. The key_buffer_size was set to 512 M yesterday after the problems. :

[mysqld] skip-innodb max_user_connections=1000 max_connections=2000 max_allowed_packet=1M key_buffer_size= 512M (was 96M) sort_buffer_size=2M join_buffer_size=1M read_buffer_size=1M read_rnd_buffer_size=1M table_cache=1024 thread_cache_size=64 tmp_table_size=48M myisam_sort_buffer_size=512M query_cache_size=32M query_cache_limit=2M interactive_timeout=60 wait_timeout=100 log-slow-queries = slow.log long_query_time = 3 log-queries-not-using-indexes log-long-format [mysql.server] user=mysql basedir=/var/lib old-passwords = 1 [mysqld_safe] err-log=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid open-files-limit=8192 [mysqldump] quick set-variable = max_allowed_packet=16M [mysql] no-auto-rehash #safe-updates [isamchk] set-variable = key_buffer=256M set-variable = sort_buffer=384M set-variable = read_buffer=16M set-variable = write_buffer=16M [myisamchk] set-variable = key_buffer=256M set-variable = sort_buffer=384M set-variable = read_buffer=16M set-variable = write_buffer=16M [mysqlhotcopy] interactive-timeout

I can access the directory where httpd.conf is, but I can't read the file with nano... ("nano httpd.conf" gives me a file which contains nothing) Which error did I make ?

Thank you ! :)

[.CMANNS]

That can't be it, as if you set PHP's max allocated memory or execution time too low for a script the script will encounter a fatal error and relate that to the user. Try restarting httpd and if that doesn't work try recompiling PHP. Make sure PHP is configured correctly in httpd.conf.

I've had it just white page before

[djbob2]

("nano httpd.conf" gives me a file which contains nothing) Which error did I make ?

Are you sure you are looking at the right httpd.conf? Sometimes complex installations have a few junk copies. Do you happen to be using a hosting control panel (ie. CPanel)?

Try /etc/httpd/conf/httpd.conf as well as /usr/local/apache/conf/httpd.conf.

[.CMANNS]

Are you sure you are looking at the right httpd.conf? Sometimes complex installations have a few junk copies. Do you happen to be using a hosting control panel (ie. CPanel)?

Try /etc/httpd/conf/httpd.conf as well as /usr/local/apache/conf/httpd.conf.

Thats when I'd hire a sysadmin

What OS is this? I'd recomend rhel/centos for you anyways run

updatedb if that works then run

locate or whereis httpd.conf

[kmf]

ah.. I see the reason that made me change to innoDB instead of myisam like 2 years ago. The locking of the tables...

when you update/insert something in a table with myisam, the table will be locked completely. this causes an avalanche of new threads and thus increasing load exponantionally. That is because the webserver will also wait for the queries to be finished....

Once I canged to innodb, the problems were gone.

I changed sessions, members, posts and topics to innodb. Gave innodb 500MB of cache to work with and tada.

[.CMANNS]

ah.. I see the reason that made me change to innoDB instead of myisam like 2 years ago. The locking of the tables...

when you update/insert something in a table with myisam, the table will be locked completely. this causes an avalanche of new threads and thus increasing load exponantionally. That is because the webserver will also wait for the queries to be finished....

Once I canged to innodb, the problems were gone.

I changed sessions, members, posts and topics to innodb. Gave innodb 500MB of cache to work with and tada.

Change sessions to heap :)

[kmf]

Change sessions to heap :)

nope. heap has table locks too, and it will eat away a lot of memory.
Besides, those are only the sessions. Topics and posts can't be heaped. Or at least I hop you don't make them heap...

[.CMANNS]

nope. heap has table locks too, and it will eat away a lot of memory.

Besides, those are only the sessions. Topics and posts can't be heaped. Or at least I hop you don't make them heap...

I was told to use heap :(

[kmf]

I was told to use heap :(

heap only helps a bit. it still causes table locking and if you have a very busy forum....

[.CMANNS]

heap only helps a bit. it still causes table locking and if you have a very busy forum....

ah thanks :thumbs:

I switched them all to innodb

I got this nice problem where i start mysql and it eats memory till swap then php dies :(

Don't youlove it when your server works fine for amonth and randomly screws up?

[kmf]

you should not switch everything to innodb, only the important and much updated tables.

Switching them all would make things really slow because you won't have enough innodb-cache.

[.CMANNS]

you should not switch everything to innodb, only the important and much updated tables.

Switching them all would make things really slow because you won't have enough innodb-cache.

heh

The forums are running about 20-70users online per 15 mins, one board is 20-150users depending on the time of day.

Should I just leave them as heap for now?

[SecondSight_merged]

Hello !

I'm back. I finally managed to read the httpd.conf file. I have these values :

Timeout 10

KeepAlive On

MaxKeepAliveRequests 2000

KeepAliveTimeout 10

MinSpareServers 10

MaxSpareServers 20

StartServers 15

MaxClients 2000

MaxRequestsPerChild 0

Here is the my.cnf file. The key_buffer_size was set to 512 M yesterday after the problems. :

[mysqld] skip-innodb max_user_connections=1000 max_connections=2000 max_allowed_packet=1M key_buffer_size= 512M (was 96M) sort_buffer_size=2M join_buffer_size=1M read_buffer_size=1M read_rnd_buffer_size=1M table_cache=1024 thread_cache_size=64 tmp_table_size=48M myisam_sort_buffer_size=512M query_cache_size=32M query_cache_limit=2M interactive_timeout=60 wait_timeout=100 log-slow-queries = slow.log long_query_time = 3 log-queries-not-using-indexes log-long-format [mysql.server] user=mysql basedir=/var/lib old-passwords = 1 [mysqld_safe] err-log=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid open-files-limit=8192 [mysqldump] quick set-variable = max_allowed_packet=16M [mysql] no-auto-rehash #safe-updates [isamchk] set-variable = key_buffer=256M set-variable = sort_buffer=384M set-variable = read_buffer=16M set-variable = write_buffer=16M [myisamchk] set-variable = key_buffer=256M set-variable = sort_buffer=384M set-variable = read_buffer=16M set-variable = write_buffer=16M [mysqlhotcopy] interactive-timeout

At the moment, I have to restart httpd and MySQL once a week otherwise the server is having problems.

Thank you for your help ! :)

[.CMANNS]

query_cache_size=64M

set-variable = key_buffer=256M
set-variable = sort_buffer=384M
set-variable = read_buffer=24M
set-variable = write_buffer=24M

[SecondSight_merged]

Thank you for your help. I will try these values. :)
Could you tell me what they are for ?

Thank you ! :)

[.CMANNS]

[myisamchk] & [isamchk]

Put it under both.

[Sonic98]

I recently switched from IPS hosting, and I got a package that with slightly higher specs than my IPS package. I forget which one I had. I wanted SSH access, but was told I would have to upgrade to the next highest package. I decided to go with VPS hosting if I was going to be paying more anyway. Soon after I upgraded from 2.2.1 to 2.2.2, and it seems like my server goes down about twice a week now. I have to re-start my VPS a few times a week. It doesn't even seem like I'm getting anymore traffic than I was previously, and I temporarily disabled gallery, so I could look through all the uploads to make sure there was nothing adverse that had been uploaded. My host says my site has outgrown to current package, but I don't seem to be getting anymore traffic than I was getting. I did check my resources, and there seems to be three resources that are constantly close to entering the black zone. This is really weird considering the site doesn't seem to be getting that much more traffic and we don't allow outside linking of files. I wonder if it could be something posted in threads or in signatures.

[.CMANNS]

I recently switched from IPS hosting, and I got a package that with slightly higher specs than my IPS package. I forget which one I had. I wanted SSH access, but was told I would have to upgrade to the next highest package. I decided to go with VPS hosting if I was going to be paying more anyway. Soon after I upgraded from 2.2.1 to 2.2.2, and it seems like my server goes down about twice a week now. I have to re-start my VPS a few times a week. It doesn't even seem like I'm getting anymore traffic than I was previously, and I temporarily disabled gallery, so I could look through all the uploads to make sure there was nothing adverse that had been uploaded. My host says my site has outgrown to current package, but I don't seem to be getting anymore traffic than I was getting. I did check my resources, and there seems to be three resources that are constantly close to entering the black zone. This is really weird considering the site doesn't seem to be getting that much more traffic and we don't allow outside linking of files. I wonder if it could be something posted in threads or in signatures.

Then talk to someone that knows how to run servers :P