Insane Server Load

[TheEnd-]

Hello,

Recently I upgraded to IPB 3.1.4 (sometime in December) and since last July I've been on a VPS. My traffic (last July) was around 6,000-8,000 pageviews per day.

Currently I get around 15,000-25,000 pageviews a day, and my traffic is getting higher every day.

Board Specs

• IP.Board v3.1.4
• Around 30 mods installed all together (mostly small mods that aren't resource hogs)
• 100,000 members
• 300,000 posts
• Less than 100 forums
• About 1,000 posts per day
• Around 1,000 members online daily
• Average around 100-150+ users active at any time
• Peak times we get in the 180+ range for active users
• Late night we drop to around 50 active users
  

Server Specs

• Dual (x2) Intel Xeon E5620 @ 2.40GHz
• 2GB of Dedicated RAM
• 4GB of SWAP
• 285GB of disk space on RAID10 drives
  

This is on a VPS with an "Atomic" plan (XEN based architecture) essentially its a "dedicated" server, but a VPS with more isolation; it isn't exactly a dedicated server, but the best thing you can get for under $100 a month. Running Linux, CentOS 64 bit.

My problem I'm having right now is that at peak times during the day, or randomly throughout the day, my server just skyrockets in loads. It'll jump to well over 100+ load, sometimes hitting 200+, which is just beyond insane.

I've done everything I can to try to keep loads low, but I honestly don't know what I'm doing wrong. I noticed that during peak times, I only have less than 200MB of dedicated RAM free. My average serverload (when things are actually running and not crashing & burning) is around 2 to 3.

So, with all of that information, I need some help to answer these questions...

Am I on the right kind of server/using the correct hardware for the amount of power my board outputs?
If I am using 90% (or more) of my server's RAM, is that what is causing the server to have insanely high server load at times?

I need some advice because at this point I'm just confused about what I'm doing. I've spent a lot of money on hosting and whatnot, and I feel like every time I "upgrade" I'm not actually doing anything at all... just wasting money.

[TheEnd-]

I don't know if this helps: the process that is killing the VPS is MySQL. It uses 100% of the CPU and then it goes haywire and I can't access my VPS root or my website until I contact support at my hosting company and have them reset my VPS for me, which has to be annoying as hell for them.

[tkheadcase]

When the load goes high and your low on memory can you see if the amount of swap in use is climbing? My guess is that your swapping a lot, causing high i/o which is leading to the really high loads your seeing. Also hows your CPU usage? I know the box itself is dual xeon but how much of that is dedicated to you? You really might be better off with a low-end dedicated server, I mean you really have no idea how much load is getting put on the disks by the other VPS's on your node.

[TheEnd-]

When the load goes high and your low on memory can you see if the amount of swap in use is climbing? My guess is that your swapping a lot, causing high i/o which is leading to the really high loads your seeing. Also hows your CPU usage? I know the box itself is dual xeon but how much of that is dedicated to you? You really might be better off with a low-end dedicated server, I mean you really have no idea how much load is getting put on the disks by the other VPS's on your node.

So, using up all of the RAM & then using most of (if not all) of my SWAP would cause load to increase? That is something I noticed. On my old VPS plan, I never used any SWAP, nor did it ever crash. Never in my life would I have predicted this amount of downtime from upgrading.

As far as other nodes taking up resources, it really hasn't been all that bad to be honest. I've been with this VPS host now for more than a year, and I've been pretty satisfied as far as performance goes, they haven't really let me down.

[3DKiwi]

With this size board you should be able to sell advertising and or take donations and pay for a proper dedicated server. I would say your memory is too low. I run a smaller forum than you but have 3gb on a dedicated server. I make enough from ads and donations to pay for the hosting.

It might also be worth your time to pay IPS to optimize your server assuming you have access to it.

3DKiwi

[TheEnd-]

With this size board you should be able to sell advertising and or take donations and pay for a proper dedicated server. I would say your memory is too low. I run a smaller forum than you but have 3gb on a dedicated server. I make enough from ads and donations to pay for the hosting.

It might also be worth your time to pay IPS to optimize your server assuming you have access to it.

3DKiwi

We do accept donations, and I am making a small income (~$150 a month) from Google Adsense.

I will take your advice along with tkheadcase's, I think I'll upgrade to the next package (4GB of dedicate RAM) to see if that solves the issue, if not, I'll cancel and move to a dedicated server.

[Martin A.]

Have you tried running a script like mysql_primer? You can have as much RAM as possible, but when your configurations are inefficient, all that RAM goes down the drain.


EDIT: Also, keep a SSH window open that's running "tail -f /var/log/syslog", that way you may see if something unusual activity is going on before the load shoots to the roof. I had a similar issue where a Turkish hacker attempted to bruteforce his way in to my email, causing the VPS to run out of memory and shut down.

[Gary.]

Your community should be on a dedicated server, However it may run depending on how many other users are on the node, If its an OPENVZ node then you must move, If your XEN then just optimization is needed.

What apache are you running on and what apache modules,

EG: suPHP, CGI, FCGI and so on...

Now to me I can only guess your on DSO, If I was you I would recompile the apache and then run suPHP and you can be sure your not keeping open connections and only serving them as they come in, This will also reduce memory.

You wish to check the var messages, Doing sylog will only show the system, And not the users on it so for all you know it could be your site causing the issue, Check with the following to show ALL.

tail -f /var/log/messages

while your in SSH moniter your memory usuage/scripts by the top cmd:

top

Then press:

shift + m

to sort the processes by the memory.

Make sure its not your mysql

vi /etc/my.cnf

Then reply with the output,

Also use mysqltuner and post the outputs:

wget mysqltuner.pl

Install, Then run the following:

perl mysqltuner.pl

[TheEnd-]

Your community should be on a dedicated server, However it may run depending on how many other users are on the node, If its an OPENVZ node then you must move, If your XEN then just optimization is needed.

What apache are you running on and what apache modules,

EG: suPHP, CGI, FCGI and so on...

Now to me I can only guess your on DSO, If I was you I would recompile the apache and then run suPHP and you can be sure your not keeping open connections and only serving them as they come in, This will also reduce memory.

Server Version: Apache/2.0.63 (Unix)
mod_ssl/2.0.63
OpenSSL/0.9.8e-fips-rhel5
mod_auth_passthrough/2.1
mod_bwlimited/1.4
FrontPage/5.0.2.2635

I am running on a XEN VPS, I was previously on an OpenVZ, but moved to XEN a week ago, since then, I've had almost zero uptime.

I will get back to you on the other information there.

Also, granted, I don't know very much about cPanel/WHM/Linux in general... so sorry if I give you the wrong info/act noobish. I'm doing my best here, haha

[Gary.]

In WHM go to:

Service Configuration > Configure PHP and SuExec >

And whats it show for the PHP 5 Handler

[TheEnd-]

In WHM go to:

Service Configuration > Configure PHP and SuExec >

And whats it show for the PHP 5 Handler

My VPS just drove itself into oblivion yet again, so I'll try and get back to you soon I hope.

Thanks for helping me (all of you) thus far, means a lot.

[TheEnd-]

And whats it show for the PHP 5 Handler

PHP 5 Handler suphp

Also, heres my my.cnf file from /etc: (yes, I know I have some ridiculous values)

set-variable = max_connections=1000 set-variable = interactive_timeout=10 set-variable = wait_timeout=10 set-variable = connect_timeout=10 set-variable = thread_cache_size=8 set-variable = key_buffer=128M set-variable = max_allowed_packet=16M set-variable = table_cache=1024 set-variable = sort_buffer=8M set-variable = record_buffer=8M set-variable = thread_cache=16 max_user_connections=150 #set-variable = thread_concurrency=4 set-variable = myisam_sort_buffer_size=64M set-variable = query_cache_limit=8M set-variable = query_cache_size=3M set-variable = query_cache_type=1 skip-locking #skip-innodb [safe_mysqld] err-log=/var/log/mysqld.log open_files_limit=8192 [mysqldump] quick set-variable = max_allowed_packet=16M [mysql] no-auto-rehash # Remove the next comment character if you are not familiar with SQL #safe-updates [isamchk] set-variable = key_buffer=8M set-variable = sort_buffer=8M set-variable = read_buffer=8M set-variable = write_buffer=8M [myisamchk] set-variable = key_buffer=8M set-variable = sort_buffer=8M set-variable = read_buffer=8M set-variable = write_buffer=8M [client] socket=/var/lib/mysql/mysql.sock

[mysqld]

I've also been getting constant driver errors on my forums when they are actually online, heres the logfile:

Date: Mon, 17 Jan 2011 04:25:44 +0000

 Error: 1203 - User theend_TheEnd already has more than 'max_user_connections' active connections

..which is really the only notable error in the file, most common one (which is also why I have "max_user_connections" set to a stupidly high number).

[Gary.]

I think you need to contact your host for some management, That my.cnf is 100% not right for that sort of enviroment and needs some heavy optimization.

whats the current cpu load ?

And you pay $100 a month for a VPS ?

[TheEnd-]

I think you need to contact your host for some management, That my.cnf is 100% not right for that sort of enviroment and needs some heavy optimization.

whats the current cpu load ?

About 81% or so, I'm looking at System Health > Process Manager

Screen:


I grabbed an example cnf file and just changed two values "max_connections" & "max_user_connections", thats it. So thats most likely why its extremely wrong. What would you suggest I use?

Again, sorry for my terrible knowledge of WHM and general server configs, I'm doing everything from self-taught trial and error.

[Gary.]

Each machine has it's own my.cnf, Nobody can every use the same rule sets, It will only cause issues, You need a fresh rebuild from the mysql settings and work on that starting with the mysql tuner.

I would also check into apache settings and the var/logs to see whats happening and where the load is coming from.

[cnmoore]

This kind of syntax is deprecated - see your /var/log/mysql-error.log
'set-variable = write_buffer=8M' should be 'write_buffer=8M', and so forth.

[mat206]

Install Varnish in front of IPS and convert your tables to InnoDB. That should fix most of your problems if not all. My site usually has 1100-1200 active members on within any 15 minute span.

[TheEnd-]

Each machine has it's own my.cnf, Nobody can every use the same rule sets, It will only cause issues, You need a fresh rebuild from the mysql settings and work on that starting with the mysql tuner.

I would also check into apache settings and the var/logs to see whats happening and where the load is coming from.

I went with my gut feeling along with your point there and went ahead and deleted the my.cnf file and just wrote a new one:

[mysqld]

skip-innodb

set-variable = max_connections=450

...and things are running pretty stable thus far. Seems to be okay.

Thanks for the help guys, and sorry for confusing the hell out of all of you. I can't believe I pretty much bent myself over in this situation. Sometimes I just wish I could take a class or two for this stuff in college, but that just isn't the case.

Again, thanks and the issue is resolved, for the most part. :thumbsup:

[Gary.]

set-variable = max_connections=450

This is still a HUGE amount and seting that high will consume and allow alot mysql to use the memory.

People must thing this is the max allowed user connections to the server, That is totally wrong. All that does is limit the user connections to the mysql, Why you would want 450 is beyond me and I cannot see even one of the biggest IPB forum's going any where near that, Infact you would be pushing it even by making it have 100.

Remember, The more you raise the more you allow your memory to be eaten up. That's how simple it can be :)