Advice request for identifying high server load cause(s)

[Cretster]

Hi guys

I've had problems over the last month or two where my forum has been suffering a lot of outages (usually for just a few minutes at a time) due to high server load.

I've tried to act on this to reduce things that might be responsible, and in the last couple of weeks it seemed to have improved to the point where users weren't complaining, and I wasn't noticing any issues (I know there were still sporadic but brief outages still).

I have quite a few hooks so i've disabled some of them, and have made a few other tweaks, but a day or two back the forum had a really big out (several hours) which appears to have really upset a few people. It's a bit upsetting to see people who love the community getting so annoyed that they can't be bothered visiting due to the problem.

Unfortunately my tech knowledge on server stuff is limited, as is my access to a certain extent, so while I have suspicions I've not been able to identify what's causing this (since it has fairly rapidly appeared as a problem).

I'll try to give any potentially relevant details, but apologies in advance if I omit stuff that would normally be obvious to include:

It's a shared server but as daft as it sounds I don't know the actual hardware spec, sorry (is there something in Cpanel that will have this?)

I'm on v3.3.4, although I changed to this after the trouble began (previously was on 3.2.2 when it kicked off).

My host has showed me a screenshot from the server which showed the following (removed database username):

(tried posting actual pic rather than link to it, but not allowed .jpg it seems)

Pages seem a bit slow when the site is up too, but that might be entirely subjective I guess.

One thing I've noticed that I'm finding suspicious is that when I open the error log in cpanel (it shows the last 300 errors) it has a lot of errors relating to files not found, but I'm not talking about images/thumbnails etc, but it looks to me like someone is looking for the admin directory and things like that. I don't like the look of it one bit, and am worried there's some sort of attack going on that could therefore be presenting a high load on the server, but not sure how to tell for sure or what to do about it.

Apologies for sounding vague, not to mention perhaps incompetent, but I'd be grateful if anyone has pointers for more specific things I can look at, in order to narrow the problem of server load down.

Many thanks indeed!

[srpurdy]

It sounds like you might be getting DDOS attacked. I do have a question though. Is the server Shared as in your just 1 of many clients? or is it shared amoung a bunch of your own websites, and it's your server? I'm assuming it's a VPS or a server since I never seen a host provide an output of TOP like that before.

So assuming it's your server and you have root access.

What I suggest you do is since your running cPanel (if you have root access with WHM) you need to install a firewall. cPanel has a free 3rd party plugin called CSF/LFD. This will likely help. I'd also suggest installing mod_evasive although you'll need to compile that from source since cPanel doesn't provide this. However what you really need is a server admin that know's what to do, because any large site will run into these issues. You should look your access log and see if you have a large amount of accesses from a single ip address, and than block those ip's with iptables.

Also if your running php in fcgi mode. I'd suggest configuring fcgi so it can't run that many processes.

Hope that helps a bit. :smile:

And if it isn't your server, than I would say you're host doesn't know what they are doing. :smile:

[srpurdy]

To also add to my post above,

Your server is swaping, as you can see the cpu WA% is at 77% which means disk i/o is through the roof. This can also be a mysql issue with it's configuration. I would run mysqltuner.pl script to find out if mysql needs some adjustments.

You can also install iotop to view io usage. This might shed some light on the issue too.

Another question what is your dailiy or monthly traffic. So I can get a general idea of weither it's really a DDOS attack or a bad configuration.

[Cretster]

Many thanks for the replies - it's much appreciated.

The server isn't my own one (in any context), it's shared with other sites, but I have no info on what else it's shared with other than a local forum owned & run by my host.

I can't comment about his abilities as a host but he's hosted my forum for about 8 years now and I've never had any real problems in the past except for bandwidth getting a bit bad due to bots & spammers some time ago (I get round that with cloudflare now and it's very nicely kept in check). He's always been helpful enough and things have worked well over the years.

Am I right in thinking that with it not being my own server I won't be able to install the firewall you described?

Someone else I mentioned these problems to suggested the possibility of a DDOS attack potentially being the culprit. Is there any way to tell for sure if that's what is happening? I don't know a great deal about those, but as I understand it there's not a huge amount you can do about them right?

I can't help think that someone is trying to find their way into the admin panel though more than anything since a lot of the errors recorded in the log on cpanel are ones that relate to non existent file paths being tried (some have "administration" in them) by guest accounts. Now I can't see how those sort of errors could be caused by anything really other than someone trying to break in.

Blocking the IP addresses is a good idea actually. I know it's easy to use a different one, but I will have a closer look at the error files that point to attempted hacking (imho - could be wrong) and see if the same ones are cropping up.

"Also if your running php in fcgi mode. I'd suggest configuring fcgi so it can't run that many processes"

Sorry to be so noobish, but where do I check this, and what are the implications of making changes to that? I.e. if the number of php processes are limited, would that not just result in people receiving errors that stop them loading the site anyway when the load is high (so they're no better off really)? Sorry if I've misunderstood.

I'll have a look into that sql tuning script you've mentioned. I can bungle my way around sql at times when it's necessary but with a very cautious village idiot like approach to avoid trashing anything! :D

Re' iotop - I presume that's something that again, with it not being my own server I won't be able to install?

Traffic wise - here's a screenshot for a month worth:

Sorry if this isn't the right sort of data that you were after.

My bandwidth was getting up to about 80gb/month (and climbing each month) before I started using cloudflare, but since that's been in place the bandwidth is usually fairly steady around the 20gb/month mark I think.

Sorry for lots of questions, but the help is certainly appreciated.

BTW my host did message me a couple of nights ago to say he's arranged for a better server which he says will solve these issues, but while a better server is good to have of course my concern is that it's a band aid that doesn't address the underlying problem, so I'm keen to eradicate any risk that might be affecting things currently.

[srpurdy]

Well depends on the type of DDOS attacks. Some are very easy to prevent. Others are quite complicated.

As for tracking down a DDOS attack. Having a firewall in place makes this much easier to track the connections and access to SSH, but you might be able to look through your access_log and error log to find large amounts of traffic from the same ip address, but the server admin will have much information available through various server side logs. secure log is one place to start looking and the messages log.

And yes all the suggestions I made are Server Side suggestions. So there isn't really much you can do from your end with regards to those suggestions.

Also a site with your traffic can easly run off 3-4 php processes (infact it can probably run off 2) so this would not be an issue. (The fact the server is spawning so many processes is actually counter produtive to performance). This actually increases load by a lot if it keeps having to fork processes.

The reason I brought up mysqltuner is mysql is usually the culprit when it comes to high loads and large cpu waits, and the fact the server is swapping means it's writing/reading from disk instead of memory. This is really slow.

In many cases this could seem like the site is timing out because the actual confgiuration is setup to timeout after a certian amount of time has passed. So it seems like your having an "outage" but it's really just cause your request couldn't be met in time.

Your traffic looks good. And honestly from what I'm seeing this is a server problem. I run a couple servers myself. One site with 2 million monthly views and no issues. I would say though your borderline "Shared" as 236K pageviews is getting close to where you would benefit from a VPS solution. (keep in mind though you'll need to know much much more about running a server if you ever did goto a VPS)

The only thing I can suggest is your server admin to install a firewall, and even mod_security, (I'm assuming they already have a firewall at least) but they should be limiting the php processes, specially on a shared server, and also sorry if I came across mean. (about the host)

However proper isolation of sites should be in place on a shared server. If a site uses up all the power of 4 PHP processes for example They can't really continue to increase load on the entire server. with a 100+ load like that server is seeing. All of the sites on that server are going to be having problems. It's also a bit of a snowball effect rolling down a hill. The more process forking going on the worse it gets.

[Cretster]

Many thanks for the detailed explanation, and apologies for the delay acknowledging this. I'm afraid I've been somewhat tied up in the issues with the site itself and got sidetracked sorry.

That's a lot of useful background info for me to digest thanks. I appreciate the explanation you've taken the time to write out like that.

What you've said does in some respects give me a bit of peace of mind anyway, as I received an email yesterday from my host to advise that a new server is being implemented tomorrow evening, and an upgraded OS. He'd mentioned this was on the cards but I didn't realise so soon.

He's confident this will be the end of these problems, although presumably just because a newer more powerful server can cope with this load easily won't address the issue of these many processes running, unless I've misunderstood what you said and that they're the result of the slow hardware performance and thus should disappear with the new machine?

Am very much looking forward to the results of the upgrade anyway!

[srpurdy]

Well hardware is a weird thing sometimes lol. You can most certianly have what's known as zombie processes for many reasons. So I guess it's best to wait to see how things go on your new server. :)

[Cretster]

That's very true. Well, site is offline now, but at least this time it's actually intended.

Fingers crossed it all goes to plan. My forum members are for the most part a very good crowd and very patient. They know their site owner isn't the most adept person for the job really, but they're pretty fanatical about the site and I get a lot of praise for the way it's run. I'd have packed it in long ago if it wasn't for that.

Regardless of the outcome with this upgrade, changing to a dedicated server is something I've given thought to, but to afford that I'd need to get some money from the forum to cover it. I realise it's not a huge forum by any means but is pretty consistent, well thought of, and has been around for approx 8 years now. I get a few quid from viglink each month, and a small amount of sporadic subscriptions, but I've been wondering whether a site the size of mine is likely to be able to generate much from the likes of Adsense?

It's installed but barely pays me a penny, yet I've had someone else look at my traffic stats and claim I could be earning up to about £10k a year from adsense. Seems very much like wishful thinking, and I know there's no real way this stuff can be accurately predicted, but in the experience of people experienced with, and using tools like that already is my sort of traffic likely to get any useful sort of return?

I think the lowest I've seen for a dedicated server cost was about £50 a month, and if i could get that from some ads then I'd be pleased but I'm very ignorant of anything really beyond just installing the ad code and setting a basic ad.

Apologies if discussion on such matters is not ok here - it's certainly not my intention to break any rules if for some reason I have!!

[srpurdy]

Well I don't know if it's against the rules or not?

Anyway 10K a year is definitely do-able, and I would even argue if you play your cards well you can make 20K this wouldn't be all from ad-sense, but if you know your community very well you can target industries that fit into your communities demograph. And you can charge a premium for that kind of targeted advertising. Pitch to those kind of companies about what you can offer and how your users fit into they're industry, if you take a really professional approach to this you could likely land actual advertising besides ad-sense. :smile:

Btw with your traffic you don't really need a full dedicated server. A Small sized VPS server would be enough. :smile:

I run a site for a client that has 2 million monthly page views on a small VPS. mind you it's very well optimized both site wise and server side, but there is no reason with 250K views you need more than a 1 or 2GB memory VPS, and really I would only suggest 2GB to run extra security stuff. :)

[Analogged]

At first glace i would definitely say IO Wait is to blame for the high load avarages and i don't think you are running out of Memory or we would see the "kswapd" process in "top" which is the process that moves data back and forth from virtual memory.

You say that this has been happening for a month or 2, did anything change a month or 2 ago ? More traffic, Installed any new forum addons ?

Also does it happen during only certain times of the day ? If so are these high traffic peak times or off peak times? If it only happends during peak times it would lead me to believe that it the higher load causing issues and if MySQL is to blame possibly converting your sesssions table to memory if you have memory avalible might be an option if it's during off peak times usually during the same time of the day it might be a cron or a system task trying to do something such as optomise all your tables which might (if there's a problem or a huge table) cause IOWait.

Your unable to see any real info about the server in just a cPanel account if you do not have access to WHM but if you go to the server status window we can at least see how many cores you have to estimate if the CPU is low-med-high end.

[Cretster]

Thanks for the additional comments/explanation. In terms of pinpointing it back to a specific point it'd be a bit tricky I'm afraid. I used to get the occasional Cloudflare error showing the site as being offline, and it was normally assumed (incorrectly I think in hindsight) that these were brief transient issues with cloudflare itself, and were minor enough to not really investigate much further.

Hard to say when they increased but now I think of it, there was an increase over some months rather than an abrupt start date as it were.

I took the liberty of rolling back any recently installed hooks, but this was to no avail.

Resource wise, the server had 2 CPUs, but as of this morning has 4, and presumably they're much better ones now as well!

[Cretster]

Wow - never really thought even the lower number was a realistic thing to be honest. Either of those sort of figures would make a genuine difference to the standard of my life in terms of extra income that would benefit my family, but I wouldn't quite be sure how to go about making that happen really. It's not the sort of thing I'm good at.

Currently I get a small amount from subs paying members, a small amount from a couple of paying traders on the site, and a bit more from viglink (mostly from links to relevant ebay auctions), but that amounts to a small proportion of what's mentioned above here.

As I say adsense is on there but doesn't really generate anything.

I know my community well, as I started it up from scratch and am the sole admin for the last 8 years or so (it's a specific brand/model vehicle owner forum), but I think I'd have to try and find someone who's an expert at this sort of thing and offer them a bit of money to get things set up to start the gravy train rolling, since I presume it takes a lot of experience & knowledge in that area, and probably time, which I don't have in abundance.

Problem potentially with that is knowing who you can rely on to deliver, rather than just relieve me of some money and provide no benefit in return.

On another note (ie the original topic!!), the upgrade has gone through last night/this morning, and everything seems to be going really great so far aside from an error or two I had to fix re' memory allocation etc. I'm very very happy about that being dealt with I can tell you!

[Dmacleo]

adsense banned me right before first payout <sigh>

I wonder if the issues were happening at same time backups were.

[Analogged]

Yeah Cloudflare i find is more trouble then it's actually worth in some occations especially when dealing with their outages and issues. Then you have to instal an Apache Mod to make sure you can still see all your visters real Ip's and not cloudflare IP's. Also if not properly setup an attacker would still be able to find the true server ip just by doing some snooping then i guess the only plus would be when your offline you can greet your viewers with either a offline space page advertising cloudflare or a cache pages of your site also advertising cloudflare.

Double CPU is a nice upgrade, probably from a single hyperthreaded to a duel or possibly something along the lines of a core2quad. Point is moot now that the old CPU is out but it would have been nice to see what the CPU had itself clocked to if it was an intel using speedstep, using "cat /proc/cpuinfo | grep MHz" in putty would show you each core and current speed.

Adsence in my opionin is one of the best advertisers to use for the quality but the problem is that you sign your life away when you use them as they can essentially cancel your account and keep all your funds (returning them to advetiser...Hopefully) without warning if your site is breaching any TOU policies.

[srpurdy]

Well don't break the TOU and problem solved. :)

@Cretster

Yeah what you could is implement adsense and make some $ first before shelling out $ on someone that can help you.

I don't want to make it sound easy, cause it's not. You'll likely get more no's than yes' but the effort can pay off. :) if you can combine that with some growth that would help too. Sponsors typically want to see growth from your site. Or at least ones just paying for a banner location over say clicks and or impressions.

But from the sounds of it you can be doing better. So hopefully 2013 will be a good year for you. :)

[Dmacleo]

Well don't break the TOU and problem solved. :smile:

they offered ads tht were extremely pertinent to some users (regional issues) so the users looked at the ads. one user looked at about 4 a week max and he was highest though, others were once or twice a week.

thats only thing I can figure was going on. of course they refuse to tell you what was done wrong so I don't really know.

the one time I saw their crawler hitting the ads I reported it.

but at that time there were many similar sites getting the same stuff done so who knows.

[Cretster]

Yeah what you could is implement adsense and make some $ first before shelling out $ on someone that can help you.

..........

But from the sounds of it you can be doing better. So hopefully 2013 will be a good year for you. :smile:

Cheers mate. Yeah, I've had adsense running for some months now but I'd be lucky if it's earning £5 a month so far.

I've got no idea basically on what are the major aspects that dictate how it performs for the most part though, and I think that's the problem.

I realise that the positioning of the ad and the style of them affects things, but I'm struggling to accept that this could be the difference between £5 a month and a useful amount of money.

Surely there's a lot more to doing well from it than just gradual trial & error in terms of the positioning on the site and the style of the ad?

I'm assuming there's work that can be done in terms of really getting it to show the right ads, rather than ones that are vaguely in the right ballpark/subject, but realistically not ones that my members would likely ever use? I don't know if you can tailor it in that way since I've basically installed the code to show it on the footer of the site (all pages I think), with a basic text banner (as I'd read that generally they work better than ones with graphics or videos etc).

Hope everyone had a grand Christmas by the way!

[srpurdy]

Cheers mate. Yeah, I've had adsense running for some months now but I'd be lucky if it's earning £5 a month so far.

I've got no idea basically on what are the major aspects that dictate how it performs for the most part though, and I think that's the problem.

I realise that the positioning of the ad and the style of them affects things, but I'm struggling to accept that this could be the difference between £5 a month and a useful amount of money.

Surely there's a lot more to doing well from it than just gradual trial & error in terms of the positioning on the site and the style of the ad?

I'm assuming there's work that can be done in terms of really getting it to show the right ads, rather than ones that are vaguely in the right ballpark/subject, but realistically not ones that my members would likely ever use? I don't know if you can tailor it in that way since I've basically installed the code to show it on the footer of the site (all pages I think), with a basic text banner (as I'd read that generally they work better than ones with graphics or videos etc).

Hope everyone had a grand Christmas by the way!

I'd move the one in the footer into the header or stick it in the topic of the forum after the first post, and maybe make a "side" one for the side area and use that on your main site assuming you have one besides a forum. But I've seen around $500 USD monthly from just 1 ad with almost no effort. (although more like 2x the traffic at the time) From that simple math shouldn't be too hard to make at least $200 as a start. :)

Then again it may make some sense to investigate your traffic a bit more, for example if your getting a lot of "bad traffic" then this can really skew your stats. One of the sites I work on has 180,000 filtered views which never make it to the site, and probably another 200K or more than are dealt with at the server level. That's out of around 2.5million page views. another with around 350K page views and 50K filtered traffic, and probably another 50K after that. Really cuts the number down more into the 200K+ range in that case. So would be good to know what kind of traffic your actually getting.

One of my servers gets from 5000 to 10,000 attacks per day. So filtering out bad traffic and finding out what your real stats are may shed some light on things. How many active members are on your forum on average?

Maybe hook up cloudflare, it will at least weed out some bad traffic for you, even the free version :)