Can anyone offer advice on mmap2 .. ?

[markopolo2002]

Hi guys, I've been trying to figure out a resource issue for some time now ... basically every single weekend (Sat night into Sun morn) for the past month or so, our site experiences major resource issues - it's almost like someone is attacking our site - yet during the week everything is fine, we can have a couple of hundred people on the site at any one point and you would hardly know they are there, but at the weekend it doesn't matter if we have 10 users on the site, the server is crippled.

Before I go any further, please let me explain I am by no means a server administrator or claim to even understand the troubleshooting I have done so far, I have merely picked up these things along the way.

From my WHM, processes, I was able to trace what was going on and all the PID's that were causing resource issues came back with the following:

munmap(0xb0d09000, 6295552) = 0

mmap2(NULL, 6295552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0d09000

munmap(0xb0d09000, 6295552) = 0

mmap2(NULL, 6295552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0d09000

munmap(0xb0d09000, 6295552)

Whereas those PID's that weren't causing any resource issues came back with a whole load of other stuff which I don't really want to post because it contains IP addresses, etc - basically ALL the PIDs causing resource issues came back with things similar to that above. I since disabled "keep_alive" in Apache, and things seemed to have went back to normal and the server loads are extremely acceptable once more so I'm not sure of the relevancy of all this but I'd appreciate any advice anyone can offer :)
Cheers
marko

[Grumpy]

I read this...
> basically every single weekend (Sat night into Sun morn)
and this...
> WHM

and I would come to the conclusion that you have an automated backup generated via WHM every sat night.
Check your backup schedule. If my guess is right, you know what the issue was. :P

[markopolo2002]

Hey Grumpy, thanks for the suggestion, but I had a ticket open on this one and Jason thought that as well, since checking the backup schedule on WHM there is nothing enabled for this - the host does perform off-site backups each day and as far as I am led to believe, there is nothing more taxing about the backup at weekends than there is during the week so I'm certain we can exclude backups as being the issue. The problem does disappear late Sunday night, early Monday morning but I can reboot the server as many times as I like on Sunday, the issue just always creeps back until Monday when it disappears totally ... the host just don't have a clue what the problem is, and I've only started noticing it since I upgraded to the latest version of IPB which I am presuming is sheer coincidence because as I say, Jason has looked at the setup and found nothing wrong at all.

I am considering moving host's but other than this problem, the host we have are fast and I've never been let down by them before with anything, it's just this issue that's driving me (and presumably them) up the wall !!

Jason also suggested looking into the fact there didn't appear to be any swap file enabled for our server, but from what I've searched virtuozzo doesn't support swap files as such, it has it's own method of swap files!. As I say, during the week the server is fast, but come Sat night/Mon morning it's all bet's off!

[Grumpy]

I guess we missed the window a bit... but try this test.

During weekdays (when no prob)

netstat -an | grep :80 | wc -l

iostat -x 60 3

and ioping (http://code.google.com/p/ioping/)

(the two lines of code in [ code] tags are separate. and the 2nd one will take 60 seconds to give results. First one will also take some time if you have lots of connections. So, don't make funny faces when it take a while)

Try it again when it's crappy.


Right now, my guess is that someone ELSE on the node is running backups or a heavy disk op at late saturday / early sunday.

[markopolo2002]

Grumpy, let me try and explain what's been happening in better detail so I don't waste your time mate ...

First time it happened I contacted the host and asked if they could take a look at what was happening - they explained they couldn't determine what was causing the high loads but they restarted the server and "cleared" the issue - LOL!. About 30 minutes later the server was crawling to a halt again, so I replied to the ticket and they took another look - they knew Apache was being stressed but again, couldn't figure out why. They installed memcached which did appear to do the trick, but bear in mind we didn't know then that the issue disappeared on Sun night/Mon morning. The following weekend, the same thing happened, so I raised another ticket and they basically said they still didn't know why these httpd requests were maxing out the CPUs, but would try and investigate.

I mentioned to them that could they check the entire server, rather than just my VPS and tell me if anyone else was causing these issues. They did and confirmed that the CPUs across the server were fine, it was just my "container" that was showing unusually high usage. By this time I'd made another post here on Invision and an extremely helpful guy called Peter offered to help investigate. He determined that the only thing he could conclude was some of the downloads we held locally were stressing out Apache and that an nginx server could help - so he talked me through that process and again, all seemed to go great - again, by this time we were close to the Sun night/Mon morning. The next weekend, same again and by this time I realised it was isolated to weekends only and we had only Sunday to troubleshoot the issue otherwise we'd loose the window, so to speak.

Another ticket and a request for IPB to become involved and still no further forward unfortunately, but the support guys here did confirm as far as they were concerned there was nothing wrong with IPB and so it had to be a host/external issue. Using what limited knowledge I have on troubleshooting I monitored "top" from the shell and when I noticed a PID stressing out the CPU I'd take a note of it and run "lsof -p #####" on the PID to see who was making the request's but nothing unusual came back, as far as I was concerned these requests were valid enough, or at least I could see nothing which would make me concerned.

So, I'm back to square one, but I will run those commands you gave and see what results we get during the week and over the weekend - but at this stage I can only be sure of one thing, and that is we will most certainly have the same issues come this weekend. I've already explained to my host that if the same thing happens this weekend and they can't offer any solution or explanation then I'll be seeking alternative hosting - I'd hate the thought though of moving to another host and finding the same issue!!.