Advice on Server Configuration

[Linda A]

For the last few years, our website -- which features a forum with ~6 million posts and hundreds of concurrent users (with spikes sometimes into ~1000 concurrent, when a popular TV show is airing) and a MediaWiki install -- has struggled with the load of traffic it's received. We've augmented things by adding a Varnish cache, and even a second server, and still we struggle a bit.

So looking for some advice about whether the issues is tuning things right, or if we need to augment with more RAM or even another server!

Server 1:

Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

8GB RAM

Presently very underutilized, only handling the web traffic and MySQL DB of a CMS that is smaller than the wiki and forum by some margin. It used to handle the forum PHP until we set up IPS4 on the second server. It's often at ~0.30 load average

Server 2:

Intel(R) Xeon(R) CPU E31270 @ 3.40GHz

16GB RAM.

This one handles web traffic for the wiki and the forum, and the MySQL for both, plus a Varnish cache (though I just noticed it's reporting 0 hits... hrm!)

Besides fixing the Varnish issue, obviously, I know a major thing is that our forum is not set to use InnoDB for post tables. Last time I calculated, about 12GB of RAM would be effective for dealing with that. Hence if we could move off some or most of the PHP, we could then devote more memory to the database by reducing/eliminating PHP calls and Varnish cache requirements.

 

Does that sound like the right approach? Any other thoughts?

 

[p4guru]

Well first thing is you need to pin point exactly what is causing higher load issues at peak spike times. Monitoring and analysis of that is key. Sticky has tools https://community.invisionpower.com/topic/362126-tools-commands-to-diagnose-problems/. The mentioned mysqlmymonlite.sh tool is useful too http://mysqlmymon.com/ for this.

Newrelic or similar monitoring tools are very useful too

 

 

[Linda A]

The main issue is that until problems actually start (and they won't until a few months from now), it's kind of hard to say. As I said, in the past the issues were largely due to MySQL and its MyISAM tables choking, causing PHP to choke in turn. IPS4 seems to have less trouble with this, but we have lower traffic right now than expected peak traffic.

[p4guru]

Actually the lower traffic is also a good time to get that monitoring setup. It means you can regular analysis your current traffic and resource usage and do tweaking and tuning and double check those stats over time to see how setting changes respond in terms of resource usage for given traffic levels now. If you have regular daily peak and off peak times, then those are good times to test out as well.

Basically, the goal of the exercise is to get to know your server environment, resource usage requirements based on different traffic levels and patterns. You probably want to disable varnish cache so you can get a feel of the full load at current lower traffic peaks to get a better idea of how server setting changes and resource usage changes with traffic levels.

[Linda A]

All right, sounds like a reasonable approach to try and figure out a bit more about how the servers are functioning. Thanks!

[RevengeFNF]

@Linda A you need to tell us a little bit more about your server configurations. Right now, we only know it uses MyISAM tables.

First things first, i think any of your servers are more than enough to handle 1000 concurrent users. You need to convert all your tables to InnoDB, the performance gains under many concurrent connections are a lot. MyISAM uses table locking where InnoDB uses row locking.

Then, if you use a combination of Nginx + MariaDB, with all very well configured, there is nothing that can go wrong.

[Linda A]

What sort of information would be useful to know regarding the server environment/configuration?

I've looked into Nginx and a lot of the benchmarks I've seen suggest it's pretty much on par with Varnish. What would be the reason to favor Nginx?

And MariaDB definitely seems interesting. Is it basically a fork of MySQL? I can see it's supposed to have some performance improvements. Are they really substantial, or marginal?

 

[Gnuru]

25 minutes ago, Linda A said:

What sort of information would be useful to know regarding the server environment/configuration?

I've looked into Nginx and a lot of the benchmarks I've seen suggest it's pretty much on par with Varnish. What would be the reason to favor Nginx?

And MariaDB definitely seems interesting. Is it basically a fork of MySQL? I can see it's supposed to have some performance improvements. Are they really substantial, or marginal?

 

Hi Linda,

actually I don't use MariaDB, but MySQL 5.7 which brought a huge perfomance boost compared to 5.6 .

Also using PHP 7.02 while IPS doesn't officialy support it.

Shame on me: I noticed only after a year that the default file limits on my system where slowing down.

[RevengeFNF]

33 minutes ago, Linda A said:

What sort of information would be useful to know regarding the server environment/configuration?

I've looked into Nginx and a lot of the benchmarks I've seen suggest it's pretty much on par with Varnish. What would be the reason to favor Nginx?

And MariaDB definitely seems interesting. Is it basically a fork of MySQL? I can see it's supposed to have some performance improvements. Are they really substantial, or marginal?

 

Varnish is a cache server, where nginx is a webserver. While Nginx can do the same as Varnish, they are 2 different things.

Im asking what webserver you use, Apache?

MariaDB is a fork of Mysql, it shares the same features plus a lot more.

[Linda A]

We use Lighttpd primarily (Apache is used for a sub-site for complicated reasons). Our sysadmin has helped enormously in putting things together, but I admit I'm not 100% on everything.

Still on a somewhat older MySQL server, 5.5.45. 5.7 really improves IPB performance a lot? I'm more comfortable with MySQL, since, well, it's what we've always used. I'll have to see if there's any benchmark reviews comparing 5.7 to MariaDB.

 

 

[RevengeFNF]

You won't notice any diference between Mysql and Mariadb. Its a fork, the same code with more features. MariaDB 10.1 is really fast  Even the commands to backups etc etc, are the same.

If you are using Mysql 5.5, you can't use Innodb for all the tables, because it does not support fulltext search. Only after Mysql 5.6 it supports it.

[Linda A]

Oh, that's certainly good to know as well about MySQL. Guess it's time to upgrade that first and see what happens.

[RevengeFNF]

If i was you, but this is also with personal preferences, i would setup your webserver using Nginx+php-fpm and MariaDB 10.1 with all the tables using InnoDB engine. After that is just configuring MariaDB and Nginx to your site needs. Its almost bullet proof this configuration, and one of the most used out there.

[Ahmad E.]

6 minutes ago, RevengeFNF said:

If i was you, but this is also with personal preferences, i would setup your webserver using Nginx+php-fpm and MariaDB 10.1 with all the tables using InnoDB engine. After that is just configuring MariaDB and Nginx to your site needs. Its almost bullet proof this configuration, and one of the most used out there.

Cannot agree more. I use this setup on all of my servers

[IveLeft...]

+1 - I would also recommend NginX+php-fpm and MariaDB10.1

[Linda A]

We do use php-fpm at least. I will look into whether Nginx is a possibility instead, though.