IPB Aggressive caching

[Gabriel Torres]

Hi guys,

We have a high-traffic forums at the moment with:

Queries per second avg: 189.481

I'd like to know about aggressive caching techniques for IPB.

We've just came from vBullletin, and on that platform we used a terrific product called vb Optimise:

http://www.dragonbyte-tech.com/product/1-vb-optimise/

The savings with this product were phenomenal (traffic on MySQL server came down to 20% of what it used to be), and I'd like to know if there is any similar product for IPB.

I've already enabled Xcache with $INFO['use_xcache'] = 1; but at this moment I still can't figure what it exactly caches (the var_names it uses are random letters and numbers, I'd have to open each one to see what is inside), and it doesn't seem to be aggressive enough.

If you could also post links to threads/articles on IPB optimization I'd appreciate as well.

Thanks,

Gabriel.

[Aiwa]

http://www.invisionpower.com/support/guides/_/advanced-and-developers/api-methods/kernel-caching-libraries-classcachephp-r153

https://www.invisionpower.com/support/kb/_/reducing-site-resources-hosted-clients-r24

[Grumpy]

Perhaps you might want to share with us more info about your board so we can tailor a solution to fit your needs specifically.

Stats like...

Your forum stats. Like member count, post count, online per x average, number of forums, how old some posts are (and if you have archiving enabled), etc.

Your server stats. Like server clustering (if any), hardware used, current stat outputs of top, etc.

I ask these because what you have can change what you should "aggressively" cache.

[Gabriel Torres]

Here is a snapshot of our system. Two servers, one web, one MySQL. Both with the same hardware configuration: DellPowerEdge R520 (2x Xeon E5405 2.2 GHz six cores each and HyperThreading, giving us 24 threads per server) with 32 GiB of RAM.

web server running Apache 2.2.26 + PHP 5.3.28 + Xcache 3.1.0 (var cache enabled -- will read the abovementioned articles to see if memcached would be a good addition). load average: 6,09, 5,46, 5,55 as of right now (08:40 PM local time), which is not our peak time (that would be lunch time).

DB server running MySQL server 5.6.15, load average: 0.08, 0.08, 0.12, which is great/normal. If it goes much above that we know that there is something wrong going on on the webserver. The MySQL server is adjusted based on the advice of mysqltuner.pl, and it is running great.

Servers are connected through a private Gigabit connection (each server has two Ethernet ports, one public and one private), so external traffic and MySQL traffic are handled in different ports.

Quick stats of the forums: a little bit over 1 million topics, almost 6 million posts, about 450,000 users. Older post is from 18 Jan 2002. Currently we are seeing about 1,700 users online.

Now, curently I think the system is running fine. I was mad yesterday that our forum was running very slow, but found out that something was slowing the forum, disabled several things and the forum is fast and stable. Later on I will start re-enabling features to find which one is causing us trouble.

So, I really think we don't need any hardware upgrades as of now, but it is always good to tweak our servers to improve the users' experience. O believe the articles posted above will give me some insights as well.

Thanks a lot.

[Grumpy]

Okay, by the sounds of your stats, I think you are running SSDs for the database server. If not, would strongly suggest doing so.

I agree, I don't think you need to upgrade hardware, but you could balance it more if you want. Here are my suggestions.

1. You should install a faster front-end web server like nginx or varnish instead of apache as your front handler. Apache is absolutely terrible with static files (2.4 is better it seems, but still terrible). So, a simplest solution would be to have something like nginx to handle static files and pass any other requests to apache at backend. This will work without any backwards compatibility issue and should improve performance quite a bit. You should then only need 24~48 (thread count: 1x ~ 2x) apache child nodes at max. This should also improve stability greatly as it will (probably) never go to a situation where you get resource maxed and causing freezes (which is frequent behavior of apache after high loads or after ddos). There's also varnish & nginx tutorials in this forum somewhere...
2. You should check the bandwidth usage between your db server and your web server on your priv network port; not just public port. If you use a lot of ip.content, your bandwidth requirements may become very high. If not, well, keep an eye on it at least. I've had issues of running way past gigabit limit of my network. (Same goes for memcached node if you choose to use memcache). If you see a very heavy bandwidth usage, you can check process queue in mysql (or install mtop) and see what often gets held up on "transferring data". And have your developer try to optimize that query (or post here...)
3. Don't follow mysqltuner blindly. Should you find yourself being recommended to raise tmp_table_size beyond 128~256MB, stop. IPB stores a lot of inefficient queries there and it's better that you don't store them and they get pruned off. (more)
4. You may want to enable >archiving. According to dev notes, it shaves about 30% sql queries according to their tests.
5. Your DB server is having it too easy. lol You might want to consider installing apache on the db server as well and offloading partial responsibility to that server. Don't handle any requests directly, just have your choice of front-end server (from #1) do a weighted distribution. Something like 2~3:1 (web to db). It will decrease load avg significantly and improve response time. In the odd chance that one of your apache goes down, the other will then serve as your only working node, increasing stability more.
   IPB is really not sql heavy, it's php heavy (given ssd).

That's about all I can think for now.

Not entirely about optimization, but... 2x E5405 is quite outdated (2007) and something from a single E3 line may have costed you less for much more performance.

[Gabriel Torres]

Thanks for the pointers. Will do some testings and fine-tunings.

Below you will see the traffic between the dedicated connection between our web and MySQL servers as of today. Between 150 and 200 queries/sec. Data below is in Mbps. Have fun.

[Makoto]

Raw queries per second doesn't really say much. There's a big difference in overhead between a few simple indexed SELECT queries and more complicated queries that involve multiple LEFT JOINS and the sort.

My QPS average is currently 189.256 (oddly close to yours) with only 300-400 users online in a 15 minute window at peak hours.

[Gabriel Torres]

Anyone could point me on the right direction on how to have Apache and nginx on the same server at the same time, with Apache handling PHP and nginx handling IPB's static content?

PS: Will move the database to an SSD and archive older topics either tonight or tomorrow, and will post a little bit about performance increase afterwards. Thanks.

[Makoto]

I don't see the point in using Apache just to handle PHP. I run all of my servers using Nginx and PHP-FPM without issue.

[Aussie Cable]

I don't see the point in using Apache just to handle PHP. I run all of my servers using Nginx and PHP-FPM without issue.

Agreed, I have the same setup (Nginx and PHP-FPM) and the results are great, compared to Apache alone handling PHP.

[GreenLinks]

If you are not experienced on Nginx configuration,installation i strongly suggest using a CDN service. CDN service may cost extra however it also comes with huge additional benefits.

Also you may want to increase Ram on DB server and use Memory to speed up things way further. Don't forget , you can even store Sphinx data on memory ;)

[Grumpy]

Anyone could point me on the right direction on how to have Apache and nginx on the same server at the same time, with Apache handling PHP and nginx handling IPB's static content?

PS: Will move the database to an SSD and archive older topics either tonight or tomorrow, and will post a little bit about performance increase afterwards. Thanks.

Like Kirito said, you can setup with nginx + php-fpm. But I mentioned apache backend for one main reason. It's LOT simpler since you already have apache. If you were on fresh setup, I would recommend php-fpm as well since it's lighter weight due to all the non-php stuff apache comes with.

You won't have to re-write the htaccess rules to nginx equiv. You don't have to setup php-fpm (duh...).

All you have to do is change apache setting so it's listening on some port other than 80. Like 8000 for example.
Then you install nginx. From something similar to a default setup have 2 sub locations.

location ~ .(jpg|jpeg|gif|png|apng|htm|html|etc.......)$ { # list out all your static files. This regex rule states must end with .ext
    expires 12h; #you can also set instructions like expires so that static files are stored on user's temp files.
    root /path/to/your/website/static/files # if requested is domain.com/folder/file.jpg, this will get triggered and bring from: /path/to/your/website/static/files/folder/file.jpg
}
 
location / { #this matches everything else
    proxy_pass http://localhost:8000; #your apache location
    proxy_set_header X-Real-IP $remote_addr;
    # more proxy_pass stuff here: http://wiki.nginx.org/HttpProxyModule
    # or you can use fastcgi_pass if you use something like php-fpm
}

and that's pretty much the only customization needed. There are many tutorials out there for nginx front end and apache backend tutorials. Apache is honestly not bad for php processing.

[Gabriel Torres]

Thanks. I decided to go fully with nginx, and opened a new topic here: http://community.invisionpower.com/topic/396839-nginx-optimization/

Thanks for all your help.

[Gabriel Torres]

I'd like thank everybody who participated in this thread. My original post was made before I had historical data for IP.Board. Now with 10 days running it, I can see that the MySQL traffic generated is similar to vBulletin 3.8.7 + vB Optimise. This leads to the obvious conclusion that IPB is already highly optimized. Of course, this doesn't mean that we can't try a few tweaks in order to improve performance even more.

Please refer to the attached image and it will be clear what I am talking about... (Dedicated traffic between web server and MySQL server)

Cheers! :smile: