Jump to content

Speeding Up Loading Times


Recommended Posts

A lot of people on a forum I'm on that recently switched to IPB are complaining that the forum is much slower now since switching from vB to IPB. I noticed it is a bit sluggish as well, but only by a second or two. There are a LOT of members, and millions of posts. No one will sacrifice pruning forums because they know they will lose their "precious post count." Any ideas for speeding up the forum, for someone who is no good with MySQL editing?

Link to comment
Share on other sites

  • Replies 53
  • Created
  • Last Reply

I hope so. My site that recently upgraded to version 3.1.1 is loading topics and posts rather slowly for some reason. Right when it gets to the top of the "Share this topic:" section, it stops for multiple seconds and loads the rest. :|

Link to comment
Share on other sites

  • 3 months later...
  • 3 weeks later...

Introduction

My site is: http://www.cyclechat.net - and I moved to IPB in July this year. Since then have been working to try and improve the speed of my site.

I'm no expert, and you may well need to look up some of this information and get a more technically minded person to properly install/configure it for you. I've managed to cobble it together for my own site, but everyone's set-up is different, so don't take my offerings as verbatim solutions.

I'm on a dedicated server that I manage on a day-to-day basis, but I'm by no means technically knowledgeable and learn / know enough to get by, and scavenge on a need-to-know basis as new stuff comes up. I'm therefore going to assume you have a similar basic knowledge and know how to use "top" to check your server load, and can download (wget) install, and run /mysqltuner.pl and /tuning-primer.sh

Also, I can't take much credit for any of this because it was either advised directly by IPS staff / community members; picked-up from various helpful web sites; or gleaned from other forums.

My server is an AMD Athlon 64 X2 Dual Core Processor 4400+, 4GB RAM, Debian Linux 2.6.26-2-686-bigmem on i686, Apache version 2.2.9, and MySQL version 5.0.51.

As well as my forums, I run a web directory from the same server, mainly kicking out basic static HTML pages with occasional CGI calls when people submit sites or search. Not a heavy load on the server by any means.

I moved from vBulletin 3.8.4 in July this year, to IPB 3.1.2 with all the additional modules.

My vB install had a few plugins:
• Cyb - Advanced Permissions Based on Post Count (4.4)
• Cyb - PayPal Donate (4.8.2)
• nCode Image Resizer (1.0.1)
• NoSpam! (4.0)
• Opt-Out Forums from Get New Posts / Get Daily (1.0.3)
• Reply to all - PMs (0.1)
• Separate Sticky and Normal Threads (3.0.0)
• vBadvanced CMPS (3.2.1)
• vbStopForumSpam (0.6)

I had around 1000 regulars visiting each day - along with lots of guests, 1.3 million posts, and 70,000+ threads (I'm still not used to calling them topics ... lol).

My server average load was between 0.6 - 0.8 and the pages loaded fast - both from the server, and visually.

Important:

IPB 3 series is NOT vBulletin. That may seem like a crazy thing to say, but if you've used vBulletin for years then you are going to find it virtually impossible not to compare your new IPB forums to your vB ones - but there is a fundamental difference - IPB is slower. Really? Well, yes, to be honest, it is. It has a different framework - links into multiple modules - uses much more advanced and complex templates and CSS - has more built-in hooks and inter-module connectivity - and so loads a slight bit slower. You will find it very difficult to get your IPB board to run at the same speed as your vB3 forums because they are different.

That being said, there are things that you can possibly do to improve the speed of your forums, and I'm going to document the ones I've tried (or can remember trying) in the hope that it will help you get some more speed out of your site.

They are in no particular order of importance, and there is no guarantee that any of them will make a difference to your forums, or even work in your server environment - but let's hope some of it helps some of you.



Tools

# top - for checking your *nix server load and memory usage

# ./mysqltuner.pl
# ./tuning-primer.sh

Both of these will help you tweak your my.cnf and allow you to test / optimise your MySQL install.

http://www.webpagetest.org/ - general guide for response times and what is being downloaded and how long it is taking. (Not a definitive or deadly accurate guide, just a general one.)

http://www.google.com/webmasters/tools - great for seeing what Google is doing when it visits and crawls your site.

http://www.google.com/intl/en/analytics/ - great for seeing traffic patterns and "stickiness" of your visitors (you'll need to add some tracking HTML to your templates).




PLEASE PLEASE PLEASE - ** BACKUP ** BEFORE CHANGING ANYTHING.
I can't emphasise how important it is for you to backup anything you want to change so that you can revert back if things go wrong, or doesn't work out for you. Please backup, please!!!


Hooks

If you've installed any third-party mods you may have a number of hooks that load on your home page or inner pages. Not all of these hooks are optimised, and some can slow your forums down without you realising.

To test whether any of them are putting an additional load on your board, make a note of them in Notepad (and the order they appear in on the enabled-hooks like - because you'll want to put them back in the same order) and then one-by-one disable them.

When they are all disabled, check the speed with webpagetest and have a look at your server load to see if it has reduced? Then, one at a time, enable the hooks again - testing as each one is enabled to see if one in particular has a big impact on loading speed or server load?

I found a specific one that added to the load for my home page - (IM) Members Online Today - disabling this hook make a marked difference to the load time for my home page.



<?php flush(); ?>

Credit: blair

This helps to speed up the visual perception of page loading. Try adding <?php flush(); ?> to your skin after </head>

Manage Skin Sets & Templates > Manage Templates in "your skin, Global Templates -> globalTemplate


</head>


<?php flush(); ?>


<body ........ >

blair had a problem using it once with some (non-standard) ad code, but no issues with Adsense. memcached If you haven't already got a caching system installed on your server, I would highly recommend it. You are pretty much guaranteed to see some increase in performance with a caching system in place. I use memcached on my server (others are available). Caching helps dynamic database-driven sites (such as forums) by caching data and objects in RAM, thereby reducing the number of times an external data source must be read. If you cannot install it yourself, ask your host to. See the IPS tutorial here - note the important steps of updating your conf file to enable memcached for your board. PHP through FastCGI FastCGI makes CGI more efficient by allowing persistent connections (instead of using singular connections that are trashed after each use), speeding up page delivery. You can push your PHP content through FastCGI to speed up delivery too. If you don't have it installed already, ask your host to install it. You'll then need to configure Apache (or whatever web server you're using) to parse php files through FastCGI. For my site, running Apache2, I enabled the module - fcgid - then edited /etc/apache2/apache2.conf and found the AddHandler section and added:


AddHandler fcgid-script .php

The further down the file, near the bottom, I added a directive to push .php files through FastCGI (you will obviously need to tailor this to suit your site / server setup):


<Files ~ (\.php)> 

Options +ExecCGI 

FcgiWrapper /usr/bin/php-wrapper .php

</Files>

You may need to experiment with this, or even get some expert help to get it installed and working, but this is another almost guaranteed speeder-upperer ... Note: I had to chmod 0755 my index.php to make it executable. Expires Headers mod_expires - For static content that hardly changes, such as your site logo - forum icons - post/reply buttons - etc. you don't really need to upload it for every page request to every visitor; it's more efficient to cache the stuff in your visitors' browser and have them serve it up locally, reducing bandwidth, the load on your server, and speeding up page loading. They will need to load the content the first time they visit your site / page, but thereafter it will be served from the visitors browser until it is old enough to be replaced. You set the age limit for this cached content using the directive settings in Apache (below). You will also need to enable the expires module for Apache2. Here's my settings, but have a look around the web at other people's and make up your own mind about how long you want to cache things for:


ExpiresActive on

ExpiresByType text/css "access plus 1 month"

ExpiresByType text/javascript "access plus 1 month"

ExpiresByType application/javascript "access plus 1 month"

ExpiresByType application/x-javascript "access plus 1 month"

ExpiresByType application/x-shockwave-flash "access plus 1 month"

ExpiresByType image/gif "access plus 1 month"

ExpiresByType image/jpg "access plus 1 month"

ExpiresByType image/jpeg "access plus 1 month"

ExpiresByType image/png "access plus 1 month"

ExpiresByType image/ico "access plus 1 month"

ExpiresByType image/icon "access plus 1 month"

Again, this can go down near the bottom of /etc/apache2/apache2.conf MySQL settings - my.cnf This is one item that can make a big difference, but which also cannot be covered here in simple form. You need to make small changes and monitor and re-do over quite a period of time, and because everyone's server and set-up is different, I would suggest you start a topic in the server optimisation forum, outlining your server and software details, and posting a copy of your my.cnf file: http://community.invisionpower.com/forum/406-server-management-resources-optimization/ mod_deflate This is an Apache module (mod_deflate) that compresses output before sending it to the client. Reducing the effective size of your web pages / content, allowing it to be sent faster. The browser then decompresses it at the other end. You can define what content is compressed (for example, you wouldn't want to waste time compressing already compressed image files) - here's my settings:


<IfModule mod_deflate.c>

BrowserMatch ^Mozilla/4 gzip-only-text/html

BrowserMatch ^Mozilla/4.0[678] no-gzip

BrowserMatch \bMSIE !no-gzip !gzip-only-text/html


DeflateCompressionLevel 9


AddOutputFilterByType DEFLATE text/html

AddOutputFilterByType DEFLATE text/plain

AddOutputFilterByType DEFLATE text/xml

AddOutputFilterByType DEFLATE text/css

AddOutputFilterByType DEFLATE image/svg+xml

AddOutputFilterByType DEFLATE image/x-icon

AddOutputFilterByType DEFLATE application/xml

AddOutputFilterByType DEFLATE application/xhtml+xml

AddOutputFilterByType DEFLATE application/rss+xml

AddOutputFilterByType DEFLATE application/atom_xml

AddOutputFilterByType DEFLATE application/x-javascript

AddOutputFilterByType DEFLATE application/x-httpd-php

AddOutputFilterByType DEFLATE application/x-httpd-fastphp

AddOutputFilterByType DEFLATE application/x-httpd-eruby

AddOutputFilterByType DEFLATE application/javascript

</IfModule>

(Note: You may want to turn off Gzip in the ACP - System Settings > System > Server Environment --- Disable GZIP encoding? = Yes) Apache KeepAlive I've changed these settings in /etc/apache2/apache2.conf to (I hope) lessen the load on my server: MaxKeepAliveRequests 500 KeepAliveTimeout 1 'sessions' table in memory If you have a fairly large or busy site and you haven't already done this, convert your sessions table to memory (HEAP):


ALTER table ibf_sessions ENGINE=MEMORY

Using Sphinx for search (large boards) Sphinx is a really fast dedicated search engine that can index your board content and produce lightning fast results. See the IPS tutorial here for setting it up. posts and topics tables to InnoDB This is not something that should be undertaken lightly, and is not guaranteed to work for every site, but because InnoDB offers record-level locking (your whole table isn't locked whilst data is written to it) it can speed things up for busy sites. (Note: Converting to InnoDB can use up to three times the disk space of the same table as MyISAM) STOP - Take a backup before doing anything with your tables - PLEASE - Take a backup!!!! You'll need to enable InnoDB support - edit your my.cnf file and (if it's present) comment out the following line:


#--skip-innodb

You'll also need to add some InnoDB specific settings. These are the ones I use, but you'll need to investigate which ones to use for your own server, and what levels to set them at:


innodb_log_buffer_size = 8M

innodb_flush_log_at_trx_commit = 2

innodb_flush_method = O_DIRECT

innodb_thread_concurrency = 16

innodb_additional_mem_pool_size = 16M

innodb_buffer_pool_size = 2048M

Restart MySQL to enable InnoDB support. In your ACP disable fulltext searching: System Settings > System > Search Set-Up: --- Use fulltext searching? = No. Turn off your board and wait a few minutes to ensure data isn't being written to the tables. From the command prompt, login to MySQL and issue the following command:


ALTER TABLE ibf_posts DROP index `post`;

Then ...


ALTER TABLE ibf_posts ENGINE=InnoDB;

Note: This may take quite a bit of time if you have a large posts table. Once complete, move on to the topics table:


ALTER TABLE ibf_topics DROP index `title`;

Then ...


ALTER TABLE ibf_topics ENGINE=InnoDB;



Once complete, open your browser and go to your site to ensure it is working okay? If it is, open your board back up again and away you go. If not, restore from your backup.



I'm pretty sure there were other things too, but for the life of me I can't remember them right now. If I do, I'll edit the post.


I have had a lot of support since moving over to IPB and this is my small way of saying thank you. I hope it helps some of you.

Cheers,
Shaun :D

Link to comment
Share on other sites

One thing that medium to large forums can do is to disable search engine logging. Once you've done that delete the logs. I did this last week after discovering that the logs were 67mb in my forum database. I have zero interest in viewing search engine logs. Keeping the database size down helps keep it fast. I optimize about once a week.

3DKiwi

Link to comment
Share on other sites

I'd agree with just about everything CycleChat has said. Just a couple of thoughts:

1. I'm not sure about using FastCGI over PHP as a DSO, I don't feel it is as beneficial as it sounds. I personally (and we at IPS) run PHP as a DSO and it works well.

2. If you're running PHP as a DSO, use the APC extension, it makes a big difference.

3. This script can help you find and optimize tables that need it:

<?php


$autoAnswerLarge = null;

if(array_key_exists(1, $argv))

{

	$autoAnswerLarge = ($argv[1] == 'y' || $argv[1] == '-y') ? 'y' : 'n';

}


$host     = 'localhost';

$username = null;

$password = null;

$limitdb  = false;


if(is_file('/etc/mysql/debian.cnf'))

{

	//----

	// Ubuntu:

	//----


	// Open file:

	$conf = file_get_contents('/etc/mysql/debian.cnf');


	// Get username:

	$matches  = array();

	preg_match('/user\s+\=\s+([^\n]+)/', $conf, $matches);

	$username = trim($matches[1]);


	// Get password:

	$matches  = array();

	preg_match('/password\s+\=\s+([^\n]+)/', $conf, $matches);

	$password = trim($matches[1]);

}

elseif(is_file('/root/.my.cnf'))

{

	//----

	// cPanel:

	//----


	// Open file:

	$conf = file_get_contents('/root/.my.cnf');


	// Get username:

	$matches  = array();

	preg_match('/user\=(\")?([^\"\n]+)(\")?/', $conf, $matches);

	$username = trim($matches[2]);

	$username = str_replace('"', '', $username);


	// Get password:

	$matches  = array();

	preg_match('/pass\=(\")?([^\n]+)/', $conf, $matches);


	$password = trim($matches[2]);


	if(substr($password, -1) == '"')

	{

		$password = substr($password, 0, -1);

	}

}

elseif(is_file('./conf_global.php'))

{

	//----

	// IP.Board:

	//----


	require_once('./conf_global.php');


	$host     = $INFO['sql_host'];

	$limitdb  = $INFO['sql_database'];

	$username = $INFO['sql_user'];

	$password = $INFO['sql_pass'];

}


if(!$username || !$password)

{

	die('Could not retrieve your MySQL access details.'.PHP_EOL);

}


$db = @new mysqli($host, $username, $password);


if($db->connect_error)

{

	die('Could not connect to MySQL: ' . $db->connect_error . PHP_EOL);

}


$res = $db->query('SELECT TABLE_SCHEMA, TABLE_NAME, DATA_LENGTH FROM information_schema.TABLES WHERE TABLE_SCHEMA NOT IN (\'information_schema\', \'mysql\', \'eximstats\') AND Data_free > 0');


if(!$res->num_rows && !$db->error)

{

	print 'No tables to optimize! :-)' . PHP_EOL;

}

elseif($db->error)

{

	print $db->error . ' :-( ' . PHP_EOL;

}


while($row = $res->fetch_assoc())

{

	if($limitdb !== false && $row['TABLE_SCHEMA'] != $limitdb)

	{

		continue;

	}


	$response = 'n';

	if($row['DATA_LENGTH'] >= 10485760 && is_null($autoAnswerLarge))

	{

		// If the table has more than 10mb of data in it, ask first. Otherwise just do it.

		print PHP_EOL . 'Optimize table: '. $row['TABLE_SCHEMA'] . '.' . $row['TABLE_NAME'] . '? (y/n) ';

		$response = trim(strtolower(fgets(STDIN)));

	}

	elseif($row['DATA_LENGTH'] < 10485760 || $autoAnswerLarge == 'y')

	{

		print PHP_EOL . 'Optimizing table: ' . $row['TABLE_SCHEMA'] . '.' . $row['TABLE_NAME'] . PHP_EOL;

		$response = 'y';

	}



	if($response == 'y' || $response == 'yes')

	{

		$result = $db->query('OPTIMIZE TABLE ' . $row['TABLE_SCHEMA'] . '.' . $row['TABLE_NAME'])->fetch_assoc();


		if(strtolower($result['Msg_text']) == 'ok')

		{

			print '--- Done!' . PHP_EOL;

		}

		else

		{

			print '!!! Warning !!! Response from MySQL was: ' . $result['Msg_text'] . PHP_EOL;

		}

	}

	else

	{

		print '--- Skipped.' . PHP_EOL;

	}

}


print PHP_EOL;


exit(0);

This will work on cPanel servers, Debian / Ubuntu servers, or any server with IP.Board on, but you'll need SSH access to run it. Put that in a file called optimize.php - If you have root access, save it anywhere, otherwise put it in your IP.Board root folder. When you run it, it'll come up with output like so:

[root@monkey forum]# php optimizer.php 


Optimize table: test_ipb.ipb_core_item_markers_storage? (y/n) y

--- Done!


Optimize table: test_ipb.ipb_core_sys_conf_settings? (y/n) y

--- Done!


Optimize table: test_ipb.ipb_core_sys_cp_sessions? (y/n) y

--- Done!


Optimize table: test_ipb.ipb_core_sys_login? (y/n) y

--- Done!


Optimize table: test_ipb.ipb_forum_tracker? (y/n) y

--- Done!


Optimize table: test_ipb.ipb_forums? (y/n) y

--- Done!



Obviously if you have a huge forum, you'll want to be careful about saying yes to tables like topics, posts, etc. unless you're running it at a time that is appropriate for your site to be sluggish.

Link to comment
Share on other sites


I'd agree with just about everything CycleChat has said. Just a couple of thoughts:



1. I'm not sure about using FastCGI over PHP as a DSO, I don't feel it is as beneficial as it sounds. I personally (and we at IPS) run PHP as a DSO and it works well.



2. If you're running PHP as a DSO, use the APC extension, it makes a big difference.



3. This script can help you find and optimize tables that need it:





Dan,

Another excellent addition to this topic - thanks.

I especially like the optimisation script and will add that to my server when I next get the opportunity.

One thing I don't understand though is the DSO comment in relation to FastCGI - could you expand on that with some specifics? (I don't know what DSO means, for example ... :blush: )

I'm also interested in APC too. I understand you can use both memcached and APC together to enhance caching, is that right, and how would one go about installing APC?

Thanks,
Shaun
Link to comment
Share on other sites


Dan,



Another excellent addition to this topic - thanks.



I especially like the optimisation script and will add that to my server when I next get the opportunity.



One thing I don't understand though is the DSO comment in relation to FastCGI - could you expand on that with some specifics? (I don't know what DSO means, for example ... :blush: )



I'm also interested in APC too. I understand you can use both memcached and APC together to enhance caching, is that right, and how would one go about installing APC?



Thanks,


Shaun




I should note that the DSO vs. FastCGI thing is merely my opinion. I've not benchmarked it or anything! DSO stands for Dynamic Shared Object (IIRC), mod_php is a DSO and is the way PHP runs as standard on most Apache installations. I've never had any trouble with running PHP this way. We're looking into using suPHP, but at the moment IPS is using this method of running PHP on a significant number of servers without issue.

As for APC, because of how it works I'm not sure it works at all on FastCGI. You can indeed use both memcached and APC, utilising memcached for it's key:value storage and APC for it's opcode caching (basically it caches the "compiled" PHP, so it doesn't have to be parsed on each request.
Link to comment
Share on other sites


I don't really think you can use memcached and APC at the same time.




Yes you can. You can't use IPB's APC integration at the same time as it's Memcached integration, but that's not what I was referring to. :)
Link to comment
Share on other sites


My site is lightning fast now! Thanks @CycleChat :D





Good to hear. :thumbsup:

I must admit that I'm pretty pleased with my own site now - it's responding much, much quicker than when I first moved to IPB and I'm sure it'll have a positive effect on my community too. It certainly makes the job of moderating / managing my site much easier as I can now flip from one part of the site to another and get a lot more done in the short space of time I have each day for keeping it all running.

I'll probably give APC a try sometime next week and see if that helps at all (just because it's about the only thing at server level that I haven't tried), and I'll report back if the results are favourable.

The only other optimisation I can think of is templates / CSS / images etc. but I haven't really looked at this yet because it's a fairly big job. IPB offers a lot (and I mean A LOT) of skinning flexibility (which is a good thing :thumbsup: ) and I can't imagine that removing a few small bits of code / CSS here and there are going to have much of an impact on overall speed.

If anyone else has any tips or success stories with speeding up their sites, please add to this topic so that other vB converts can use it as a reference source when they come over to IPB.

Cheers,
Shaun :D
Link to comment
Share on other sites

A question,

If I add this,

ExpiresActive on

ExpiresByType text/css "access plus 1 month"

ExpiresByType text/javascript "access plus 1 month"

ExpiresByType application/javascript "access plus 1 month"

ExpiresByType application/x-javascript "access plus 1 month"

ExpiresByType application/x-shockwave-flash "access plus 1 month"

ExpiresByType image/gif "access plus 1 month"

ExpiresByType image/jpg "access plus 1 month"

ExpiresByType image/jpeg "access plus 1 month"

ExpiresByType image/png "access plus 1 month"

ExpiresByType image/ico "access plus 1 month"

ExpiresByType image/icon "access plus 1 month"



Will the clients (members) automatically cache images or do I need to add some special code in themes too?

Thank you.

Link to comment
Share on other sites

Once you add that to your Apache conf (and restart Apache to enable it), visitors browsers will automatically cache those file types for however long you set them to - you don't need to change anything in your IPB setup.

There is one caveat that I forgot to mention - if you change any file that is of a type you've setup in the expires list, your visitors browsers won't update the files in their cache until they've 'expired'. So if you change your site logo it may be up to one month (or whatever time you set) before everyone has re-cached the new file.

You can, of course, use CTRL+R (in IE) to force refresh (re-download) everything, refreshing your local browser cache, which comes in handy if you want to check template / image changes.

Of course if you make big changes and need everyone to refresh their cache, you can set the times really low, or turn it off for a few weeks until enough time has passed for everyone's cache to fully expire.

Cheers,
Shaun :D

Link to comment
Share on other sites

There's almost no end to the number of performance tweaks that are possible. However, a couple more that deserve mention.

If on dedicated server, use an alternate webserver to Apache (lighttpd, nginx, litespeed).

If your budget will allow it, a CDN or Content Delivery Network can also make a big difference. It will likely speed the site for everyone, but especially for your international visitors. In general, the farther a visitor is from your server, the larger the benefit of a CDN. About the cheapest CDN offering I know of right now is maxcdn.com ($39 mo./1st TB). They also get good reviews. While they don't have an extension for IPB, they are working on one. I would like to see IPB offer more support, and easier integration for CDNs. Akamai is generally regarded as the best. Last I checked vps.net offered Akamai for $150 mo./TB. There are many other options, from your datacenter (ie. Softlayer CDN), to a VPS or shared plan with CDN integration. I don't know if IPS hosting offers a CDN option?

Whether dedicated server or not, optimize your graphics. There are lots of tools that will offer recommendations to speedup page load times. Firebug, YSlow, PageSpeed, and webpagetest.org are good. Often they'll recommend you smush your graphics. Tools like Yahoo Smush It and PngOptimizer will help.

Another graphic tip that has great potential is utilizing CSS Sprites. There is currently a [url=" being developed. But there are challenges to automating it, and manual optimization is probably out of the reach of most people.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.

×
×
  • Create New...