THE tunning thread [what did you do?]

Luis Manson · March 18, 2012

Hello, since is not THAT easy to find tunning advice on forums, or at least not very specific ones, im opening this thread with hope to make a "database" of tunning some experienced people do, i will post some stuff i did to make my site load bether and work with less resources, im not a senior admin, so maybe some of them are wrong or even useless...

Server hw: 8 cores intel i7 2.9Ghz - 4Gb RAM
board: board/content/chat/nexus
avg. status: ~600 online users most of the time - 500k posts - 100k members - load < 1.00 - most of the time
(my server also host other sites but not as popular as mine)

* Im also running Tomcat, dns, pop, imap, MySQL & SMTP/AV on the same server

Webserver:
Apache/mod_php +APC 5.3 for the site
nginx for static content (cache url?)

sysctl:


net.ipv4.tcp_window_scaling=0

tcp_tw_reuse=1

tcp_tw_recycle=1

net.ipv4.ip_local_port_range = 1024	 65535

net.ipv4.tcp_keepalive_intvl = 30

net.ipv4.tcp_keepalive_probes = 5

apache:


ServerLimit 120

<IfModule mpm_prefork_module>

	StartServers		  5

	MinSpareServers	   5

	MaxSpareServers	  10

	MaxClients		  120

	MaxRequestsPerChild   500

</IfModule>

.htaccess


<IfModule mod_headers.c>

	Header unset ETag

	FileETag None

	<FilesMatch ".(jpg|jpeg|png|gif|swf)$">

		Header set Cache-Control "max-age=360000, public"

	</FilesMatch>

	<FilesMatch ".(js|css)$">

		Header set Cache-Control "max-age=360000, public"

	</FilesMatch>

</IfModule>


<IfModule mod_expires.c>

	ExpiresActive On


	ExpiresByType image/x-icon A360000

	ExpiresByType image/gif A360000

	ExpiresByType image/png A360000

	ExpiresByType image/jpeg A360000

	ExpiresByType text/css A360000

	ExpiresByType application/x-javascript A360000

	ExpiresByType application/javascript A360000

</IfModule>


<IfModule mod_rewrite.c>

RewriteEngine on

RewriteBase /

Options -MultiViews



RewriteRule ^forum.php index.php?app=forums [L,R=301]

RewriteRule ^aplicaciones/ /forum/6-aplicaciones/ [L,R=301]

RewriteRule ^quench-modding/ /forum/41-quench-modding/ [L,R=301]

RewriteRule ^modding/ /forum/17-modding/ [L,R=301]

RewriteRule ^[^/]+/([0-9]+)-[^/]+.html http://www.MySite.com/showthread.php?t=$1 [L,R=301]

RewriteRule ^[^/]+/[^/]-[^/]+.html#post([0-9]+) http://www.MySite.com/showpost.php?p=$1 [L,R=301]

RewriteCond %{REQUEST_FILENAME} !-f

RewriteCond %{REQUEST_FILENAME} !-d


RewriteRule . /index.php [L]

</IfModule>



#Compression

<IfModule mod_headers.c>

	<IfModule mod_deflate.c>


		AddOutputFilterByType  DEFLATE text/css text/xml text/javascript  application/x-javascript  application/javascript application/xml applicacion/rss+xml application/xhtml+xml text/html

		BrowserMatch ^Mozilla/4  gzip-only-text/html

		BrowserMatch ^Mozilla/4.0[678] no-gzip

		BrowserMatch bMSIE !no-gzip !gzip-only-text/html

		Header append Vary User-Agent env=!dont-vary

	</IfModule>

</IfModule>

SQL




[client]

port			= 3306

socket		  = /var/run/mysqld/mysqld.sock


[mysqld_safe]

socket		  = /var/run/mysqld/mysqld.sock

nice			= 0



[mysqld]


user			= mysql

pid-file		= /var/run/mysqld/mysqld.pid

socket		  = /var/run/mysqld/mysqld.sock

port			= 3306

basedir		 = /usr

datadir		 = /var/lib/mysql

tmpdir		  = /tmp

language		= /usr/share/mysql/english

skip-external-locking

low_priority_updates = 1

open_files_limit = 2048


bind-address			= 0.0.0.0


key_buffer			  = 158M

max_allowed_packet	  = 16M

thread_stack			= 192K

thread_cache_size	   = 80

tmp_table_size		  = 64M

max_heap_table_size	 = 64M

innodb_buffer_pool_size = 128M


myisam-recover		 = BACKUP

max_connections		= 200

table_cache			= 1024

table_open_cache		= 2048

table_definition_cache  =	   2048

thread_concurrency	 = 12

concurrent_insert = 2

query_cache_min_res_unit		=1K



# * Query Cache Configuration


query_cache_limit	   = 4M

query_cache_size		= 64M

#


# Here you can see queries with especially long duration

log_slow_queries		= /var/log/mysql/mysql-slow.log

long_query_time = 10


expire_logs_days		= 10

max_binlog_size		 = 100M




[mysqldump]

quick

quote-names

max_allowed_packet	  = 16M

[mysql]


[isamchk]

key_buffer			  = 16M

raindog308 · March 18, 2012

Good thread!

How big is your APC cache? What percent full/percent hit do you usually see?

Are you using innodb? If not, remove it from your mysql.

query_cache_size might benefit from an increase. key_buffer_size is for indexes, so if it's significantly larger than the sum of all your indexes (.MYI if you're using MyISAM), you're probably wasting memory. (It's not going to hurt you, though).

You could download mysqltuner.pl (both the script name and the URL :-) and run it and post results. Its tuning method is crude but it provides some general insights.

You are oversized for CPU. If load is less than the number of cores, you have idle CPU...and since your load is less than 1.0 (8.0 would be all cores doing something), you have a lot of idle CPU. Again, this is not something that will hurt you.

At peak load, what does free -m show?

BTW, there's no reason you couldn't use nginx (+php-fpm+APC) as your sole webserver. Nginx scales better and is more memory-friendly.

Disk is always the slowest link - is this a server you built yourself or a dedicated server you rent? What is the disk configuration?

Luis Manson · March 18, 2012

thanks raindog, my xcache size is 64Mb 100% full var cache is 6.5Mb at 20% (its xcache, not apc, sorry)

im not using inno for my site, but there are other users of the SQL service

i had used tuner-primer a lot of times, but never give a good look at mysqltuner. i should!

not sure about mem on peak load, right now its midnight with 500 online, and i have 160Mb free, 2Gb cached

i was using nginx i love it but a few times pfm got crazy and the load went >50, but since i already had apache working fo other sites i enabled it again for my vhost, and "moved" nginx to listen on another IP so it can serve static content only

regarding apc+fpm i read a few times that since fpm are a lot of proceces with their own memory there were no benefif for APC

the server is for a shared hosting, im root there and the are only a few more sites. mostly static content. ;)

i should give a check at iostat right? or do you know other tool to have some statistics from the hdd

is a plain sata HDD

raindog308 · March 18, 2012

Make sure you're reading free memory correctly:

http://www.linuxatemyram.com/

(sorry if I insulted your intelligence)

You could look at the information_schema to see if anyone has created any innodb tables. If not, turn it off and tell people you don't support innodb. You'd free a couple hundred MB of RAM.

When I look at my APC stats, I see 99%+ of php is served through APC and I'm running nginx+php-fpm :-)

The reason I ask about disk is that if you only have a single hard drive, that will ultimately be your bottleneck. I would never recommend that anyway since you're only a single head crash away from having to restore from backup and long downtime.

ioping is another nice tool to see what the HDD latency is.

3DKiwi · March 18, 2012

I just paid Gary to optimize my server. Money well spent.

3DKiwi

surinp3 · March 18, 2012

I also recommend Gary. Very honest guy.
My money was well "not spent", as he told me there was nothing he could do to optimize my server further. :)

And how did I optimize my server?
I would like to give an easy answer, but the truth is you can only optimize a server so much though other peoples ideas and thoughts.
After that it takes a lot, and I mean a lot of testing and fine tuning.
Very time consuming, but I can now say: Me and my server are great friends. :)

Good luck anyways

Luis Manson · March 18, 2012

Make sure you're reading free memory correctly:

http://www.linuxatemyram.com/

(sorry if I insulted your intelligence)

You could look at the information_schema to see if anyone has created any innodb tables. If not, turn it off and tell people you don't support innodb. You'd free a couple hundred MB of RAM.

When I look at my APC stats, I see 99%+ of php is served through APC and I'm running nginx+php-fpm :-)

The reason I ask about disk is that if you only have a single hard drive, that will ultimately be your bottleneck. I would never recommend that anyway since you're only a single head crash away from having to restore from backup and long downtime.

ioping is another nice tool to see what the HDD latency is.

dont worry, i dont feel insulted, i was actually wrong since i knew about the cache/buffers but never readed the right data :|

about innodb it was easy to find a table using it, its just an old backup so i think i will remove it, although i was planing on move ipb post table to it...however i have yet to find more data about pros and cons.

i WAS using APC but had to move to xcache because i had A LOT of that "Potential cache slam" error and found no way to fix it at the time...

i checked dstat finally and got a small sample:

-dsk/total-

 read  writ

 209k  596k

   0   516k

   0	 0

4096B    0

  56k  108k

  12k    0

  40k  872k

 644k    0

 180k    0

 388k   16k

  24k    0

   0   936k

  16k    0

4096B    0

  36k  208k

   0	 0

 464k  748k^C

Grumpy · March 19, 2012

For "Potential cache slam" error, just turning it off does the magic trick. lol
apc.slam_defense=0
One of stupid features in APC imo... >.> More problems it causes than solves.

I suggest you increase your xcache size btw since you're full. IPB doesn't seem to need that much anyway (double, 128MB should be plenty).

As for your disk stats, this would be more meaningful...
iostat -x 60 2
and paste the 2nd result, not the first (first will be since boot). It'll collect the info for 60 seconds.
iostat is under the sysstat package.

Luis Manson · March 19, 2012

thanks, I added more memory to the cache, im still thinking if it will be any improvements with APC, since i saw some bench and they are head to head...

about iostat:


avg-cpu:  %user   %nice %system %iowait  %steal   %idle

		   9.72    0.00    1.27    1.38    0.00   87.62

Device:		 rrqm/s   wrqm/s	 r/s	 w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

sda			   0.02    27.92    6.63   36.82   135.73   510.00    14.86	 4.02   63.60   2.35  10.23

Grumpy · March 21, 2012

apc and xcache are almost always head to head in benches and in real life. You have to make specific scenarios if you want one actually noticeably beating the other which is a research no one would do except the companies that make it. :tongue: If your site is running fine with xcache, I suggest you stick with it. From my sysadmin who has much better stat collection than I do, he says xcache results in far fewer error rates. Though, my site specifically had problems with xcache, so, I'm personally using apc. didn't even bothered to figure out why, just went with apc.

Your iostat is showing rather a sluggish disk. You either have a single disk or a hardware raid disk. It's hitting high-ish utilization, slightly high io seek (await) and again, slightly high iowait. I think the disk quality is poor, the disk itself may be in imperfect health or you're just hitting iops serving capacity of your disk (my current guess is @ capacity. Your forum seems fairly busy). You can check the disk health if it's not on hardware raid (hw gets more complicated).

Take this comparison for example (this is RAID 1 SSD, so obviously not a fair comparison, but just to give an idea. This is about as good as it gets.)

avg-cpu: %user %nice %system %iowait %steal %idle 23.59 0.00 2.99 0.01 0.00 73.41 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 25.60 0.10 24.70 1.60 387.10 15.67 0.04 1.44 0.10 0.24 sda 0.00 25.40 0.00 24.90 0.00 387.10 15.55 0.03 1.38 0.08 0.21 md1 0.00 0.00 0.00 48.00 0.00 384.00 8.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

low util, low await even though the processor is doing more work (relatively. And this is a E3-1270 db server, and my processor's faster too).

await values of higher than 100 shows signs of problem for a site like forum to work well. For a download server, not a big deal at all.

The only tuning you can do to save disk usage notably is rather less pleasant ones like ramdisk... and opting for innodb if you have lot of spare ram by allowing write cache. Though, innodb isn't going to help with much else (well, it helps too if you have table locking issues... but that's really it! I think...).

Luis Manson · March 21, 2012

its a "consumer" samsung SATA disk...maybe thats why is so busy lol

i will check your post with more attention tomorrow :) thanks

bikedorkseattle · March 26, 2012

Who's Gary?

Aussie Cable · March 26, 2012

Who's Gary?

This is Gary.

Great advice, Highly recommended

Invision Community 4: SEO, prepare for v5 and dormant account notifications

Invision Community 5: Beta testing and latest updates

Invision Community 4: A more professional report center

Invision Community 5: A video walkthrough creating a custom theme and homepage

THE tunning thread [what did you do?]

Recommended Posts

Luis Manson

raindog308

Luis Manson

raindog308

3DKiwi

surinp3

Luis Manson

Grumpy

Luis Manson

Grumpy

Luis Manson

bikedorkseattle

Aussie Cable

Archived

Recently Browsing 0 members

More