Nginx+PHP-FPM secure server installation guide

Makoto · April 22, 2013

Deprecation Warning

This is an old guide written for IPS 3.4.x and contains advice and recommendations that are no longer valid or compatible with IPS4. For IPS4 installations, please use the link below to access the latest Nginx+PHP-FPM installation guide.

https://community.invisionpower.com/topic/424510-setting-up-a-secure-nginxphp-fpm-installation-for-ips4/

Setting up a secure IP.Board installation with NGiNX

This is a guide for those interested in setting up an IP.Board installation using NGiNX as their primary web server. This guide will mostly just cover the basics. In the future, I plan to write extensions to this guide for more advanced configuration techniques, such as global SSL/HTTPS support, Google SPDY and converting to MariaDB + XtraDB.

What will be covered in this guide:

Installing the latest stable release of NGiNX on Debian or CentOS
Basic NGiNX configuration
Setting up rewrite rules for IP.Board through NGiNX
Installing and configuring PHP-FPM
Setting up proper, secure permissions for your IP.Board powered website

Why should I use NGiNX over Apache?

If you have to ask this, you might not need or want to. Apache is the easier, more popular route people tend to take. NGiNX is a small but very powerful and highly efficient web server. Apache with mod_php can be inefficient and needlessly waste system resources. NGiNX can offer a potential gain in performance with the added benefit of not hogging your servers memory pool. NGiNX is also simply a personal preference of mine, even though I'm not resource constrained on my box.

NGiNX Installation instructions

Debian

file,

Spoiler
Add the following repository to your

/etc/apt/sources.list
# NGiNX Official Debian Repository
deb http://nginx.org/packages/debian/ squeeze nginx
deb-src http://nginx.org/packages/debian/ squeeze nginx
Now import the PGP key,
wget http://nginx.org/keys/nginx_signing.key
apt-key add nginx_signing.key
rm nginx_signing.key
Update and install,
apt-get update
apt-get install nginx
NGiNX should automatically start after installation. If you have Apache installed and running however, this will fail. So please ensure you stop (and optionally uninstall) Apache before proceeding further. If you don't uninstall Apache, just know the two will be fighting for ports and IP's to bind every time you restart the server. You can confirm NGiNX is running by using .

/etc/init.d/nginx status

CentOS

,

Spoiler
Add the following repository to

/etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
Now import the PGP key,
wget http://nginx.org/keys/nginx_signing.key
rpm --import nginx_signing.key
rm nginx_signing.key
Install and start,
yum install nginx
/etc/init.d/nginx start
If you have Apache installed and running, NGiNX will fail to start, so please ensure you stop Apache before proceeding further. You can confirm NGiNX is running by using . You can also have NGiNX start automatically on boot by running .

/etc/init.d/nginx status

chkconfig --level 345 nginx on

PHP-FPM Installation Instructions

Debian

file,

Spoiler

Add the following repository to your

/etc/apt/sources.list


# DotDeb Debian Repository
deb http://packages.dotdeb.org squeeze all
deb-src http://packages.dotdeb.org squeeze all

Now import the GPG key,


wget http://www.dotdeb.org/dotdeb.gpg
apt-key add dotdeb.gpg
rm dotdeb.gpg

Update and install,


apt-get update
apt-get install php5 php5-apc php5-cli php5-common php5-curl php5-fpm php5-gd php5-imagick php5-imap php5-mysql

CentOS

Spoiler
Nothing complicated here. Just install,
yum install php-cli php-curl php-fpm php-gd php-imap php-mysql php-xml php-pecl-apc
You can also have PHP-FPM start automatically on boot by running .

chkconfig --level 345 php-fpm on

NGiNX Configuration

Now that you have both NGiNX and PHP-FPM up and running, we can move on to configuring the web server. First, let's make some small adjustments to /etc/nginx/nginx.conf,

user  nginx;
worker_processes  4;

error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  30;

    #gzip  on;

    include conf.d/*.conf;
}

There's not much to say here. Increase worker_processes from the default of 1 to however many processor cores your server has. For example, if your server has a single quad core processor, set this value to 4. I've also changed the error_log directive to only log errors.

Now let's move on to configuring your IP.Board website. Use this as your base template for /etc/nginx/conf.d/ipboard.conf,

server {
    listen       80;
    server_name  yourdomain.com www.yourdomain.com;
    root         /srv/http/yourdomain.com/root;

    # Basic web server configuration.
    index        index.php;
    #access_log   off;
    client_max_body_size  1G;

    # GZIP static content not processed by IPB.
    gzip  on;
    gzip_static on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 3;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript application/xml application/xml+rss text/javascript application/javascript text/x-js;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6].(?!.*SV1)";

    # Set up rewrite rules.
    location / {
        try_files  $uri $uri/ /index.php;
    }
    location ~^(/page/).*(.php)$ {
        try_files  $uri $uri/ /index.php;
    }

    # Stub Status module
    location /server_status {
        stub_status on;
        #allow 127.0.0.1;
        #deny all;
    }

    # Deny access to hidden files
    location ~ /. {
        deny  all;
    }

    # Mask fake admin directory
    location ~^/admin/(.*)$ {
        deny     all;
    }

    # Secure real admin directory
    location ~^(/nimda/).*(.php) {
        #allow         127.0.0.1;
        #deny          all;
        #auth_basic    "Restricted Area";
        #auth_basic_user_file $document_root/nimda/.htpasswd;
        fastcgi_pass   unix:/var/run/php-fpm/ipboard.sock;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        /etc/nginx/fastcgi_params;
    }

    # IP.Board PHP/CGI Protection
    location ~^(/uploads/).*(.php)$ {
        deny     all;
    }
    location ~^(/hooks/).*(.php)$ {
        deny     all;
    }
    location ~^(/cache/).*(.php)$ {
        deny     all;
    }
    location ~^(/screenshots/).*(.php)$ {
        deny     all;
    }
    location ~^(/downloads/).*(.php)$ {
        deny     all;
    }
    location ~^(/blog/).*(.php)$ {
        deny     all;
    }
    location ~^(/public/style_).*(.php)$ {
        deny     all;
    }

    # Caching directives for static files.
    location ~^(/uploads/profile/).*.(jpg|jpeg|gif|png)$ {
        access_log off;
        expires    1d;
    }
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|htm|txt|swf|cur)$ {
        access_log off;
        expires    1w;
    }

    # Pass PHP scripts to php-fpm
    location ~ .php$ {
        fastcgi_pass   unix:/var/run/php-fpm/ipboard.sock;
        fastcgi_index  index.php;
        fastcgi_buffers 16 8k;
        fastcgi_buffer_size 16k;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        /etc/nginx/fastcgi_params;
    }
}

There are a lot of things to cover here. First things first, replace yourdomain.com www.yourdomain.com in the server_name directive with your forums domain name. Include both the www and non-www version of the domain as above, regardless of which you actually use. You'll also need to replace yourdomain.com in the root and error_log directives.

Next, let's cover where we're going to be installing the forum to. You'll notice I'm not using a /home/someuser heirachy as you're likely used to. There are two main reasons for this. Neatness and security. If you have your own server, you should keep your personal user account seperated from the actual web server. This is simply good practice in general. /srv/ on the Linux filesystem is for site-specific data which is served by the system. On all of my servers, I use a /srv/http/{host}/{subhost} hierarchy. "root" referrences the domains root. If you ever host another service on your forum (such as a wiki at wiki.yourdomain.com), you could store it in /srv/http/yourdomain.com/wiki

Let's go ahead and set this up for your IP.Board website now. Remember to replace yourdomain.com!

mkdir -p /srv/http/yourdomain.com/root
useradd --system ipboard
groupadd --system http
gpasswd -a nginx http
chown -R ipboard:http /srv/http/yourdomain.com

If you want to disable access logging on your site to reduce disk load, just uncomment the access_log directive under the basic web server configuration section. Either way, we'll be setting NGiNX up to not waste log entries for static resources further down.

Now adjust client_max_body_size to the maximum allowed file size users can upload to your site. So if your highest allowed upload size for any of your members anywhere on your site is 750 MegaBytes, set this to 750M.

Next, we get into configuring rewrite rules. There's nothing you really need to do here. All we have to do with NGiNX is pass a try_files directive, which is better than relying on the rewrite method required with Apache.

The Stub Status module allows you to see how many active connections your server has by visiting yourdomain.com/server_status. If you have a static IP address, you can set it up to ensure that only you will have access to this page, though leaving it public isn't really a vulnerability.

Next, we want a secure installation, right? So that means we're not going to be using /admin anymore. Come up with something creative that you can easily remember to replace /admin. The example above uses "nimda" (which is just "admin" backwards), so replace that with whatever you come up with.

Going a step further, there are two primary ways you can further secure your ACP. The most secure way would be to only allow connections from your IP. Though this will obviously not be feasible if you or another one of your administrators have an IP that constantly changes. If you do have a static IP that you can use, however, uncomment the "allow" and "deny" lines while adding an allow directive for your IP address bellow allow 127.0.0.1. The second option is to use .htpasswd authentication. IP.Board has a built in function to set this up for you in the security center. You can configure this after installing your forum. After creating the .htpasswd file in your ACP, all you will need to do is uncomment the auth_basic directives and restart NGiNX.

The caching directives are split for user avatars and everything else not processed by IP.Board. I have avatars set to expire after one day. For everything else it's one week. Feel free to adjust this to your own personal prefference.

PHP-FPM Configuration

On to PHP-FPM! First, let's go ahead and get rid of the default/example configuration we don't need. This is /etc/php5/fpm/pool.d/www.conf on Debian and /etc/php-fpm.d/www.conf on CentOS. Now, create a new file replacing www.conf with ipboard.conf and using this as the base template:

[ipboard]

; Set the prefix directory and the user/group to run under
prefix = /var/run/php-fpm
user = php-fpm
group = http

; Configure listen(2) directives
listen = ipboard.sock
listen.backlog = 4096
listen.owner = php-fpm
listen.group = http
listen.mode = 0660

; Set up the process manager
pm = static
pm.max_children = 10
pm.max_requests = 250
pm.status_path = /fpm.php
 
; The timeout for serving a single request. Prevents runaway scripts.
request_terminate_timeout = 5m
 
; Only execute .php scripts.
chdir = /srv/http/yourdomain.com/root
security.limit_extensions = .php
 
; Environment variables.
;env[HOSTNAME] = $HOSTNAME
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp
env[DOCUMENT_ROOT] = /srv/http/yourdomain.com/root

; PHP flags and security directives for just this site
php_flag[display_errors] = off
php_admin_value[open_basedir] = /srv/http/yourdomain.com/root:/tmp:/usr/bin
php_admin_value[disable_functions] = escapeshellarg,escapeshellcmd,exec,ini_alter,parse_ini_file,passthru,pcntl_exec,popen,proc_close,proc_get_status,proc_nice,proc_open,proc_terminate,show_source,shell_exec,symlink
php_admin_value[upload_max_filesize] = 1G
php_admin_value[post_max_size] = 1G

First, replace all instances of "yourdomain.com" accordingly again. Next, let's go ahead and set up everything we need for PHP-FPM to run properly,

mkdir /var/run/php-fpm
useradd --system php-fpm
chown php-fpm:root /var/run/php-fpm

You can skip creating the directory on CentOS, as it will have already been created, but you will still need to apply the new permissions.

Finding the optimal setting for pm.max_children is a bit of trial and error. I have it set to 10 as a default. You may need to increase or lower this setting depending on how constrained your server is and how much traffic your site receives. IP.Board tends to use an average of about ~52MB per process for me. So if you have 2GB of memory to spare on your server after accounting for MySQL and any other services you run, you could probably safely support up to 35 static processes, though you may not need nearly this many. Setting this too high may result in your server running out of memory, so be careful.

At this point, let's go ahead and test to make sure everything works. (Use /etc/init.d/php5-fpm restart for Debian)

/etc/init.d/nginx restart
/etc/init.d/php-fpm restart

No problems? Good! Problems? Double check your work and check your error logs.

Let's create /srv/http/yourdomain.com/root/test.php as a test file with a simple phpinfo() call,

<?php
phpinfo();
?>

Now visit yourdomain.com/test.php. You should see a PHPINFO page containing the open_basedir and other directives we set. If you don't, you did something wrong. Go back and review. If all is well, delete the test.php file. (Really, delete it. Security audit tools scan for files like this, and you don't want to needlessly expose your server information to malicious parties.)

Setting Up IP.Board

If you're setting up a new IP.Board website, you'll need to install mysql-server, set up a new database and so on. There are plenty of resources available for learning how to do this if you don't know how already. After you copy your entire forum directory into /srv/http/yourdomain.com/root, we just need to set up proper permissions,

chown -R ipboard:http /srv/http/yourdomain.com
httproot=/srv/http/yourdomain.com/root
find $httproot -type d -exec chmod 0750 {} ;
find $httproot -type f -exec chmod 0640 {} ;
find $httproot/{uploads,cache,downloads,hooks,screenshots,blog,public/style_*} -type d -exec chmod 0770 {} ;
find $httproot/{uploads,cache,downloads,hooks,screenshots,blog,public/style_*} -type f -exec chmod 0660 {} ;

If you haven't already, rename your /admin directory to whatever you chose earlier. Afterwards, create the following constants.php file in your forums root, replacing "nimda" appropriately,

<?php

    define( 'CP_DIRECTORY', 'nimda' );

?>

Update the permissions,

chmod 0640 constants.php
chown ipboard:http constants.php

And that's it! IP.Board should be up and running. If you're having trouble, review all of the above steps and check your error log (/var/log/nginx/error.log).

Aussie Cable · April 22, 2013

Well I guess I dont need to contribute to the thread now.

Dmacleo · April 22, 2013

if you don't yet have mysql installed this writeup covers it too

http://www.howtoforge.com/installing-nginx-with-php5-and-php-fpm-and-mysql-support-on-centos-6.4

have you tried anything with microcache yet? started to look into it yesterday but had issues with accessing /admin/index.php and did not have time to troubleshot.

AlexJ · April 22, 2013

All the domain related config files should go under conf.d/ folder or sites-enabled / sites-available folder?

Also for 2 different domains should i create separate files under conf.d folder? i.e. ipboard.conf and ipboard2.conf?

Thanks

Makoto · April 22, 2013

All the domain related config files should go under conf.d/ folder or sites-enabled / sites-available folder?

Also for 2 different domains should i create separate files under conf.d folder? i.e. ipboard.conf and ipboard2.conf?

Thanks

I put all site configuration files under conf.d/

You should create a separate configuration file for each site you run. If you have several IP.Board powered websites, you may want to use something more identifying than just "ipboard", "ipboard2", "ipboard3", and so on, but that's up to you.

have you tried anything with microcache yet? started to look into it yesterday but had issues with accessing /admin/index.php and did not have time to troubleshot.

Nope. I'll read into it though.

Dmacleo · April 22, 2013

would appreciate that, thank you.

Dmacleo · April 28, 2013

did you say you were looking at the ngx_pagespeed module?

Makoto · April 28, 2013

did you say you were looking at the ngx_pagespeed module?

I have. It's not ready for use in production servers though, don't bother with it right now. The developer even warns not to use it in a production environment.

Dmacleo · April 28, 2013

I did try it for about 10 minutes but did not see any gains. did see page freezes so I removed it.

Sull5 · April 30, 2013

Nice post. I upgraded from nginx 1.2.7 to 1.4.0 and wow what a difference in speed. I'm also running the pagespeed module and I'm not seeing any lockups or any issues yet, or maybe I've yet to find them. On my dev board I have spdy and pagespeed running :)

@Kirito do you have ant other speed enhancements up your sleeve, I checked out your one dev board you posted in the other nginx thread and damn that's one snappy site, really can't get much faster than that for speed :)

Makoto · April 30, 2013

Nice post. I upgraded from nginx 1.2.7 to 1.4.0 and wow what a difference in speed. I'm also running the pagespeed module and I'm not seeing any lockups or any issues yet, or maybe I've yet to find them. On my dev board I have spdy and pagespeed running :smile:

ngx_pagespeed worked fine for me on my development forum but would stop functioning under heavy load. I don't know how much it's improved since I last tested it though.

@Kirito do you have ant other speed enhancements up your sleeve, I checked out your one dev board you posted in the other nginx thread and damn that's one snappy site, really can't get much faster than that for speed :smile:

I've made a lot of various optimizations and improvements over time. My largest forum right now processes 20-25TB of data transfer a month, and I ended up having to upgrade my hardware configuration after a while due to the disk strain on my servers mostly stock single software RAID 1 setup.

This is essentially my current hardware configuration for both my new development board and production server (the only difference being my current production server uses a smaller 3x1TB array),
Processor: Intel Xeon E3-1270
Memory: 16GB ECC DDR3
Primary Disks: 2x128GB Samsung MZ7PC128HAFU SSD in a software RAID 1 configuration
Secondary Disks: 4x2TB Hardware RAID 5 configuration

My servers operating system and database server both run on a solid state drive at the moment. Along with that are some various optimizations I've made to the database server over time as well. Both my development and production board run using MariaDB with XtraDB.

But to list some more random things off the top of my head,

I have APC enabled on my board with the following configuration,

[APC]
apc.enabled=1
apc.shm_size=128M
apc.max_file_size=2M
apc.slam_defense=0

The default settings for APC caused a ~30% cache miss rate on my server. Raising the memory pool to 128MB easily gives me a 99.9% cache hit ratio with memory to spare.

You're also welcome to use my current my.cnf configuration for MariaDB as a reference. Just don't try and copy and paste anything without knowing what it does.

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
 
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
 
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
 
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
 
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
# lc-message-dir is unknown to MySQL 5.1
#lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 64M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size       = 128
tmp_table_size          = 756M
max_heap_table_size     = 756M
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover          = BACKUP
max_connections         = 512
table_cache             = 4096
table_open_cache        = 4096
table_definition_cache  = 4096
concurrent_insert       = 2
read_buffer_size        = 2M
sort_buffer_size = 4M
read_rnd_buffer_size    = 1M
join_buffer_size        = 4M
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size        = 0M
query_cache_type        = OFF
#
# * InnoDB Configuration
#
default_storage_engine  = InnoDB
# you can't just change log file size, requires special procedure
innodb_log_file_size   = 1G
innodb_buffer_pool_size = 4G
innodb_additional_mem_pool_size = 64M
innodb_log_buffer_size  = 32M
#innodb_thread_concurrency = 8
innodb_file_per_table   = 1
innodb_open_files       = 800
innodb_io_capacity      = 800
innodb_flush_log_at_trx_commit  = 0
innodb_flush_method     = O_DIRECT
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
log_error = /var/log/mysql.err
#general_log             = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#slow_query_log = 1
#slow_query_log_file = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
log_warnings = 2
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size         = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
 
 
 
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
 
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
 
[isamchk]
key_buffer = 16M
 
#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

On the operating system level, I still use Debian as my servers preferred distribution, but I have a slightly more up-to-date installation for both my servers. More or less just several small tweaks, such as installing using ext4 over Debian's default ext3, using the latest kernel from Debian backports, setting swappiness to 0, using noop for both the SSD and RAID 5 array, and tweaking the mount options for both performance and security. (The storage array, where uploaded files and attachments are stored, is mounted with noexec for example).

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/md1 during installation
UUID=bf526d3c-a28b-448d-91b1-bbec500e0ee3 /               ext4    noatime,discard,data=writeback,commit=100,errors=remount-ro 0 1
# /boot was on /dev/md0 during installation
UUID=43ac75aa-38be-4ea4-8e61-2680c6b850bb /boot           ext4    defaults        0       2
# RAID 5 Storage
UUID=07640f52-e2af-4602-bada-970842cbc107 /media/storage  ext4    noatime,nosuid,noexec,nodev,data=writeback,commit=100,barrier=0,errors=remount-ro  0  2

I also have the attachments class and IP.Downloads modified to use X-Accel-Redirect, so I'm not relying on PHP processes to handle file transfers. As both my site's are designed to process a lot of large concurrent transfers, having IP.Board pass them over to NGiNX to handle directly is far more efficient.

Though I guess my actual NGiNX configuration might be a more useful reference for you. The same above applies, just don't copy any thing from it without understanding what it does first, NGiNX is pretty well documented and straightforward anyways.

nginx.conf:

user  nginx;
worker_processes  6;
worker_priority   -5;

error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens on;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    tcp_nodelay     on;
    #tcp_nopush      off;

    open_file_cache max=750 inactive=30m;
    open_file_cache_valid    5m;
    open_file_cache_min_uses 3;
    open_file_cache_errors   on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Site configuration:

server {
    listen       [Redacted]:443 ssl spdy default_server;
    server_name  anime-social.com www.anime-social.com;
    root         [Redacted];
    index        index.php;
    ssl          on;
    ssl_session_cache     shared:SSL:10m;
    ssl_certificate       [Redacted]/main.crt;
    ssl_certificate_key   [Redacted]/main.key;
    spdy_headers_comp     1;

    # Basic web server configuration.
    client_max_body_size  15G;
    #tcp_nopush           off;
    access_log   off;
    error_log    [Redacted]/error.log crit;

    # Enabling gzip for files not processed by IPB.
    gzip  on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 3;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript application/xml application/xml+rss text/javascript application/javascript text/x-js;
    gzip_buffers 32 8k;
    gzip_disable "MSIE [1-6].(?!.*SV1)";

    # Google PageSpeed Configuration
    # This is a temporary workaround that ensures requests for pagespeed
    # optimized resources go to the pagespeed handler.
    #location ~ ".pagespeed.([a-z].)?[a-z]{2}.[^.]{10}.[^.]+" { }
    #location ~ "^/ngx_pagespeed_static/" { }
    #location ~ "^/ngx_pagespeed_beacon$" { }
    #pagespeed on;
    #pagespeed RewriteLevel CoreFilters;
    #pagespeed FileCachePath /var/cache/nginx/pagespeed_cache;
    #pagespeed EnableFilters collapse_whitespace,combine_css,defer_javascript,rewrite_css,fallback_rewrite_css_urls,lazyload_images,local_storage_cache,move_css_to_head,outline_css,outline_javascript,add_instrumentation;
    #pagespeed BeaconUrl /ngx_pagespeed_beacon;
    
    # Passing friendly URL rewrites
    location / {
        try_files  $uri $uri/ /index.php;
	add_header Strict-Transport-Security max-age=31536000;
        # rewrite  ^ /index.php? last;
    }

    location /nginx_status {
        stub_status on;
        allow [Redacted];
        deny all;
    }

    # Masked admin directory
    location ~^/admin/(.*)$ {
        deny     all;
    }

    # Process uploads from RAID 5 storage
    location ~^/uploads/(.*)$ {
        alias [Redacted]/uploads/$1;
        expires 1d;
    }

    # Process anime downloads from RAID 5 storage
    location ^~ /downloads/ {
        root [Redacted];
        internal;
    }

    # Process attachment downloads from RAID 5 storage
    location ~^/attachments/(.*)$ {
        alias [Redacted]/uploads/$1;
        internal;
    }

    # PHP execution security directives
    location ~^(/uploads/).*(.php)$ {
        deny     all;
    }
    location ~^(/hooks/).*(.php)$ {
        deny     all;
    }
    location ~^(/cache/).*(.php)$ {
        deny     all;
    }
    location ~^(/screenshots/).*(.php)$ {
        deny     all;
    }
    location ~^(/public/style_).*(.php)$ {
        deny     all;
    }

    # Caching, logging and other directives for static files.
    #location ~* ^(/uploads/profile/).*.(jpg|jpeg|gif|png)$ {
    #    access_log off;
    #    expires    1d;
    #}
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|htm|txt|swf|cur)$ {
        access_log off;
        expires    1w;
    }

    error_page  403              /403.html;

    # Leaving NGiNX defaults because I'm lazy.
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # Pass PHP scripts to php-fpm
    location ~ .php$ {
        fastcgi_pass   unix:/var/run/php-fpm/forum.sock;
        fastcgi_index  index.php;
        fastcgi_buffers 16 16k;
        fastcgi_buffer_size 32k;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        /etc/nginx/fastcgi_params;
    }
}

(Please also keep in mind this is the configuration for my development board. It is messy, unpolished and some settings are inconsistent and only applied for testing purposes.)

I have no secrets really. Just a bunch of studying/researching and time to kill :tongue:

Dmacleo · April 30, 2013

x-accel is something I was looking at too

Sull5 · April 30, 2013

ngx_pagespeed worked fine for me on my development forum but would stop functioning under heavy load. I don't know how much it's improved since I last tested it though.

I've made a lot of various optimizations and improvements over time. My largest forum right now processes 20-25TB of data transfer a month, and I ended up having to upgrade my hardware configuration after a while due to the disk strain on my servers mostly stock single software RAID 1 setup.

This is essentially my current hardware configuration for both my new development board and production server (the only difference being my current production server uses a smaller 3x1TB array),
Processor: Intel Xeon E3-1270
Memory: 16GB ECC DDR3
Primary Disks: 2x128GB Samsung MZ7PC128HAFU SSD in a software RAID 1 configuration
Secondary Disks: 4x2TB Hardware RAID 5 configuration

My servers operating system and database server both run on a solid state drive at the moment. Along with that are some various optimizations I've made to the database server over time as well. Both my development and production board run using MariaDB with XtraDB.

But to list some more random things off the top of my head,

I have APC enabled on my board with the following configuration,

[APC]
apc.enabled=1
apc.shm_size=128M
apc.max_file_size=2M
apc.slam_defense=0

The default settings for APC caused a ~30% cache miss rate on my server. Raising the memory pool to 128MB easily gives me a 99.9% cache hit ratio with memory to spare.

apc.jpg

You're also welcome to use my current my.cnf configuration for MariaDB as a reference. Just don't try and copy and paste anything without knowing what it does.

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
 
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
 
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
 
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
 
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
# lc-message-dir is unknown to MySQL 5.1
#lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 64M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size       = 128
tmp_table_size          = 756M
max_heap_table_size     = 756M
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover          = BACKUP
max_connections         = 512
table_cache             = 4096
table_open_cache        = 4096
table_definition_cache  = 4096
concurrent_insert       = 2
read_buffer_size        = 2M
sort_buffer_size = 4M
read_rnd_buffer_size    = 1M
join_buffer_size        = 4M
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size        = 0M
query_cache_type        = OFF
#
# * InnoDB Configuration
#
default_storage_engine  = InnoDB
# you can't just change log file size, requires special procedure
innodb_log_file_size   = 1G
innodb_buffer_pool_size = 4G
innodb_additional_mem_pool_size = 64M
innodb_log_buffer_size  = 32M
#innodb_thread_concurrency = 8
innodb_file_per_table   = 1
innodb_open_files       = 800
innodb_io_capacity      = 800
innodb_flush_log_at_trx_commit  = 0
innodb_flush_method     = O_DIRECT
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
log_error = /var/log/mysql.err
#general_log             = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#slow_query_log = 1
#slow_query_log_file = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
log_warnings = 2
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size         = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
 
 
 
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
 
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
 
[isamchk]
key_buffer = 16M
 
#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

On the operating system level, I still use Debian as my servers preferred distribution, but I have a slightly more up-to-date installation for both my servers. More or less just several small tweaks, such as installing using ext4 over Debian's default ext3, using the latest kernel from Debian backports, setting swappiness to 0, using noop for both the SSD and RAID 5 array, and tweaking the mount options for both performance and security. (The storage array, where uploaded files and attachments are stored, is mounted with noexec for example).

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/md1 during installation
UUID=bf526d3c-a28b-448d-91b1-bbec500e0ee3 /               ext4    noatime,discard,data=writeback,commit=100,errors=remount-ro 0 1
# /boot was on /dev/md0 during installation
UUID=43ac75aa-38be-4ea4-8e61-2680c6b850bb /boot           ext4    defaults        0       2
# RAID 5 Storage
UUID=07640f52-e2af-4602-bada-970842cbc107 /media/storage  ext4    noatime,nosuid,noexec,nodev,data=writeback,commit=100,barrier=0,errors=remount-ro  0  2

I also have the attachments class and IP.Downloads modified to use X-Accel-Redirect, so I'm not relying on PHP processes to handle file transfers. As both my site's are designed to process a lot of large concurrent transfers, having IP.Board pass them over to NGiNX to handle directly is far more efficient.

Though I guess my actual NGiNX configuration might be a more useful reference for you. The same above applies, just don't copy any thing from it without understanding what it does first, NGiNX is pretty well documented and straightforward anyways.

nginx.conf:

user  nginx;
worker_processes  6;
worker_priority   -5;

error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens on;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    tcp_nodelay     on;
    #tcp_nopush      off;

    open_file_cache max=750 inactive=30m;
    open_file_cache_valid    5m;
    open_file_cache_min_uses 3;
    open_file_cache_errors   on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Site configuration:

server {
    listen       [Redacted]:443 ssl spdy default_server;
    server_name  anime-social.com www.anime-social.com;
    root         [Redacted];
    index        index.php;
    ssl          on;
    ssl_session_cache     shared:SSL:10m;
    ssl_certificate       [Redacted]/main.crt;
    ssl_certificate_key   [Redacted]/main.key;
    spdy_headers_comp     1;

    # Basic web server configuration.
    client_max_body_size  15G;
    #tcp_nopush           off;
    access_log   off;
    error_log    [Redacted]/error.log crit;

    # Enabling gzip for files not processed by IPB.
    gzip  on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 3;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript application/xml application/xml+rss text/javascript application/javascript text/x-js;
    gzip_buffers 32 8k;
    gzip_disable "MSIE [1-6].(?!.*SV1)";

    # Google PageSpeed Configuration
    # This is a temporary workaround that ensures requests for pagespeed
    # optimized resources go to the pagespeed handler.
    #location ~ ".pagespeed.([a-z].)?[a-z]{2}.[^.]{10}.[^.]+" { }
    #location ~ "^/ngx_pagespeed_static/" { }
    #location ~ "^/ngx_pagespeed_beacon$" { }
    #pagespeed on;
    #pagespeed RewriteLevel CoreFilters;
    #pagespeed FileCachePath /var/cache/nginx/pagespeed_cache;
    #pagespeed EnableFilters collapse_whitespace,combine_css,defer_javascript,rewrite_css,fallback_rewrite_css_urls,lazyload_images,local_storage_cache,move_css_to_head,outline_css,outline_javascript,add_instrumentation;
    #pagespeed BeaconUrl /ngx_pagespeed_beacon;
    
    # Passing friendly URL rewrites
    location / {
        try_files  $uri $uri/ /index.php;
	add_header Strict-Transport-Security max-age=31536000;
        # rewrite  ^ /index.php? last;
    }

    location /nginx_status {
        stub_status on;
        allow [Redacted];
        deny all;
    }

    # Masked admin directory
    location ~^/admin/(.*)$ {
        deny     all;
    }

    # Process uploads from RAID 5 storage
    location ~^/uploads/(.*)$ {
        alias [Redacted]/uploads/$1;
        expires 1d;
    }

    # Process anime downloads from RAID 5 storage
    location ^~ /downloads/ {
        root [Redacted];
        internal;
    }

    # Process attachment downloads from RAID 5 storage
    location ~^/attachments/(.*)$ {
        alias [Redacted]/uploads/$1;
        internal;
    }

    # PHP execution security directives
    location ~^(/uploads/).*(.php)$ {
        deny     all;
    }
    location ~^(/hooks/).*(.php)$ {
        deny     all;
    }
    location ~^(/cache/).*(.php)$ {
        deny     all;
    }
    location ~^(/screenshots/).*(.php)$ {
        deny     all;
    }
    location ~^(/public/style_).*(.php)$ {
        deny     all;
    }

    # Caching, logging and other directives for static files.
    #location ~* ^(/uploads/profile/).*.(jpg|jpeg|gif|png)$ {
    #    access_log off;
    #    expires    1d;
    #}
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|htm|txt|swf|cur)$ {
        access_log off;
        expires    1w;
    }

    error_page  403              /403.html;

    # Leaving NGiNX defaults because I'm lazy.
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # Pass PHP scripts to php-fpm
    location ~ .php$ {
        fastcgi_pass   unix:/var/run/php-fpm/forum.sock;
        fastcgi_index  index.php;
        fastcgi_buffers 16 16k;
        fastcgi_buffer_size 32k;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        /etc/nginx/fastcgi_params;
    }
}

(Please also keep in mind this is the configuration for my development board. It is messy, unpolished and some settings are inconsistent and only applied for testing purposes.)

I have no secrets really. Just a bunch of studying/researching and time to kill :tongue:

Nice setup. I have similar spec'ed servers. I am running apc as well, but hey maybe you can assist me with an issue that has been boggling my brain. A while ago I wanted to switch from MySQL to MariaDB, it's basically a drop in replacement, though I was having a heck of a time get ssl to work.. even at all, I was and still am stuck with "have_ssl DISABLED" anyway because I couldn't get rid of this issue I checked out percona and I was able to get ssl working perfectly fine, same exact way I did with MySQL. A few days ago there was a Percona update that killed ssl for me, none of the client servers could connect and so I did some research and found this which appears to be the bug I'm experiencing, but I had already reinstalled MySQL and haven't had time to test if that was the case or not, anyhow I thought it would be a good time to try out mariadb again, I'm still facing the same issue, no matter what I do I cannot get ssl to work, I really do not understand why, it works perfect with MySQL 5.5 and it did up until the last update with percona, but no dice with mariadb, maybe I am configuring it wrong I don't know. I'm running Debian Squeeze, do you have any advice for me.

I've done a lot of research and no matter what I try seems to fail, all I know is the method I use to create certs etc works perfectly fine with MySQL 5.5 from dot deb repo, but no dice with mariadb :(

P.S, I'll be mulling over your configs soon, I slowly tinkered with my.cnf until I found optimum results for my forum, I'm curious to compare the differences with yours.

Dmacleo · May 2, 2013

gzip_http_version 1.1;

if using amazon cloudfront for cdn may need to change that to 1.0

Dmacleo · May 2, 2013

forgot to post the source for that

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html#CompressedNGINX

gzip_http_version: CloudFront sends requests in HTTP 1.0 format. In some versions of NGINX, the default value for the gzip_http_version setting is 1.1. If your version of NGINX includes this setting, change the value to 1.0.

I see this on nginx 1.2.8 which directadmin installs

I have not tried 1.4 yet

Hawking · May 4, 2013

Thank you @Kirito for the amazingly informative post!

In my case, yourdomain.com is actually a wordpress site, and my IP.board exists under yourdomain.com/community. I would really appreciate it if you can help me adapt the above configuration file to my setup.

Thank you one more time.

Dmacleo · May 4, 2013

if using amazon cloudfront for cdn may need to change that to 1.0

forgot to post the source for that

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html#CompressedNGINX

I see this on nginx 1.2.8 which directadmin installs

I have not tried 1.4 yet

fwiw I see this with 1.4 also.

Dmacleo · May 6, 2013

trying this on test vps with centos 6.4, phpinfo page works but getting 403 errors on admin/install that I cannot figure out. tried setting files/folders 777 as a test but did not help.

any thoughts?

where the phpinfo page worked I can't see it being a user chown issue

Makoto · May 6, 2013

trying this on test vps with centos 6.4, phpinfo page works but getting 403 errors on admin/install that I cannot figure out. tried setting files/folders 777 as a test but did not help.

any thoughts?

where the phpinfo page worked I can't see it being a user chown issue

You need to follow the instructions on renaming your admin directory prior to attempting an install.

If you really want to keep your admin directory as /admin, remove the following lines from your configuration,

   # Mask fake admin directory
    location ~^/admin/(.*)$ {
        deny     all;
    }
 
    # Secure real admin directory
    location ~^(/nimda/).*(.php) {
        #allow         127.0.0.1;
        #deny          all;
        #auth_basic    "Restricted Area";
        #auth_basic_user_file $document_root/nimda/.htpasswd;
        fastcgi_pass   unix:/var/run/php-fpm/ipboard.sock;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        /etc/nginx/fastcgi_params;
    }

Dmacleo · May 6, 2013

dang I thought I fixed that and set back to admin but looking now I didn't.

good catch, thanks.

Sull5 · May 7, 2013

running nginx 1.4.1 and pagespeed 1.5.27.2-beta now :) I think this version is suppose to fix those random lockups which I think I had one or two :(

Makoto · May 8, 2013

Thank you @Kirito for the amazingly informative post!

In my case, yourdomain.com is actually a wordpress site, and my IP.board exists under yourdomain.com/community. I would really appreciate it if you can help me adapt the above configuration file to my setup.

Thank you one more time.

Sorry for the slow response. In this instance you would have to merge the IP.Board configuration into your wordpress configuration.

It would probably be easier to use a subdomain for your community if possible, so you can use a separate configuration file for your IP.Board website.

(community.yourdomain.com instead of yourdomain.com/community)

Dmacleo · May 9, 2013

over last few days been seeing (in top) php-fpm for the forum jumping from 5 to 30% cpu a lot with not many users online. had been experimenting with on demand instead of dynamic.

have you messed around with these to see what works best for you?

I am on directadmin so paths may be different but the principles should be the same, wondering if you've tried all the process manager methods.

Makoto · May 9, 2013

I have. I use static on most all of my websites, as all of my servers are dedicated to running a single or small group of websites. There is no need for the added overhead of dynamic process spawning.

Static allows your website to run at full capacity at all times.

You shouldn't use on demand for a server dedicated to serving a single web application. It just adds an unnecessary overhead of needing to constantly re-spawn processes.

Dmacleo · May 9, 2013

looking at it today theres a high chance clamd was somehow causing it.

stopped that and my php-fpm down to 2% cpu with 100 or so online.

was looking towards static too so will look closer at that too

Invision Community 4: SEO, prepare for v5 and dormant account notifications

Invision Community 5: Beta testing and latest updates

Invision Community 4: A more professional report center

Invision Community 5: A video walkthrough creating a custom theme and homepage

Nginx+PHP-FPM secure server installation guide

Recommended Posts

Makoto

Deprecation Warning

Aussie Cable

Dmacleo

AlexJ

Makoto

Dmacleo

Dmacleo

Makoto

Dmacleo

Sull5

Makoto

Dmacleo

Sull5

Dmacleo

Dmacleo

Hawking

Dmacleo

Dmacleo

Makoto

Dmacleo

Sull5

Makoto

Dmacleo

Makoto

Dmacleo

Archived

Recently Browsing 0 members

More