Jump to content

New IPB Behavior Causes High Load

Recommended Posts

Well, it's more complicated than that, but I can't come up with a good title.

So. This past Sunday, our forum at http://asoiaf.westeros.org/ received unprecedented levels of traffic thanks to the latest episode of the TV show based on the novels our site is about. Big, big load. And for the most part, our set up -- lighttpd+php-fpm, with Varnish set up for guests -- handled it okay. Note the server also hosts a popular Mediawiki which draws more visitors than the forum does.

But after Sunday... things got weird. I'm going to quote our valiant server host and admin:

The primary issue at present is that PHP-FPM is bumping up against memory limits; it's demonstrably the issue, as it's logging the fact. If I turn up PHP-FPM's memory allocation, MySQL starts swapping things out and causing queries to run really slowly, which causes PHP-FPM to take forever to resolve searches and thus causes PHP-FPM to run out of available helper processes. Worse, in some cases PHP-FPM seems not to be recovering. If I dump PHP to caching to disk, we don't have the helper process starvation issue but the site runs reaaaaaaaally sloooooooow.

The obvious question at this point is why PHP-FPM has suddenly become memory-bound, when it had not been for weeks beforehand; that's the root issue from which all others have been springing. (Save the stale Varnish cache, which was just a ruleset we'd never triggered before.) Something has clearly changed in the way Invision is doing things — and it's definitely the board, not the wiki — but after several days of trying to track down what has changed in how Invision's doing things — including literally rolling back every memory file to Sunday's copy to see if that improved things — I'm ready to just take the easy way out and buy some breathing room by throwing memory at PHP-FPM.

So... any ideas as to what might have happened to start leading to this memory-chewing problem? I've done no installs of mods or upgrades in the intervening time, so it's not that.

The only thing I can think of is that the database has simply gotten large enough that it takes too long for IPB to deal with it, which leads to a cascade effect as PHP processes pile up waiting for MySQL queries to finish. I can certainly prune the forum significantly -- it's long over due -- but if there are other ideas, I'd be interested in hearing them.

The obvious fix right now is to get more memory installed (from 2GB to 4GB) but it bothers me that we don't really know why this is suddenly necessary.
Link to comment
Share on other sites

I'm afraid I'm not sure what more information might be useful! I guess the thing is that it has been enough for weeks, and suddenly... it isn't.

That doesn't sound like normal behavior. Do we blame gremlins? We hadn't changed anything about the site configuration from one day to the next, so far as we know. Even yesterday, our slowest day where traffic simply wasn't all that high, we had issues with having to kick PHP-FPM every once in awhile because of whatever it is the forum is doing.

That said, we're planning to upgrade the RAM. Just wanted to see if there's any idea for what could be causing the memory death-spiral.

As a random stab in the dark regarding configuration information, here's some of our output from the APC regarding configuration stuff. Afraid I'm not privy to the php-fpm conf file so I'm not sure what's set on that:

PHP Version     5.3.10 (fpm-fcgi) ( PHP INFO )

Disabled PHP Functions     No Information

Loaded Extensions     Core, PDO, Phar, Reflection, SPL, SimpleXML, apc, bz2, calendar, cgi-fcgi, ctype, curl, date, dom, ereg, exif, fileinfo, filter, ftp, gd, gettext, gmp, hash, iconv, json, ldap, libxml, mysql, mysqli, openssl, pcre, pdo_mysql, pdo_sqlite, session, shmop, sockets, standard, tokenizer, wddx, xml, xmlreader, xmlwriter, xsl, zip, zlib


Directive    Local Value    Master Value

allow_call_time_pass_reference    Off    Off

allow_url_fopen    On    On

allow_url_include    On    On

always_populate_raw_post_data    Off    Off

arg_separator.input    &    &

arg_separator.output    &    &

asp_tags    Off    Off

auto_append_file    no value    no value

auto_globals_jit    On    On

auto_prepend_file    no value    no value

browscap    no value    no value

default_charset    no value    no value

default_mimetype    text/html    text/html

define_syslog_variables    Off    Off

disable_classes    no value    no value

disable_functions    no value    no value

display_errors    Off    Off

display_startup_errors    Off    Off

doc_root    no value    no value

docref_ext    no value    no value

docref_root    no value    no value

enable_dl    On    On

error_append_string    no value    no value

error_log    /var/log/php-fpm/www-error.log    /var/log/php-fpm/www-error.log

error_prepend_string    no value    no value

error_reporting    6983    30719

exit_on_timeout    Off    Off

expose_php    On    On

extension_dir    /usr/lib/php/modules    /usr/lib/php/modules

file_uploads    On    On

highlight.bg    #FFFFFF    #FFFFFF

highlight.comment    #FF8000    #FF8000

highlight.default    #0000BB    #0000BB

highlight.html    #000000    #000000

highlight.keyword    #007700    #007700

highlight.string    #DD0000    #DD0000

html_errors    On    On

ignore_repeated_errors    Off    Off

ignore_repeated_source    Off    Off

ignore_user_abort    Off    Off

implicit_flush    Off    Off

include_path    .:/usr/share/pear:/usr/share/php:/home/sites/westeros.org/forum_web/ips_kernel/    .:/usr/share/pear:/usr/share/php

log_errors    Off    Off

log_errors_max_len    1024    1024

magic_quotes_gpc    Off    Off

magic_quotes_runtime    Off    Off

magic_quotes_sybase    Off    Off

mail.add_x_header    Off    Off

mail.force_extra_parameters    no value    no value

mail.log    no value    no value

max_execution_time    0    30

max_file_uploads    20    20

max_input_nesting_level    64    64

max_input_time    60    60

max_input_vars    1000    1000

memory_limit    32M    32M

open_basedir    no value    no value

output_buffering    4096    4096

output_handler    no value    no value

post_max_size    8M    8M

precision    14    14

realpath_cache_size    16K    16K

realpath_cache_ttl    120    120

register_argc_argv    Off    Off

register_globals    Off    Off

register_long_arrays    Off    Off

report_memleaks    On    On

report_zend_debug    On    On

request_order    no value    no value

safe_mode    Off    Off

safe_mode_exec_dir    no value    no value

safe_mode_gid    Off    Off

safe_mode_include_dir    no value    no value

sendmail_from    no value    no value

sendmail_path    /usr/sbin/sendmail -t -i    /usr/sbin/sendmail -t -i

serialize_precision    100    100

short_open_tag    On    On

SMTP    localhost    localhost

smtp_port    25    25

sql.safe_mode    Off    Off

track_errors    Off    Off

unserialize_callback_func    no value    no value

upload_max_filesize    2M    2M

upload_tmp_dir    no value    no value

user_dir    no value    no value

user_ini.cache_ttl    300    300

user_ini.filename    .user.ini    .user.ini

variables_order    EGPCS    EGPCS

xmlrpc_error_number    0    0

xmlrpc_errors    Off    Off

y2k_compliance    On    On

zend.enable_gc    On    On


MySQL Support    enabled

Active Persistent Links     0

Active Links     0

Client API version     5.0.86

MYSQL_MODULE_TYPE     external

MYSQL_SOCKET     /var/lib/mysql/mysql.sock

MYSQL_INCLUDE     -I/usr/include/mysql

MYSQL_LIBS     -L/usr/lib/mysql -lmysqlclient

Directive    Local Value    Master Value

mysql.allow_local_infile    On    On

mysql.allow_persistent    On    On

mysql.connect_timeout    60    60

mysql.default_host    no value    no value

mysql.default_password    no value    no value

mysql.default_port    no value    no value

mysql.default_socket    /var/lib/mysql/mysql.sock    /var/lib/mysql/mysql.sock

mysql.default_user    no value    no value

mysql.max_links    Unlimited    Unlimited

mysql.max_persistent    Unlimited    Unlimited

mysql.trace_mode    Off    Off


MysqlI Support    enabled

Client API library version     5.0.86

Active Persistent Links     0

Inactive Persistent Links     0

Active Links     1

Client API header version     5.0.77

MYSQLI_SOCKET     /var/lib/mysql/mysql.sock

Directive    Local Value    Master Value

mysqli.allow_local_infile    On    On

mysqli.allow_persistent    On    On

mysqli.default_host    no value    no value

mysqli.default_port    3306    3306

mysqli.default_pw    no value    no value

mysqli.default_socket    no value    no value

mysqli.default_user    no value    no value

mysqli.max_links    Unlimited    Unlimited

mysqli.max_persistent    Unlimited    Unlimited

mysqli.reconnect    Off    Off

Link to comment
Share on other sites

Oh, I forgot to update.... but I'll say you were looking in the right direction, Luis.

We decided to move to a brand new server (a much newer, more powerful one with 8GB of RAM)... and in the course of the move, our admin discovered that the disk controller was shot. Not so much that it failed completely, but under heavy load it would start repeating paging actions, leading to massive slow down of MySQL, which would then cascade through to PHP-FPM and then the memory would get eaten up by processes, start the process of off-loading to an already crippled disk...

So, that was what happened. That Sunday evening was the highest level of traffic we ever had, and it seems that it was too much for the server.

So, all solved.

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...