Linda A Posted June 2, 2012 Posted June 2, 2012 Well, it's more complicated than that, but I can't come up with a good title. So. This past Sunday, our forum at http://asoiaf.westeros.org/ received unprecedented levels of traffic thanks to the latest episode of the TV show based on the novels our site is about. Big, big load. And for the most part, our set up -- lighttpd+php-fpm, with Varnish set up for guests -- handled it okay. Note the server also hosts a popular Mediawiki which draws more visitors than the forum does. But after Sunday... things got weird. I'm going to quote our valiant server host and admin: The primary issue at present is that PHP-FPM is bumping up against memory limits; it's demonstrably the issue, as it's logging the fact. If I turn up PHP-FPM's memory allocation, MySQL starts swapping things out and causing queries to run really slowly, which causes PHP-FPM to take forever to resolve searches and thus causes PHP-FPM to run out of available helper processes. Worse, in some cases PHP-FPM seems not to be recovering. If I dump PHP to caching to disk, we don't have the helper process starvation issue but the site runs reaaaaaaaally sloooooooow. The obvious question at this point is why PHP-FPM has suddenly become memory-bound, when it had not been for weeks beforehand; that's the root issue from which all others have been springing. (Save the stale Varnish cache, which was just a ruleset we'd never triggered before.) Something has clearly changed in the way Invision is doing things — and it's definitely the board, not the wiki — but after several days of trying to track down what has changed in how Invision's doing things — including literally rolling back every memory file to Sunday's copy to see if that improved things — I'm ready to just take the easy way out and buy some breathing room by throwing memory at PHP-FPM. So... any ideas as to what might have happened to start leading to this memory-chewing problem? I've done no installs of mods or upgrades in the intervening time, so it's not that. The only thing I can think of is that the database has simply gotten large enough that it takes too long for IPB to deal with it, which leads to a cascade effect as PHP processes pile up waiting for MySQL queries to finish. I can certainly prune the forum significantly -- it's long over due -- but if there are other ideas, I'd be interested in hearing them. The obvious fix right now is to get more memory installed (from 2GB to 4GB) but it bothers me that we don't really know why this is suddenly necessary.
Rhett Posted June 2, 2012 Posted June 2, 2012 Depending on what your idea of very heavy traffic is, 2gb of memory is most likely not enough. There isn't enough information in your post to be able to provide you with anything but guesses however.
3DKiwi Posted June 3, 2012 Posted June 3, 2012 Ditto. 2gb for a dedicated server is not enough. You need a minimum of 3gb for a medium sized forum / site. 3DKiwi
Linda A Posted June 3, 2012 Author Posted June 3, 2012 I'm afraid I'm not sure what more information might be useful! I guess the thing is that it has been enough for weeks, and suddenly... it isn't. That doesn't sound like normal behavior. Do we blame gremlins? We hadn't changed anything about the site configuration from one day to the next, so far as we know. Even yesterday, our slowest day where traffic simply wasn't all that high, we had issues with having to kick PHP-FPM every once in awhile because of whatever it is the forum is doing. That said, we're planning to upgrade the RAM. Just wanted to see if there's any idea for what could be causing the memory death-spiral. As a random stab in the dark regarding configuration information, here's some of our output from the APC regarding configuration stuff. Afraid I'm not privy to the php-fpm conf file so I'm not sure what's set on that:PHP Version 5.3.10 (fpm-fcgi) ( PHP INFO ) Disabled PHP Functions No Information Loaded Extensions Core, PDO, Phar, Reflection, SPL, SimpleXML, apc, bz2, calendar, cgi-fcgi, ctype, curl, date, dom, ereg, exif, fileinfo, filter, ftp, gd, gettext, gmp, hash, iconv, json, ldap, libxml, mysql, mysqli, openssl, pcre, pdo_mysql, pdo_sqlite, session, shmop, sockets, standard, tokenizer, wddx, xml, xmlreader, xmlwriter, xsl, zip, zlib PHP Directive Local Value Master Value allow_call_time_pass_reference Off Off allow_url_fopen On On allow_url_include On On always_populate_raw_post_data Off Off arg_separator.input & & arg_separator.output & & asp_tags Off Off auto_append_file no value no value auto_globals_jit On On auto_prepend_file no value no value browscap no value no value default_charset no value no value default_mimetype text/html text/html define_syslog_variables Off Off disable_classes no value no value disable_functions no value no value display_errors Off Off display_startup_errors Off Off doc_root no value no value docref_ext no value no value docref_root no value no value enable_dl On On error_append_string no value no value error_log /var/log/php-fpm/www-error.log /var/log/php-fpm/www-error.log error_prepend_string no value no value error_reporting 6983 30719 exit_on_timeout Off Off expose_php On On extension_dir /usr/lib/php/modules /usr/lib/php/modules file_uploads On On highlight.bg #FFFFFF #FFFFFF highlight.comment #FF8000 #FF8000 highlight.default #0000BB #0000BB highlight.html #000000 #000000 highlight.keyword #007700 #007700 highlight.string #DD0000 #DD0000 html_errors On On ignore_repeated_errors Off Off ignore_repeated_source Off Off ignore_user_abort Off Off implicit_flush Off Off include_path .:/usr/share/pear:/usr/share/php:/home/sites/westeros.org/forum_web/ips_kernel/ .:/usr/share/pear:/usr/share/php log_errors Off Off log_errors_max_len 1024 1024 magic_quotes_gpc Off Off magic_quotes_runtime Off Off magic_quotes_sybase Off Off mail.add_x_header Off Off mail.force_extra_parameters no value no value mail.log no value no value max_execution_time 0 30 max_file_uploads 20 20 max_input_nesting_level 64 64 max_input_time 60 60 max_input_vars 1000 1000 memory_limit 32M 32M open_basedir no value no value output_buffering 4096 4096 output_handler no value no value post_max_size 8M 8M precision 14 14 realpath_cache_size 16K 16K realpath_cache_ttl 120 120 register_argc_argv Off Off register_globals Off Off register_long_arrays Off Off report_memleaks On On report_zend_debug On On request_order no value no value safe_mode Off Off safe_mode_exec_dir no value no value safe_mode_gid Off Off safe_mode_include_dir no value no value sendmail_from no value no value sendmail_path /usr/sbin/sendmail -t -i /usr/sbin/sendmail -t -i serialize_precision 100 100 short_open_tag On On SMTP localhost localhost smtp_port 25 25 sql.safe_mode Off Off track_errors Off Off unserialize_callback_func no value no value upload_max_filesize 2M 2M upload_tmp_dir no value no value user_dir no value no value user_ini.cache_ttl 300 300 user_ini.filename .user.ini .user.ini variables_order EGPCS EGPCS xmlrpc_error_number 0 0 xmlrpc_errors Off Off y2k_compliance On On zend.enable_gc On On mysql MySQL Support enabled Active Persistent Links 0 Active Links 0 Client API version 5.0.86 MYSQL_MODULE_TYPE external MYSQL_SOCKET /var/lib/mysql/mysql.sock MYSQL_INCLUDE -I/usr/include/mysql MYSQL_LIBS -L/usr/lib/mysql -lmysqlclient Directive Local Value Master Value mysql.allow_local_infile On On mysql.allow_persistent On On mysql.connect_timeout 60 60 mysql.default_host no value no value mysql.default_password no value no value mysql.default_port no value no value mysql.default_socket /var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock mysql.default_user no value no value mysql.max_links Unlimited Unlimited mysql.max_persistent Unlimited Unlimited mysql.trace_mode Off Off mysqli MysqlI Support enabled Client API library version 5.0.86 Active Persistent Links 0 Inactive Persistent Links 0 Active Links 1 Client API header version 5.0.77 MYSQLI_SOCKET /var/lib/mysql/mysql.sock Directive Local Value Master Value mysqli.allow_local_infile On On mysqli.allow_persistent On On mysqli.default_host no value no value mysqli.default_port 3306 3306 mysqli.default_pw no value no value mysqli.default_socket no value no value mysqli.default_user no value no value mysqli.max_links Unlimited Unlimited mysqli.max_persistent Unlimited Unlimited mysqli.reconnect Off Off
AlexJ Posted June 3, 2012 Posted June 3, 2012 Are u using any kind of php cache system? Like xcache, apc or anything?
Luis Manson Posted June 5, 2012 Posted June 5, 2012 how is your IO ? what kind of hard drives do you have?
Linda A Posted June 5, 2012 Author Posted June 5, 2012 Oh, I forgot to update.... but I'll say you were looking in the right direction, Luis. We decided to move to a brand new server (a much newer, more powerful one with 8GB of RAM)... and in the course of the move, our admin discovered that the disk controller was shot. Not so much that it failed completely, but under heavy load it would start repeating paging actions, leading to massive slow down of MySQL, which would then cascade through to PHP-FPM and then the memory would get eaten up by processes, start the process of off-loading to an already crippled disk... So, that was what happened. That Sunday evening was the highest level of traffic we ever had, and it seems that it was too much for the server. So, all solved.
Luis Manson Posted June 7, 2012 Posted June 7, 2012 thanks for reporting any ideas on how did hi found that?
Recommended Posts
Archived
This topic is now archived and is closed to further replies.