Finding the cause of a slow forum

[HAIF]

I'm trying to figure out what is causing posting to be slow on my forum.

Right now some posts go through fine, and some take a few seconds to several minutes to complete.  No data is ever lost, but since some of the actions can take upward of six minutes, it's become something of an issue.

Here is the background:

• Recently moved to a new server.
• Does not appear to be a load issue since CPU rarely gets above 30%, bandwidth rarely gets above 1Mbps, and disk (SSD) is negligible.
• It's a dedicated server, not shared with any other site.
• Runs Ubuntu 16
• Affects any kind of posting function - posting messages, flagging a user spammer, personal messages, etc...
• Running the latest version of IPS, all nice and upgraded
• The ips4.php file says the server is capable of running IPS4.
• PHP: 7.0.32-0ubuntu0.16.04.1
• MySQL: 5.7.25-0ubuntu0.16.04.2 
• Already flushed the caches in the ACP Support section
• IPS error log: empty
• MySQL error log: empty
• IPS System check: all green

In investigating, I found a couple of small problems, but nothing that (IMO) would account for such long posting delays.  However, there is this:

• The Apache error log shows this a number of times: PHP Fatal error:  Uncaught IPS\\Db\\Exception: No such file or directory in /var/www/html/haif/system/Db/Db.php:205\nStack trace:\n#0 /var/www/html/haif/system/Db/Db.php(118): IPS\\_Db::_establishConnection('IPS\\\\Db', Array, true)\n#1 /var/www/html/haif/system/Patterns/ActiveRecord.php(63): IPS\\_Db::i()\n#2 /var/www/html/haif/system/Patterns/ActiveRecord.php(179): IPS\\Patterns\\_ActiveRecord::db()\n#3 /var/www/html/haif/system/Patterns/ActiveRecord.php(110): IPS\\Patterns\\_ActiveRecord::constructLoadQuery(NULL, 'g_id', NULL)\n#4 /var/www/html/haif/system/Member/Group.php(111): IPS\\Patterns\\_ActiveRecord::load(NULL, 'g_id', NULL)\n#5 /var/www/html/haif/system/Session/Store/Redis.php(55): IPS\\Member\\_Group::load(NULL)\n#6 /var/www/html/haif/system/Session/Front.php(283): IPS\\Session\\Store\\_Redis->updateSession(Array)\n#7 [internal function]: IPS\\Session\\_Front->write('r2tff1ejuij567h...', '')\n#8 /var/www/html/haif/system/Redis/Redis.php(184): session_write_close()\n#9 [internal function]: IPS\\_Redis->IPS\\{closure}(Object(IPS\\_Redis))\n#10 {main in /var/www/html/haif/system/Db/Db.php on line 205, referer: https://www.google.com/
• And the IPS system log shows a bunch of these: UnexpectedValueException: Missing: /applications/gallery/Application.php (0)
  #0 /var/www/html/haif/system/Patterns/ActiveRecord.php(120): IPS\_Application::constructFromData(Array)
  #1 /var/www/html/haif/system/Application/Application.php(229): IPS\Patterns\_ActiveRecord::load('gallery', 'app_directory', NULL)
  #2 /var/www/html/haif/system/Dispatcher/Standard.php(159): IPS\_Application::load('gallery')
  #3 /var/www/html/haif/system/Dispatcher/Front.php(102): IPS\Dispatcher\_Standard->init()
  #4 /var/www/html/haif/system/Dispatcher/Dispatcher.php(103): IPS\Dispatcher\_Front->init()
  #5 /var/www/html/haif/index.php(13): IPS\_Dispatcher::i()
  #6 {main}

One thought I have is that since this is a fresh new server, perhaps all of the necessary mod_xxx items aren't installed.  I couldn't find a list of the Apache mods that IPS requires.

Any thoughts on how I can track down and correct the problem?

 

[opentype]

11 minutes ago, HAIF said:

Right now some posts go through fine, and some take a few seconds to several minutes to complete.

Is there a difference between them? For example: number of followers? If so, I would guess sending out emails is the problem. 

[HAIF]

17 hours ago, opentype said:

Is there a difference between them? For example: number of followers? If so, I would guess sending out emails is the problem. 

There doesn't seem to be any correlation between slow messages that I can see.  I don't think it's a followers thing because it affects new topics, replies, and even private messages between just two people.

Everything seem OK when it comes to e-mail.  The IPS e-mail error log is empty, and I get notifications when people send me private messages.

[Martin A.]

When we restored our test installation with a recent version of the live site, we encountered a similar error as your first one. I "solved" this by defaulting the 'CACHE_METHOD' and 'STORE_METHOD' to the default values (just remove or comment the lines from constants.php), then set it up again from the ACP to get the new constants.php file. I am not sure why this worked, but it's worth a shot. Also worth to keep it at the default values to see if this makes things better. Could also be that you've forgotten to install/start Redis.

The second error is due to a missing application directory/file. Make sure /applications/gallery/Application.php exists, and that it and its related folders have the correct file permissions.

And as this is a new server, it might be worth setting it up with MariaDB, nginx and php-fpm right away. And you should choose a PHP version greater than 7.0, as that was EOLed in December 2018, and IPS4.4 will require PHP 7.1.

[bfarber]

You might want to check if tasks are being ran with activity, or via cron. This might explain random "page loaded slow" issues.

[HAIF]

On 1/28/2019 at 7:42 AM, bfarber said:

You might want to check if tasks are being ran with activity, or via cron. This might explain random "page loaded slow" issues.

Thanks for the suggestion.  I chose to try that first because cronjobs I understand.  Hopefully that will do the trick.

[HAIF]

So that didn't work.  I still have the problem.  I took a look at the Apache logs and there are a bunch of these:

[Sat Feb 23 01:30:00.417718 2019] [core:error] [pid 19601] [client 207.46.13.146:18048] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

Any thoughts on this?

I looked at my server load once again during one of these snoozes, and everything looks fine to me.

top - 01:38:05 up 27 days, 21:18,  2 users,  load average: 0.47, 0.44, 0.43
Tasks: 135 total,   2 running, 133 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11.7 us,  0.8 sy,  0.0 ni, 87.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  4046376 total,   297416 free,   564692 used,  3184268 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  3128260 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23321 www-data  20   0  430136  38252  20428 S   8.0  0.9   0:00.64 apache2
 4411 mysql     20   0 1586756 337028   7380 S   4.7  8.3   1876:18 mysqld
23349 www-data  20   0  431848  38792  19268 S   2.3  1.0   0:00.34 apache2
23342 www-data  20   0  429992  40244  22620 S   2.0  1.0   0:00.52 apache2
23348 www-data  20   0  423760  31576  20100 S   1.3  0.8   0:00.34 apache2
23340 www-data  20   0  425636  35396  22052 S   1.0  0.9   0:00.47 apache2
 1428 redis     20   0  298876  23196   3436 S   0.7  0.6 223:48.04 redis-server
23320 www-data  20   0  424080  35820  23952 S   0.7  0.9   0:01.07 apache2
23346 www-data  20   0  423852  32296  20760 S   0.7  0.8   0:00.79 apache2
23352 www-data  20   0  421424  24276  15064 S   0.7  0.6   0:00.05 apache2
23354 www-data  20   0  421308  24092  15084 S   0.7  0.6   0:00.04 apache2
23356 www-data  20   0  425668  32676  19268 S   0.7  0.8   0:00.24 apache2
23360 root      20   0   92680   6540   5644 S   0.7  0.2   0:00.02 sshd
    1 root      20   0  185176   5788   4024 S   0.0  0.1   0:42.76 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.03 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   3:27.07 ksoftirqd/0

 

The only squirrel I can se is that redis-server.  Should it be running that long?  Could that be what's stalled and causing problems?

[HAIF]

On 1/27/2019 at 12:46 AM, Martin A. said:

The second error is due to a missing application directory/file. Make sure /applications/gallery/Application.php exists, and that it and its related folders have the correct file permissions.

You are correct, there is no applications/gallery directory.  But there isn't one in the latest IPB download, either.  So I'm not sure why it's looking for the gallery, or how to replace those missing files.  I checked the ACP for a gallery plugin, and there are no plugins at all.  This is a very old Invision board (from the early 2000's), so I wonder if it's some cruft from all of the upgrades over the years.  Perhaps the previous owner had a gallery plug-in that the forum is still looking for.

[bfarber]

On 2/22/2019 at 8:39 PM, HAIF said:

So that didn't work.  I still have the problem.  I took a look at the Apache logs and there are a bunch of these:

[Sat Feb 23 01:30:00.417718 2019] [core:error] [pid 19601] [client 207.46.13.146:18048] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

Any thoughts on this?

I looked at my server load once again during one of these snoozes, and everything looks fine to me.

top - 01:38:05 up 27 days, 21:18,  2 users,  load average: 0.47, 0.44, 0.43
Tasks: 135 total,   2 running, 133 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11.7 us,  0.8 sy,  0.0 ni, 87.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  4046376 total,   297416 free,   564692 used,  3184268 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  3128260 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23321 www-data  20   0  430136  38252  20428 S   8.0  0.9   0:00.64 apache2
 4411 mysql     20   0 1586756 337028   7380 S   4.7  8.3   1876:18 mysqld
23349 www-data  20   0  431848  38792  19268 S   2.3  1.0   0:00.34 apache2
23342 www-data  20   0  429992  40244  22620 S   2.0  1.0   0:00.52 apache2
23348 www-data  20   0  423760  31576  20100 S   1.3  0.8   0:00.34 apache2
23340 www-data  20   0  425636  35396  22052 S   1.0  0.9   0:00.47 apache2
 1428 redis     20   0  298876  23196   3436 S   0.7  0.6 223:48.04 redis-server
23320 www-data  20   0  424080  35820  23952 S   0.7  0.9   0:01.07 apache2
23346 www-data  20   0  423852  32296  20760 S   0.7  0.8   0:00.79 apache2
23352 www-data  20   0  421424  24276  15064 S   0.7  0.6   0:00.05 apache2
23354 www-data  20   0  421308  24092  15084 S   0.7  0.6   0:00.04 apache2
23356 www-data  20   0  425668  32676  19268 S   0.7  0.8   0:00.24 apache2
23360 root      20   0   92680   6540   5644 S   0.7  0.2   0:00.02 sshd
    1 root      20   0  185176   5788   4024 S   0.0  0.1   0:42.76 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.03 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   3:27.07 ksoftirqd/0

 

The only squirrel I can se is that redis-server.  Should it be running that long?  Could that be what's stalled and causing problems?

I would enable the LogLevel debug it references. I don't know what that error is in reference to off-hand.

On 2/22/2019 at 8:47 PM, HAIF said:

You are correct, there is no applications/gallery directory.  But there isn't one in the latest IPB download, either.  So I'm not sure why it's looking for the gallery, or how to replace those missing files.  I checked the ACP for a gallery plugin, and there are no plugins at all.  This is a very old Invision board (from the early 2000's), so I wonder if it's some cruft from all of the upgrades over the years.  Perhaps the previous owner had a gallery plug-in that the forum is still looking for.

It's only present if you have an active license for Gallery. It may have previously been installed (and thus a record in the database), but is no longer present because you cancelled Gallery, which would result in the notice you previously mentioned. You can ignore it and/or just remove Gallery completely.