Website crippled every once in a while with template errors...not sure why

[LiquidFractal]

Hello,

Every so often I will log in to my test site (and my production site, although this definitely happens less often there) to find the following:

So, perhaps obviously, I can't log in to the adminCP when this happens to clear caches or switch themes.  So far as I can tell it also happens randomly (at least I haven't been able to discern a pattern to this).  This happened when I tried today; I did not perform any updates or anything the day before.

The Technical Details are as follows:

OutOfRangeException:  (0)
#0 /XXXX/system/Member/Member.php(804): IPS\Patterns\_ActiveRecord::load()
#1 /XXXX/init.php(927) : eval()'d code(30): IPS\_Member->get_group()
#2 /XXXX/system/Patterns/ActiveRecord.php(335): IPS\videos_hook_memberSource->get_group()
#3 /XXXX/system/Session/Front.php(255): IPS\Patterns\_ActiveRecord->__get()
#4 [internal function]: IPS\Session\_Front->read()
#5 /XXXX/system/Session/Session.php(94): session_start()
#6 /XXXX/system/Theme/Theme.php(417): IPS\_Session::i()
#7 /XXXX/system/Theme/Theme.php(312): IPS\_Theme::currentThemeId()
#8 /XXXX/system/Dispatcher/Standard.php(54): IPS\_Theme::i()
#9 /XXXX/init.php(927) : eval()'d code(38): IPS\Dispatcher\_Standard::baseCss()
#10 /XXXX/init.php(927) : eval()'d code(43): IPS\Dispatcher\forumcover_hook_includeJSandCSS::baseCss()
#11 /XXXX/init.php(927) : eval()'d code(17): IPS\Dispatcher\slidermaker_hook_includeJSandCSS::baseCss()
#12 /XXXX/init.php(927) : eval()'d code(35): IPS\Dispatcher\notices_hook_dispatcher_standard::baseCss()
#13 /XXXX/system/Dispatcher/Front.php(759): IPS\Dispatcher\testm_hook_includeJSandCSS::baseCss()
#14 /XXXX/init.php(927) : eval()'d code(40): IPS\Dispatcher\_Front::baseCss()
#15 /XXXX/init.php(927) : eval()'d code(66): IPS\Dispatcher\easypopup_hook_includeJSandCSS::baseCss()
#16 /XXXX/system/Dispatcher/Front.php(76): IPS\Dispatcher\stickynotes_hook_Outputs::baseCss()
#17 /XXXX/init.php(927) : eval()'d code(23): IPS\Dispatcher\_Front->init()
#18 /XXXX/init.php(927) : eval()'d code(20): IPS\Dispatcher\stickynotes_hook_Outputs->init()
#19 /XXXX/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\dp45guestviewlimits_hook_dp45guestViewLimitsDispatcherFro->init()
#20 /XXXX/index.php(13): IPS\_Dispatcher::i()
#21 {main}

When this happens, my only recourse is to reinstall a backup.

I can definitely say that my custom theme was the most recent version.  Apart from that, well, unless I restore a backup I can't tell re: the other 3rd-party apps/plugins mentioned (although I know for sure that several of them were already updated to be 4.7.x compatible).

For what it's worth, on both sites I'm running IPB 4.7.4 with PHP 8.1.11 and Version 10.6.10-MariaDB.  Importantly, though, this has happened on occasion with previous versions of IPB, PHP, and MariaDB.

I was hoping someone could help diagnose this issue.  Thanks!

[Marc Stridgen]

It looks like there is an issue with the videos application, looking at the error message there. I would suggest disabling that first of all. Other than that, I would need to see the issue while its in place.

[LiquidFractal]

8 hours ago, Marc Stridgen said:

It looks like there is an issue with the videos application, looking at the error message there. I would suggest disabling that first of all. Other than that, I would need to see the issue while its in place.

Marc, thanks very much for your response.  This is a bit of a pain, since both sites tick along nicely and then this happens all of a sudden...so there's no way to predict if/when it will happen.  But I will look in to disabling it on my test site and see if that has any long-term impact.  Thanks again!

[Jim M]

8 minutes ago, LiquidFractal said:

Marc, thanks very much for your response.  This is a bit of a pain, since both sites tick along nicely and then this happens all of a sudden...so there's no way to predict if/when it will happen.  But I will look in to disabling it on my test site and see if that has any long-term impact.  Thanks again!

There are a few other third party applications/plugins mentioned there in the stack trace which may need to be disabled as well. It is hard to say which one is the culprit, though Videos is the furtherest one up so a good place to start. However, if you don't run into this again with all of them disabled, you would want to turn them on one-by-one till you do find the issue.

[Randy Calvert]

The other thing to consider is that there are conditions that happen that cause a conflict between two resources as well. It may be fine when working by itself, but under the right circumstances...  two of the above might be stepping on each other.

[LiquidFractal]

What perplexes me is the fact that these incidents happen at long intervals, and thus can't be predicted (at least not by any means I have).  I'm not "blaming" Invision software at all when I speculate this, but is it possible that IPB internal operations somehow eventually clash with one or more of these apps over time due simply to IPB doing what it's supposed to be doing?

Edited November 17, 2022 by LiquidFractal

[Randy Calvert]

Also...  earlier you mentioned..

Quote

When this happens, my only recourse is to reinstall a backup.

When that happened, before restoring did you try enabling recovery mode to get back online?

[opentype]

17 hours ago, LiquidFractal said:

perhaps obviously, I can't log in to the adminCP when this happens to clear caches or switch themes. 

Next time, also try clearing the datastore directory via FTP. That’s usually my magic bullet against such random/complete failures. 

[Mark H]

Just a followup note.... if you clear the datastore folder, be sure to leave the (blank) index.html in place.

[LiquidFractal]

5 hours ago, opentype said:

Next time, also try clearing the datastore directory via FTP. That’s usually my magic bullet against such random/complete failures. 

This is a great bit of advice.  Thanks for this!  I presume this is the equivalent of clearing the cache?  Or is it something else?

Edited November 18, 2022 by LiquidFractal

[Marc Stridgen]

9 hours ago, LiquidFractal said:

This is a great bit of advice.  Thanks for this!  I presume this is the equivalent of clearing the cache?  Or is it something else?

Its part of it, yes. And the main part that would generally cause an issue if you have incorrectly cached data

[LiquidFractal]

Sorry for the bump, but this has happened again and this time I cleared the /datastore directory enough to get me in, and this is the template_error / OutOfRangeException (0) which has occured approx. 290 times in the past hour.

Strangely enough, this was caused by a "guest" (who cannot access the current test site due to restrictions I imposed) who tried to visit this URL:

http://liquidfractal.space/LF_DEV/theme/?csrfKey=b0696c58afd5403058bc423deac2e089

which no longer exists on the site at all (as LF_DEV was deleted ages ago).

This is the backtrace:

#0 /XXX/system/Member/Member.php(804): IPS\Patterns\_ActiveRecord::load()
#1 /XXX/system/Patterns/ActiveRecord.php(335): IPS\_Member->get_group()
#2 /XXX/system/Member/Member.php(1962): IPS\Patterns\_ActiveRecord->__get()
#3 /XXX/applications/core/sources/Announcements/Announcement.php(132): IPS\_Member->isBanned()
#4 /XXX/system/Theme/Theme.php(885) : eval()'d code(104): IPS\core\Announcements\_Announcement::loadAllByLocation()
#5 /XXX/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_global->announcementContentTop()
#6 /XXX/system/Theme/Theme.php(885) : eval()'d code(18544): IPS\Theme\_SandboxedTemplate->__call()
#7 /XXX/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_global->widgetContainer()
#8 /XXX/system/Theme/Theme.php(885) : eval()'d code(6817): IPS\Theme\_SandboxedTemplate->__call()
#9 /XXX/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_global->globalTemplate()
#10 /XXX/system/Output/Output.php(745): IPS\Theme\_SandboxedTemplate->__call()
#11 /XXX/init.php(927) : eval()'d code(26): IPS\_Output->error()
#12 /XXX/init.php(1070): IPS\cms_hook_Output->error()
#13 [internal function]: IPS\IPS::exceptionHandler()
#14 {main}

 

Now, strangely enough, there is a distinct pattern in these errors.  This error will occur twice in succession, followed by the following uncaught_exception error:

OutOfRangeException:  (0)
#0 /XXX/system/Member/Member.php(804): IPS\Patterns\_ActiveRecord::load()
#1 /XXX/system/Patterns/ActiveRecord.php(335): IPS\_Member->get_group()
#2 /XXX//system/Dispatcher/Standard.php(87): IPS\Patterns\_ActiveRecord->__get()
#3 /XXX/init.php(927) : eval()'d code(15): IPS\Dispatcher\_Standard::baseJs()
#4 /XXX/init.php(927) : eval()'d code(15): IPS\Dispatcher\forumcover_hook_includeJSandCSS::baseJs()
#5 /XXX/init.php(927) : eval()'d code(42): IPS\Dispatcher\slidermaker_hook_includeJSandCSS::baseJs()
#6 /XXX/init.php(927) : eval()'d code(15): IPS\Dispatcher\notices_hook_dispatcher_standard::baseJs()
#7 /XXX/system/Dispatcher/Front.php(717): IPS\Dispatcher\testm_hook_includeJSandCSS::baseJs()
#8 /XXX/init.php(927) : eval()'d code(16): IPS\Dispatcher\_Front::baseJs()
#9 /XXX/init.php(927) : eval()'d code(108): IPS\Dispatcher\easypopup_hook_includeJSandCSS::baseJs()
#10 /XXX/init.php(927) : eval()'d code(20): IPS\Dispatcher\stickynotes_hook_Outputs::baseJs()
#11 /XXX/init.php(927) : eval()'d code(17): IPS\Dispatcher\faq_hook_FaqDispatcher::baseJs()
#12 /XXX/system/Dispatcher/Front.php(79): IPS\Dispatcher\invite_hook_dispatcherFront::baseJs()
#13 /XXX/init.php(927) : eval()'d code(28): IPS\Dispatcher\_Front->init()
#14 /XXX//init.php(927) : eval()'d code(23): IPS\Dispatcher\gms_hook_gmsJSHook->init()
#15 /XXX//init.php(927) : eval()'d code(20): IPS\Dispatcher\stickynotes_hook_Outputs->init()
#16 /XXX/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\dp45guestviewlimits_hook_dp45guestViewLimitsDispatcherFro->init()
#17 /XXX/l/index.php(13): IPS\_Dispatcher::i()
#18 {main}

This is the backtrace for the uncaught_exception:

#0 /XXX/init.php(1029): IPS\_Log::log()
#1 [internal function]: IPS\IPS::exceptionHandler()
#2 {main}

So: 2 template_errors and 1 uncaught_exception - This sequence has happened up to 7 times a minute (possibly more; I'm not combing through 290 pages!).  If someone can point me to whatever is causing this I'd appreciate it!

[opentype]

So is the site working again or not?

Keep in mind: you might be looking at symptoms, not the cause of the problems. If there are faulty files in the datastore (for example because of a one-time read/write error), then receiving all sorts of errors is expected, as the various modules try to process data that simply isn’t there or faulty. 

[Marc Stridgen]

And of course, in addition to what opentype said above, this does mean it may well not even happen on your live site. Clear the datastore again first of all, and let us know if this goes away again

[LiquidFractal]

@opentype and @Marc Stridgen Thanks for your insights.  And yes, for some reason these errors happen at completely different times on my test and production sites.  To answer the question, yes: I cleared the /datastore directory and the site was working fine again (didn't even have to clear it entirely).  Perhaps if, next time, I deleted /datastore files bit by bit in an effort to isolate which app may be causing these issues (sort of like gradually re-enabling apps/plugins)?

[Jim M]

31 minutes ago, LiquidFractal said:

@opentype and @Marc Stridgen Thanks for your insights.  And yes, for some reason these errors happen at completely different times on my test and production sites.  To answer the question, yes: I cleared the /datastore directory and the site was working fine again (didn't even have to clear it entirely).  Perhaps if, next time, I deleted /datastore files bit by bit in an effort to isolate which app may be causing these issues (sort of like gradually re-enabling apps/plugins)?

The issue may not be related specifically to 1 application/plugin though. It could just be your server having a particular issue once action x, y, z happens and thus creates corrupted/faulty files. To rule out the storage device on your server, you can always switch to Database storage for the datastore in ACP -> System -> Advanced Configuration -> Data storage. If it doesn't happen any more this would point to storage on your server.

[LiquidFractal]

Thanks Jim - I'll check this out.

[LiquidFractal]

OK: I just received this error again on my test site: site crashed, but I copy/pasted this from the technical information window which was displayed:

OutOfRangeException:  (0)
#0 /XXXX/system/Member/Member.php(804): IPS\Patterns\_ActiveRecord::load()
#1 /XXXX/system/Patterns/ActiveRecord.php(335): IPS\_Member->get_group()
#2 /XXXX/system/Session/Front.php(255): IPS\Patterns\_ActiveRecord->__get()
#3 [internal function]: IPS\Session\_Front->read()
#4 /XXXX//system/Session/Session.php(94): session_start()
#5 /XXXX/system/Theme/Theme.php(417): IPS\_Session::i()
#6 /XXXX/system/Theme/Theme.php(312): IPS\_Theme::currentThemeId()
#7 /XXXX/system/Dispatcher/Standard.php(54): IPS\_Theme::i()
#8 /XXXX/public_html/init.php(927) : eval()'d code(38): IPS\Dispatcher\_Standard::baseCss()
#9 /XXXX/init.php(927) : eval()'d code(43): IPS\Dispatcher\forumcover_hook_includeJSandCSS::baseCss()
#10 /XXXX/init.php(927) : eval()'d code(17): IPS\Dispatcher\slidermaker_hook_includeJSandCSS::baseCss()
#11 /XXXX/init.php(927) : eval()'d code(35): IPS\Dispatcher\notices_hook_dispatcher_standard::baseCss()
#12 /XXXX/system/Dispatcher/Front.php(759): IPS\Dispatcher\testm_hook_includeJSandCSS::baseCss()
#13 /XXXX/init.php(927) : eval()'d code(40): IPS\Dispatcher\_Front::baseCss()
#14 /XXXX/init.php(927) : eval()'d code(66): IPS\Dispatcher\easypopup_hook_includeJSandCSS::baseCss()
#15 /XXXX/system/Dispatcher/Front.php(76): IPS\Dispatcher\stickynotes_hook_Outputs::baseCss()
#16 /XXXX/init.php(927) : eval()'d code(28): IPS\Dispatcher\_Front->init()
#17 /XXXX/init.php(927) : eval()'d code(23): IPS\Dispatcher\gms_hook_gmsJSHook->init()
#18 /XXXX/init.php(927) : eval()'d code(20): IPS\Dispatcher\stickynotes_hook_Outputs->init()
#19 /XXXX/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\dp45guestviewlimits_hook_dp45guestViewLimitsDispatcherFro->init()
#20 /XXXX//index.php(13): IPS\_Dispatcher::i()
#21 {main}

Now, this time I tried deleting the cache files in /datastore bit by bit, then reloading the test site to see which files, when deleted, fixed the problem.

I think I've nailed it down to the following group of files (don't have time to do them one by one at this point):

/datastore/applications.bf64aba2f6.php
/datastore/cms_databases.bf64aba2f6.php
/datastore/cms_fieldids_1.bf64aba2f6.php
/datastore/defaultStreamData.bf64aba2f6.php
/datastore/forumsCustomNodes.bf64aba2f6.php
/datastore/frontNavigation.bf64aba2f6.php
/datastore/furl_configuration.bf64aba2f6.php
/datastore/globalStreamIds.bf64aba2f6.php
/datastore/groups.bf64aba2f6.php

At the very least, I can confidently say that when I deleted the above block of files from /datastore, my test site reverted to normal.

I have my layman suspicions about which of the above files might be the culprit, but does this information help anyone in the know offer insights as to what might be causing this?

Thanks!

 

Edited December 17, 2022 by LiquidFractal

[Miss_B]

6 hours ago, LiquidFractal said:

#9 /XXXX/init.php(927) : eval()'d code(43): IPS\Dispatcher\forumcover_hook_includeJSandCSS::baseCss()
#10 /XXXX/init.php(927) : eval()'d code(17): IPS\Dispatcher\slidermaker_hook_includeJSandCSS::baseCss()
#11 /XXXX/init.php(927) : eval()'d code(35): IPS\Dispatcher\notices_hook_dispatcher_standard::baseCss()
#12 /XXXX/system/Dispatcher/Front.php(759): IPS\Dispatcher\testm_hook_includeJSandCSS::baseCss()
#13 /XXXX/init.php(927) : eval()'d code(40): IPS\Dispatcher\_Front::baseCss()
#14 /XXXX/init.php(927) : eval()'d code(66): IPS\Dispatcher\easypopup_hook_includeJSandCSS::baseCss()
#15 /XXXX/system/Dispatcher/Front.php(76): IPS\Dispatcher\stickynotes_hook_Outputs::baseCss()
#16 /XXXX/init.php(927) : eval()'d code(28): IPS\Dispatcher\_Front->init()
#17 /XXXX/init.php(927) : eval()'d code(23): IPS\Dispatcher\gms_hook_gmsJSHook->init()
#18 /XXXX/init.php(927) : eval()'d code(20): IPS\Dispatcher\stickynotes_hook_Outputs->init()
#19 /XXXX/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\dp45guestviewlimits_hook_dp45guestViewLimitsDispatcherFro->init()

All the errors quoted above mention custom apps. 

[Adriano Faria]

11 minutes ago, Miss_B said:

All the errors quoted above mention custom apps. 

That doesn’t mean ALL these resources has issues. The error is indicating and showing all hooks that extends same class. Usually the error is in the #0.

[Ramsesx]

I had something similar some time ago, every some weeks some pages weren't accessible and it throws out template errors. Duplication of the custom style was the only thing that helps. After disabling a plugin the occurence was lower. Now since some months no more issues. So I guess, try to disable some plugins and observe. Having custom styles and a lot of plugins will cause issues, that's my experience.

[Miss_B]

4 hours ago, Adriano Faria said:

That doesn’t mean ALL these resources has issues. The error is indicating and showing all hooks that extends same class. Usually the error is in the #0.

I know that and I didn't mean to imply that. I wanted to bring it to the OP 's attention so he could disable them and see if one of them was the culprit.

[LiquidFractal]

2 hours ago, Miss_B said:

I know that and I didn't mean to imply that. I wanted to bring it to the OP 's attention so he could disable them and see if one of them was the culprit.

I've tried this before on others' suggestions, but it didn't seem to resolve the issue.  At least it seems as if I can re-enable a couple of apps with this new information, as the above block of files doesn't seem to refer to 3P apps.

4 hours ago, Ramsesx said:

I had something similar some time ago, every some weeks some pages weren't accessible and it throws out template errors. Duplication of the custom style was the only thing that helps. After disabling a plugin the occurence was lower. Now since some months no more issues. So I guess, try to disable some plugins and observe. Having custom styles and a lot of plugins will cause issues, that's my experience.

Yeah, I've actually been cutting down on 3P apps and plugins over the last while...too many moving parts and some devs are far less dedicated than others.  I'm hoping someone can tell me if these cache files refer to something in particular (really hope it isn't my theme!!).

[Richard Arch]

I'm no expert but as Adriano pointed out, the issue is normally got to do with the first line in the trace.  That is in member.php line 804 which is something to do with primary groups.

Have changed them recently?  Could you have a member that has an invalid group?

[LiquidFractal]

3 hours ago, Richard Arch said:

I'm no expert but as Adriano pointed out, the issue is normally got to do with the first line in the trace.  That is in member.php line 804 which is something to do with primary groups.

Have changed them recently?  Could you have a member that has an invalid group?

Hmm....haven't created any new member groups or altered anything there for quite awhile now.  It also doesn't explain (to me anyway) why this only happens every once in a while and isn't a constant issue.

And I just logged in to my test site to find that it's crapped out again...and I've definitely not touched member groups at all in the interim between this event and the last one.  The plot thickens!

EDIT: it appears as if you're right Richard - groups might very well be the issue after all!  I used my previous list and deleted them one by one.  Once I deleted this cache file:

/datastore/groups.bf64aba2f6.php

everything returned to normal.

So how might I address this?  Anyone?  Should I wipe custom groups on my test site and recreate them and see if that does anything?

Edited December 17, 2022 by LiquidFractal