filmix

Just got the reply from the server host tech. I did give them a link to this thread, not sure if he checked it out. I gave this a quick read-through but it's 1am and I'm too tired to make much sense of it all. I can see that your Droplet's CPU usage was spiking and I am also seeing minor CPU steal. While we don't have any visibility of what exactly happened inside the droplet, we would love to guide you through investigating it in detail. Resource contention can impact a lot of services and can be triggered by a large variety of issues. My first recommendation would be to begin troubleshooting the processes on your server. You can look at the output of the "top" command and run an assorted view to sort by top processes. We need to check which process or service is hitting the resources badly. Please make sure to run this command especially when your droplet's CPU usage is spiking to get accurate information. Please also share this output with us. Additionally, you can also view our tutorial on monitoring and troubleshooting high CPU: https://www.digitalocean.com/community/tutorials/how-to-monitor-cpu-use-on-digitalocean-droplets To kill all processes and attempt to free up memory, you can power off and on your server. Please keep in mind that this will reset the processes and you will no longer be able to see which processes were using the resources. Please review: https://www.digitalocean.com/community/tutorials/how-to-use-ps-kill-and-nice-to-manage-processes-in-linux There are some tools that you can use to keep track of the resource usage on your droplet. I will recommend you install 'atop' on your droplet. This tool saves the log for all processes date wise. It makes it easier to investigate processes in the past. Please refer: https://www.tecmint.com/how-to-install-atop-to-monitor-logging-activity-of-linux-system-processes/ We can also offer to migrate your Droplet to a different Hypervisor to rule out the Hypervisor issue. Let us know if you want us to live migrate the Droplet. You can read more about this here: https://www.digitalocean.com/docs/droplets/resources/live-migration/ If you have any further questions, feel free to reach back. We will be happy to help.

Just submitted a support ticket with our server admin, will update soon. The thing that still has us baffled is that the board runs normally when it's closed. Only when we open it to the public does it freeze up.

We tried that, it didn't work, CPU pinned after a couple of minutes and the board 504ed. We're closed again. There are no indications of any system log problems. We're at a loss on what to try next.

We're going to try switching that Data Storage setting back to MySQL and open the board back up.

Yeah, we need to keep a better eye on purging those logs. Right now the log entries date back to the 21st which jives with the escalation of the CPU issue. We had a storage issue about five weeks ago -- our HD was filling up. During troubleshooting we switched the Data Strorage setting from MySQL to File System, just to try it. We switched that and pruned some old logs and it was fixed. We have no idea if switching that setting helped, but our storage has been problem-free since then. This is the current setting.

I'll paste the list here. Reoccurring system log entries may be indicative of a problem that should be addressed. A summary of the top reoccurring recent log entries are shown below. Review the system log for further details. TypeError: Illegal offset type in isset or empty (0) #0 /var/www/html/system/Application/Application.php(236): IPS\Patterns\_ActiveRecord::load() #1 /var/www/html/system/Dispatcher/Standard.php(161): IPS\_Application::load() #2 /var/www/html/system/Dispatcher/Admin.php(159): IPS\Dispatcher\_Standard->init() #3 /var/www/html/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Admin->init() #4 /var/www/html/admin/index.php(13): IPS\_Dispatcher::i() #5 {main} 60 TypeError: preg_match(): Argument #2 ($subject) must be of type string, array given (0) #0 /var/www/html/system/Dispatcher/Controller.php(102): preg_match() #1 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #2 /var/www/html/index.php(13): IPS\_Dispatcher->run() #3 {main} 50 TypeError: Illegal offset type in isset or empty (0) #0 /var/www/html/system/Application/Application.php(236): IPS\Patterns\_ActiveRecord::load() #1 /var/www/html/system/Dispatcher/Standard.php(161): IPS\_Application::load() #2 /var/www/html/system/Dispatcher/Front.php(146): IPS\Dispatcher\_Standard->init() #3 /var/www/html/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Front->init() #4 /var/www/html/index.php(13): IPS\_Dispatcher::i() #5 {main} 36 TypeError: Illegal offset type in isset or empty (0) #0 /var/www/html/system/Dispatcher/Standard.php(218): IPS\Application\_Module::get() #1 /var/www/html/system/Dispatcher/Front.php(146): IPS\Dispatcher\_Standard->init() #2 /var/www/html/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Front->init() #3 /var/www/html/index.php(13): IPS\_Dispatcher::i() #4 {main} 36 TypeError: trim(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/system/Helpers/Form/Captcha/Invisible.php(66): trim() #1 /var/www/html/system/Helpers/Form/Captcha.php(149): IPS\Helpers\Form\Captcha\_Invisible->verify() #2 /var/www/html/system/Helpers/Form/FormAbstract.php(167): IPS\Helpers\Form\_Captcha->getValue() #3 /var/www/html/system/Helpers/Form/FormAbstract.php(147): IPS\Helpers\Form\_FormAbstract->setValue() #4 /var/www/html/system/Helpers/Form/Captcha.php(70): IPS\Helpers\Form\_FormAbstract->__construct() #5 /var/www/html/applications/core/modules/front/contact/contact.php(77): IPS\Helpers\Form\_Captcha->__construct() #6 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\contact\_contact->manage() #7 /var/www/html/applications/core/modules/front/contact/contact.php(43): IPS\Dispatcher\_Controller->execute() #8 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\core\modules\front\contact\_contact->execute() #9 /var/www/html/index.php(13): IPS\_Dispatcher->run() #10 {main} 34 TypeError: base64_decode(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/system/Request/Request.php(714): base64_decode() #1 /var/www/html/applications/core/modules/front/system/login.php(44): IPS\_Request->referrer() #2 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_login->manage() #3 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #4 /var/www/html/index.php(13): IPS\_Dispatcher->run() #5 {main} 28 TypeError: base64_decode(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/system/Request/Request.php(714): base64_decode() #1 /var/www/html/applications/core/modules/front/system/theme.php(66): IPS\_Request->referrer() #2 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_theme->manage() #3 /var/www/html/applications/core/modules/front/system/theme.php(32): IPS\Dispatcher\_Controller->execute() #4 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\core\modules\front\system\_theme->execute() #5 /var/www/html/index.php(13): IPS\_Dispatcher->run() #6 {main} 23 TypeError: htmlspecialchars(): Argument #1 ($string) must be of type string, array given in /var/www/html/system/Theme/Theme.php(885) : eval()'d code:3632 Stack trace: #0 /var/www/html/system/Theme/Theme.php(885) : eval()'d code(3632): htmlspecialchars() #1 /var/www/html/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_system->loginForm() #2 /var/www/html/system/Theme/Theme.php(885) : eval()'d code(3390): IPS\Theme\_SandboxedTemplate->__call() #3 /var/www/html/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_system->login() #4 /var/www/html/applications/core/modules/front/system/login.php(132): IPS\Theme\_SandboxedTemplate->__call() #5 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_login->manage() #6 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #7 /var/www/html/index.php(13): IPS\_Dispatcher->run() #8 {main} 18 TypeError: Illegal offset type (0) #0 /var/www/html/applications/core/modules/front/system/login.php(59): IPS\_Login->authenticate() #1 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_login->manage() #2 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #3 /var/www/html/index.php(13): IPS\_Dispatcher->run() #4 {main} 16 TypeError: urldecode(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/applications/core/modules/front/system/ajax.php(563): urldecode() #1 /var/www/html/system/Dispatcher/Controller.php(107): IPS\core\modules\front\system\_ajax->passwordStrength() #2 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #3 /var/www/html/index.php(13): IPS\_Dispatcher->run() #4 {main} 16 TypeError: mb_strlen(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/applications/core/modules/admin/system/login.php(280): mb_strlen() #1 /var/www/html/applications/core/modules/admin/system/login.php(126): IPS\core\modules\admin\system\_login->_log() #2 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\admin\system\_login->manage() #3 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #4 /var/www/html/admin/index.php(13): IPS\_Dispatcher->run() #5 {main} 16 TypeError: mb_substr(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/applications/core/modules/front/search/search.php(1266): mb_substr() #1 /var/www/html/system/Dispatcher/Controller.php(107): IPS\core\modules\front\search\_search::quicksearch() #2 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #3 /var/www/html/index.php(13): IPS\_Dispatcher->run() #4 {main} 12TypeError: Illegal offset type in isset or empty (0) #0 /var/www/html/system/Application/Application.php(236): IPS\Patterns\_ActiveRecord::load() #1 /var/www/html/system/Dispatcher/Standard.php(161): IPS\_Application::load() #2 /var/www/html/system/Dispatcher/Admin.php(159): IPS\Dispatcher\_Standard->init() #3 /var/www/html/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Admin->init() #4 /var/www/html/admin/index.php(13): IPS\_Dispatcher::i() #5 {main} 60 TypeError: preg_match(): Argument #2 ($subject) must be of type string, array given (0) #0 /var/www/html/system/Dispatcher/Controller.php(102): preg_match() #1 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #2 /var/www/html/index.php(13): IPS\_Dispatcher->run() #3 {main} 50 TypeError: Illegal offset type in isset or empty (0) #0 /var/www/html/system/Application/Application.php(236): IPS\Patterns\_ActiveRecord::load() #1 /var/www/html/system/Dispatcher/Standard.php(161): IPS\_Application::load() #2 /var/www/html/system/Dispatcher/Front.php(146): IPS\Dispatcher\_Standard->init() #3 /var/www/html/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Front->init() #4 /var/www/html/index.php(13): IPS\_Dispatcher::i() #5 {main} 36 TypeError: Illegal offset type in isset or empty (0) #0 /var/www/html/system/Dispatcher/Standard.php(218): IPS\Application\_Module::get() #1 /var/www/html/system/Dispatcher/Front.php(146): IPS\Dispatcher\_Standard->init() #2 /var/www/html/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Front->init() #3 /var/www/html/index.php(13): IPS\_Dispatcher::i() #4 {main} 36 TypeError: trim(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/system/Helpers/Form/Captcha/Invisible.php(66): trim() #1 /var/www/html/system/Helpers/Form/Captcha.php(149): IPS\Helpers\Form\Captcha\_Invisible->verify() #2 /var/www/html/system/Helpers/Form/FormAbstract.php(167): IPS\Helpers\Form\_Captcha->getValue() #3 /var/www/html/system/Helpers/Form/FormAbstract.php(147): IPS\Helpers\Form\_FormAbstract->setValue() #4 /var/www/html/system/Helpers/Form/Captcha.php(70): IPS\Helpers\Form\_FormAbstract->__construct() #5 /var/www/html/applications/core/modules/front/contact/contact.php(77): IPS\Helpers\Form\_Captcha->__construct() #6 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\contact\_contact->manage() #7 /var/www/html/applications/core/modules/front/contact/contact.php(43): IPS\Dispatcher\_Controller->execute() #8 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\core\modules\front\contact\_contact->execute() #9 /var/www/html/index.php(13): IPS\_Dispatcher->run() #10 {main} 34 TypeError: base64_decode(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/system/Request/Request.php(714): base64_decode() #1 /var/www/html/applications/core/modules/front/system/login.php(44): IPS\_Request->referrer() #2 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_login->manage() #3 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #4 /var/www/html/index.php(13): IPS\_Dispatcher->run() #5 {main} 28 TypeError: base64_decode(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/system/Request/Request.php(714): base64_decode() #1 /var/www/html/applications/core/modules/front/system/theme.php(66): IPS\_Request->referrer() #2 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_theme->manage() #3 /var/www/html/applications/core/modules/front/system/theme.php(32): IPS\Dispatcher\_Controller->execute() #4 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\core\modules\front\system\_theme->execute() #5 /var/www/html/index.php(13): IPS\_Dispatcher->run() #6 {main} 23 TypeError: htmlspecialchars(): Argument #1 ($string) must be of type string, array given in /var/www/html/system/Theme/Theme.php(885) : eval()'d code:3632 Stack trace: #0 /var/www/html/system/Theme/Theme.php(885) : eval()'d code(3632): htmlspecialchars() #1 /var/www/html/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_system->loginForm() #2 /var/www/html/system/Theme/Theme.php(885) : eval()'d code(3390): IPS\Theme\_SandboxedTemplate->__call() #3 /var/www/html/system/Theme/SandboxedTemplate.php(61): IPS\Theme\class_core_front_system->login() #4 /var/www/html/applications/core/modules/front/system/login.php(132): IPS\Theme\_SandboxedTemplate->__call() #5 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_login->manage() #6 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #7 /var/www/html/index.php(13): IPS\_Dispatcher->run() #8 {main} 18 TypeError: Illegal offset type (0) #0 /var/www/html/applications/core/modules/front/system/login.php(59): IPS\_Login->authenticate() #1 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\front\system\_login->manage() #2 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #3 /var/www/html/index.php(13): IPS\_Dispatcher->run() #4 {main} 16 TypeError: urldecode(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/applications/core/modules/front/system/ajax.php(563): urldecode() #1 /var/www/html/system/Dispatcher/Controller.php(107): IPS\core\modules\front\system\_ajax->passwordStrength() #2 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #3 /var/www/html/index.php(13): IPS\_Dispatcher->run() #4 {main} 16 TypeError: mb_strlen(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/applications/core/modules/admin/system/login.php(280): mb_strlen() #1 /var/www/html/applications/core/modules/admin/system/login.php(126): IPS\core\modules\admin\system\_login->_log() #2 /var/www/html/system/Dispatcher/Controller.php(118): IPS\core\modules\admin\system\_login->manage() #3 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #4 /var/www/html/admin/index.php(13): IPS\_Dispatcher->run() #5 {main} 16 TypeError: mb_substr(): Argument #1 ($string) must be of type string, array given (0) #0 /var/www/html/applications/core/modules/front/search/search.php(1266): mb_substr() #1 /var/www/html/system/Dispatcher/Controller.php(107): IPS\core\modules\front\search\_search::quicksearch() #2 /var/www/html/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute() #3 /var/www/html/index.php(13): IPS\_Dispatcher->run() #4 {main} 12 Here are the latest system logs from the past few days. 01/27/25 03:16 PM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_b65823d9b0a67dc... 01/27/25 10:05 AM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_62038bbd7bd51f2... 01/27/25 03:11 AM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_fa9d47b1b2babc6... 01/27/25 02:06 AM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_a583ccf7160873c... 01/26/25 11:07 PM geoip GeoIP Error Requested IP: 52.167.144.169 Response: IPS\Http\Response Object ( [httpResponseVer... 01/26/25 09:15 PM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_5f773484eb2e69a... 01/26/25 03:17 PM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_170af3c0b78f776... 01/26/25 10:56 AM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_1c6a97b0b846167... 01/26/25 06:52 AM uncaught_exception Error: Failed opening required '/var/www/html/datastore/widget_activeUsers_47xt0cvrd_a7df6113e2f785d...

No third party apps were added. We’re not running any right now. Miss_B, we were wondering about the query thing too. Going back to my first post I mentioned that there’s a recommendation in the ACP to fix some repeated log entries but they don’t look familiar to us. We were wondering if they were queries run by a guest, maybe trying to access a known exploit?

Oops, sorry, misread you. MySQL 8.0.40

PHP Version 8.0.30 There’s a recommendation to upgrade to at least 8.1 but that the current installed version is fine for now.

For the past two weeks or so we’ve noticed our cloud server’s CPU operating at an increasingly higher level than normal. Normally the graph will show lots of peaks and valleys throughout the day ranging from say 20% to 90% depending on the board’s activity levels. I’ll attach a screenshot of the graph from two weeks ago which shows normal operating levels (first graph). Over the past two weeks we saw the CPU peaking more often, leading to board slowdowns. This morning we noticed the board was timing out. A look at the graph showed the CPU pinned at 100%, and it was pinned for increasingly long stretches (graph 2 screenshot). Rebooting the server did not solve the problem. Only when we turned the community off did the CPU% drop. As soon as we opened it back up it shot back up to 100% within a minute. What we’ve tried today on the ACP side of things was clearing the system cache and cleaning up some risky code which was brought to our attention in the ACP’s notifications section (see ACP screenshot). There’s also a recommendation to fix some repeated log entries but they look like something on the Invision side of things — they don’t look familiar to us. Right now the board is powered up but closed to traffic — and it’s running great while it’s closed. Hopefully you guys can help us out. Thanks in advance! Jim

That’s actually easier than my old method. I also like Jim’s group promotion method, so I can see implementing that as well. Thanks for the help, guys!

In our community we have a special badge (a jpg made in Photoshop) that we hand out in a dedicated thread when a member hits a certain post count. We’ve been doing this for 20 years now. I used to be able to keep track of the members who were approaching that number in the members list in the ACP. I could sort the list by post count and it was super easy. I’m now noticing that post count is no longer a sort-able option in that list. When did that go away, and why? Is there another way view this info? It would be unfortunate to abandon a 20+year community tradition that people really looked forward to.

Our community needed an upgrade from 3.3.4 to 4.6.12. First we tried the upgrade ourselves but it was unsuccessful -- there are almost 10 years between those versions. Looking at the roster of third party developers we saw that Ahmed had experience with legacy upgrades from much older versions, with excellent reviews. We hired him, and his work was detailed and efficient, and he was there after the upgrade was complete if we needed any help or answers to questions. Highly recommended.

No problem, we'll adjust. Use multi-quote where needed. Funny thing, the new software displays the nested quotes from before the upgrade with no problem. Except for the 10+ quotes on the mobile version. That's a jumbled mess.

This question is still up for grabs. Thanks to all for the help so far with other matters!