System Log approx 2gig; Core_Sessions Keeps crashing and needs rebuild

[dalbert]

System log in the database is approx 2.3gig, and core_sessions keeps on needing to be rebuilt every week or so (getting more frequent now -- maybe every 4 days). Appears that today alone there were 400k entries.

 

We updated from approx 4.7.8 to 4.17.13, and the issue seems to have gotten worse.

Issue began happening between Sept. 1st and 2nd, when the database grew 500 meg overnight.... We didn't do any board changes on that date, and the other IPS board on the same server seems to be running fine.

 

 

Here is one example of a frequent error - I would post more, but I just had to reboot the server

 

Error: Call to a member function language() on null (0)
#0 /usr/www/users/myuser/ourboard/forums/system/Dispatcher/Front.php(719): IPS\Dispatcher\_Standard::baseJs()
#1 /usr/www/users/myuser/ourboard/forums/init.php(936) : eval()'d code(45): IPS\Dispatcher\_Front::baseJs()
#2 /usr/www/users/myuser/ourboard/forums/system/Dispatcher/Front.php(79): IPS\Dispatcher\agepopup_hook_dispatcherFront::baseJs()
#3 /usr/www/users/myuser/ourboard/forums/init.php(936) : eval()'d code(98): IPS\Dispatcher\_Front->init()
#4 /usr/www/users/myuser/ourboard/forums/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\memberscountry_hook_dispatcherFront->init()
#5 /usr/www/users/myuser/ourboard/forums/index.php(13): IPS\_Dispatcher::i()
#6 {main}

 

 

 

[dalbert]

Server just came back up after reboot, and the following error is in the admin CP... typically a repair of the sql table will resolve this, and get things back on track

 

SELECT * FROM `core_sessions` WHERE ( core_sessions.id IN('SELECT MAX(id) FROM `core_sessions` AS `s` WHERE s.running_time>'SELECT MAX(id) FROM `core_sessions` AS `s` WHERE s.running_time>? AND s.login_type!=? AND (s.login_type!=? OR s.member_id=?) AND s.member_id IS NULL GROUP BY `ip_address`' AND s.login_type!=? AND (s.login_type!=? OR s.member_id=?) AND s.member_id IS NOT NULL GROUP BY `member_id`') OR core_sessions.id IN(?) ) ORDER BY core_sessions.running_time desc
IPS\Db\Exception: Table 'core_sessions' is marked as crashed and should be repaired (1194)
#0 /usr/www/users/myuser/myboard/forums/system/Db/Select.php(286): IPS\Db\_Select->runQuery()
#1 /usr/www/users/myuser/myboard/forums/system/Session/Store/Database.php(253): IPS\Db\_Select->setKeyField()
#2 /usr/www/users/myuser/myboard/forums/applications/core/extensions/core/Dashboard/OnlineUsers.php(52): IPS\Session\Store\_Database->getOnlineUsers()
#3 /usr/www/users/myuser/myboard/forums/applications/core/modules/admin/overview/dashboard.php(64): IPS\core\extensions\core\Dashboard\_OnlineUsers->getBlock()
#4 /usr/www/users/myuser/myboard/forums/system/Dispatcher/Controller.php(118): IPS\core\modules\admin\overview\_dashboard->manage()
#5 /usr/www/users/myuser/myboard/forums/system/Dispatcher/Dispatcher.php(153): IPS\Dispatcher\_Controller->execute()
#6 /usr/www/users/myuser/myboard/forums/admin/index.php(13): IPS\_Dispatcher->run()
#7 {main}

 

 

 

[Randy Calvert]

33 minutes ago, dalbert said:

IPS\Db\Exception: Table 'core_sessions' is marked as crashed and should be repaired (1194)

There is your problem. You need to repair that table. 

Edited September 27, 2023 by Randy Calvert

[dalbert]

Correct, but it keeps on corrupting - I have gone from having to do it every 3-4mo, to every week, and now every 4 days.

At this point I just emptied the core_log that was 2.31gig in hopes that would help the issue, or at least give me more leeway. 

[teraßyte]

The table shouldn't crash so often, and the big logs table is another concern.

Is your site getting hit by some kind of (DDOS?) attack, perhaps? 🤨

 

It could also be that your MySQL configuration is not correct and needs to be optimized/tuned. That's best discussed with your hosting, though. Or, if you're managing the server yourself, perhaps you can try the mysqltuner script: https://github.com/major/MySQLTuner-perl

Edited September 27, 2023 by teraßyte

[Randy Calvert]

Just now, dalbert said:

Correct, but it keeps on corrupting - I have gone from having to do it every 3-4mo, to every week, and now every 4 days.

At this point I just emptied the core_log that was 2.31gig in hopes that would help the issue, or at least give me more leeway. 

That’s typically a hosting issue. There might be file system corruption (needing to be fsck’ed), or something else. That’s not a software issue. 

[dalbert]

I have submitted a case to the host to check the disk, however our other Invision site as well as our Xenforo site are both working normally.. So at present that makes me wonder about ddos/effort to remotely cause hiccups or some other internal  file corruption, that is just becoming the gift that keeps on giving. 

 

Since clearing the whole core_logs a little bit ago, nothing has popped up into the logs... Lets see how we do over the next few hours / overnights.

 

Thanks.

[dalbert]

As a quick follow up, the ISP noted that the drive was passing a short test, but failing on the long test... Drive was replaced. They also noted a DOS login attack from another dedicated server provider, and blocked it. We will see how the forum behaves over the coming days.

[teraßyte]

So it ended up being a bit of both, then. 😅