Not related to IPS and just throwing this out there to see if anyone has suggestions on how to troubleshoot. My server went down last night and I only found my sites were all offline in the morning. Fast forward and I can't get mysql server to restart. I can get it to restart if I add to  /etc/my.cnf  innodb_force_recovery = 2

Error log gives a couple of warnings but only one error

2024-03-30T21:08:57.164248Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: trx0types.h:541:m_rsegs_n < 2 thread 139670608385792

Ideas on troubleshooting? Error log doesn't give me any corrupt database or files..

I was going to start dropping tables but thinking maybe this is a sign I need to upgrade from Centos 7 to AlmaLinux (running cpanle/whm) and move to a new box. 

It sounds like your mySQL database might have some corruption in it.  You might want to restore it from backup.  Since you can get it back online with recovery mode 2, you can just export the database and rescue the data.  

https://mariadb.com/kb/en/innodb-recovery-modes/

I would suggest immediately dumping the database, andreimport it.  (You could potentially just import it into a new database with a new username/password as well just to be safe.)

Did the server run out of space or was it rebooted without a clean shutdown?  I typically only see that sort of thing happen when mySQL loses the ability to read data that happened to be in the memory buffer before it has a chance to write to the disk.

Edited March 30, 20241 yr by Randy Calvert

I have backups and would do that but my issue is that I’m not sure which database is causing the issue. I’ve got 13 or so and none of them come up in the error log.

I think something happened at the data center overnight and then something got corrupted.

Edited March 31, 20241 yr by AlexWebsites

You might be able to do something from the command line with something like:

for i in $(ls /var/lib/mysql/*/*.ibd)
do
  innochecksum -v $i
done

That’s assuming the path to your raw db files are in that path. Also I would first try to run that command with MySQL offline first. I’m on my phone now so I can’t fix formatting.

Edited March 31, 20241 yr by Randy Calvert

This ended up being non-recoverable. I ended up moving my sites to a new setup and trying out plesk over cpanel, so far so good.