Database professional required - we need expert help

[Cyboman]

Hi,

unfortunately we had a bad database crash one week ago, and since then, we are totally offline for our community.

We don't really know what happend. We operate on Ubuntu, PHP 5.6 and MySQL in a Docker framework with nginx and Apache. Latest IPS version was 4.1.15. We updated the software 1 day before the crash, also the plugins some hours before the crash.

Then suddenly during the (crash) night, the server got slower and slower and finally wasn't available anymore. Apache produced log files > 100GB in size during the night, and the forum, that has been working the day before, first produced an error message on frontend (EX1146), then denied access to the ACP with additional error messages... Tried to connect to database tables that don't exist... IPS support just told us "to repair the database" (without having a look), what we did, but it doesn't work. No more assistance by IPS.

We have daily backups, 6,5GB in size each, but when we try to reimport the backups we get error messages from the MySQL server. It seems as if the error was there before, kind of indexing problems.

ERROR 1214 (HY000) at line 6413: The used table type doesn't support FULLTEXT indexes

The error messages are far beyond our knowledge and we don't get it managed to go back online.

So I'm asking for a SysOp & SQL professional to hire, that could identify our problems, solve them, reconfigure our SQL server to the best operation state, tell us some documentation points what happened and what was configured, and helps as go back online with our backups as soon as possible. We're looking for an expert!

I hope this is the right area to find someone, that could help. We're already loosing our members (>100K registered in the community) to other forums.

Thanks for any help or recommendations.

[RevengeFNF]

Are your tables in Innodb? What version of Mysql you have?

That seems an error if you are using Mysql 5.5 or lower, and your tables are in Innodb.

If that's the case, i suggest you to update your Mysql to version 5.6 or higher, or MariaDB 10.1 and problem will be solved.

[ASTRAPI]

It is better to upgrade to latest version as it will automatically fix your issue as RevengeFNF recommend already and get better performance also

[Cyboman]

Thanks a lot for your responses!

The database is repaired, but there is a persistant critical problem with our server, that probably caused this situation. Let me explain:

We detected a very huge amout of data in /apache-htdocs/uploads/JavaScript_global

There are thousands of files named root_front.js.2bb848136453cd6....js, the hash ending is always varying. All of the files are 80KB in size, after one night 40GB in size!!!

We don't know where they come from and if we can delete them. But the very big problem is: after one night, the server crashes regularily due to "not enough disk storage space".

Anyone knows this phenomen? Or has an idea where it could come from?

By the way, since we upgraded to PHP7, we can't post anymore, nor message and the menus don't work as they should. It looks as if the whole JavaScript doesn't work anymore.

Thanks for any hint.

[ASTRAPI]

I think that errors are related to javascript but it will be better to let IPB Support to verify it ....

[dikkertjedap]

Table 'secret_ipb.admin_permission_rows' doesn't exist

And how can i insert admin_permission_rows i never delete this row.

What is wrong Db version is MySQL version 5.5.41. and compatible with invision version 4.

[Cyboman]

Part of the solution to my problems:

"Database Tables disappeared": Some oracle SQL upgrades cause to not import several standard tables unless forced to (this is a known problem I experienced...)! So if the database might have been updated, this could be the problem. If you had to upgrade PHP to v7 to keep support, SQL usually is recommended to also upgrade. But under special circumstances it doesn't import all standard tables!

"Apache generating trash in wrong directories": Some configuration inside IPB links to Apache default directories (you can set these anywhere in ACP). Apache changed their default directories in the past, so if this is the case, the links could direct to the wrong location.