Rebuilding millions of posts after 4.0 upgrade

[James Ford]

I've upgraded our IP.Board from 3.4.7 to 4.0.4.

But with 3 million+ posts and 1 million+ messages the rebuild process is set to take 14 days to complete!

Problem is, our database now locks up when more than 400 users are active. I suspect because the posts and topics are not yet reindexed.

My questions:

1. Is there a faster command line tool for rebuilding of posts and messages?
2. Is it safe to manually adjust priorities in the "core_queue" table to force reindexing before rebuild?

More than 24 hours after upgrade, this is what our "Background processes" panel looks like in the admin dashboard:

And this is the contents of our "core_queue" table in MySQL:

[James Ford]

I reduced the search depth to 90 days, and purged our "trash" subforum which contained 0.8m posts.

That, plus various other optimisations, I managed to get the rebuild for 2.5m remaining posts down to 4 days.

Meanwhile the problem with thte database locking up is unrelated, it's due to this reported bug about suboptimal SQL:

- http://community.invisionpower.com/4bugtrack/404-site-bogged-down-bad-r5784/

[Abuzaid]

on my board 3.4.8 there is about 900000 post, the background tasks took so far ten days to rebuild half the posts ;

logically I can't accept the this task takes  this amount of time  because this means me closing the board for nearly a month to get this posts being rebuild.

the support staff advised either running in manually as I'm doing now or run an acron if I knew how to do it ;  fact is I don't knew this expression at all and how to do it

the only option left for me is forget completely about moving to the new version !! 

[prupdated]

1 hour ago, Abuzaid said:

on my board 3.4.8 there is about 900000 post, the background tasks took so far ten days to rebuild half the posts ;

logically I can't accept the this task takes  this amount of time  because this means me closing the board for nearly a month to get this posts being rebuild.

the support staff advised either running in manually as I'm doing now or run an acron if I knew how to do it ;  fact is I don't knew this expression at all and how to do it

the only option left for me is forget completely about moving to the new version !! 

With 3.5 million posts, the fastest way I found was to:

1) Use cron to run tasks on my server and 2) reduce search depth to 90 days. It took about a week total.

#1 - using cron - was essential for it getting done anytime soon. If you haven't set up your server to run IPS jobs via cron, I'd highly recommend getting it figured out no matter what it takes - especially as your forum size is larger. The server needs to be kicking off the cron jobs each minute regardless of how much activity is on the forum. The manual method in my experience was slower than cron and not as consistent.