Upgrade to v4 took more than 7 days to complete!

[Tom_K]

I upgraded from v3 to v4 (test upgrade to see how it goes). The upgrade itself was done fairly quickly but the issue is the post-upgrade tasks.

I have a board with 2.5 million posts and 20.000 members. It's running on a dedicated server and it's the only site on the server. It took more than 7 days for the post-upgrade tasks to finish (post conversion and other stuff).

This is totally unacceptable. The main reason is the script converts 50 posts at a time. I can understand this is reasonable for smaller forums but with 2.5 million posts? Come on. My server was idling when it could do the job much faster.

In those 7 days most posts and quotes were broken. I don't know how I can ask my members to wait out a week to be able to get proper functionalities back.

How can a simple upgrade to a newer version take more than a week to complete?

[Jed Rosenzweig]

I'm on enterprise with similar stats and we were upgraded a month ago with the same experience but live and public. It was a PITA. It's also search and activity streams that break due to incomplete rebuilds.

I don't have an answer for you except if there's a way you can protect your users from this, do it because THEY WILL NOT LIKE IT.

[Tom_K]

That's what scares me. It basically criples the board for a week. Broken posts, no search, no activity. Yikes!

Does the activity start working again after the rebuild or does it start marking new posts/topics from scratch?

Did you have any kind of notice warning your users that it is just temporary and did it help?

[Jed Rosenzweig]

Activity wasn't error-broken but gave incomplete searches and so the users think it is broken. They know there's a post in X but the search/activity doesn't see it. Honestly no amount of announcements helped. I had global announcements on, forum-specific pinned posts and even a non-IPS annc bar and people still reported X,Y and Z as new issues over and over. I like 4.x a lot but it wasn't a good transition.

[White Miku]

7 hours ago, Tomzl said:

script converts 50 posts at a time

You used cron or you have runned tasks from ACP's mainpage?

[Lindy]

As CV01 eluded to, you may speed this up significantly by utilizing a cron. If that's somehow not clear, I'm sure we can make it so. 

[Tom_K]

I did use the cron option. You don't really think I had my PC on 24/7 for more than 7 days to have the tasks run manually?

Perhaps I should have mentioned that I contacted support during the upgrade and asked how I can speed up the process. They told me I can't.

That's why I'm posting my feedback and experience with the upgrade process here.

I think there is something fundamentally flawed in the upgrade process, if it criples the most important functions of the system for more than a week.

@David T. Cole shared his experience on your hosted solution and it was the same as mine. I think that confirms it's not just a simple matter of setting up a cron job.

[sasiko]

no but you can use crontab -e to automatically do the job for you.

[Tom_K]

2 minutes ago, maidos said:

no but you can use crontab -e to automatically do the job for you.

Who were you replying to? I stated cron was setup to perform the post-migration.

[Charles]

14 hours ago, Tomzl said:

I upgraded from v3 to v4 (test upgrade to see how it goes). The upgrade itself was done fairly quickly but the issue is the post-upgrade tasks.

I have a board with 2.5 million posts and 20.000 members. It's running on a dedicated server and it's the only site on the server. It took more than 7 days for the post-upgrade tasks to finish (post conversion and other stuff).

This is totally unacceptable. The main reason is the script converts 50 posts at a time. I can understand this is reasonable for smaller forums but with 2.5 million posts? Come on. My server was idling when it could do the job much faster.

In those 7 days most posts and quotes were broken. I don't know how I can ask my members to wait out a week to be able to get proper functionalities back.

How can a simple upgrade to a newer version take more than a week to complete?

I am surprised it took that long. I did a site of a similar size yesterday morning and it's at 63% complete on the rebuild tasks so I would expect it will be done by tomorrow morning which puts it around 48 hours. I do large site upgrades all the time and they never take 7 days to complete their tasks.

You are correct that the script does 50 posts at a time however you are missing that the script continues to loop 50 at a time for 45 seconds of execution. So it will do 50 cycles as many times as your server can do in 45 seconds. If you are using the cron running every minute then it's only idle for 15 seconds before it starts the next batch.

I am sorry it took your system 7 days to process but that is not the norm in my experience of doing them. Of course maybe we just have powerful servers processing these.

[RevengeFNF]

14 hours ago, Tomzl said:

I upgraded from v3 to v4 (test upgrade to see how it goes). The upgrade itself was done fairly quickly but the issue is the post-upgrade tasks.

I have a board with 2.5 million posts and 20.000 members. It's running on a dedicated server and it's the only site on the server. It took more than 7 days for the post-upgrade tasks to finish (post conversion and other stuff).

This is totally unacceptable. The main reason is the script converts 50 posts at a time. I can understand this is reasonable for smaller forums but with 2.5 million posts? Come on. My server was idling when it could do the job much faster.

In those 7 days most posts and quotes were broken. I don't know how I can ask my members to wait out a week to be able to get proper functionalities back.

How can a simple upgrade to a newer version take more than a week to complete?

Did you speed up the conversion in the Admin CP? My board with 1,6 Million posts took less than a Day to complete. Your should took no more than 2 days. 

[TSP]

Neowin took close to or over a month I believe with their 11 million posts. 

I did a pretty much standard rebuild for a 3 million posts board with all the tasks completed in about 3 days. I  did however change the order the rebuilding was done in (so newest posts would be rebuilt first and thus decrease the likelihood of users noticing broken posts) and I decided to merge two tasks since it was a better use of time to have them work simultaneously in one task instead of two (because they both worked on the content column in the posts table and thus were both querying the posts table).

For a 15 million posts board I decided this was not sufficient at all. So I decided to create a script that would allow me to run the rebuild with a 1 million interval on our 7 different production servers with each their starting interval, and also made changes to the standard queue task to not try to rebuild a post twice if it should happen to "catch up". Rebuild of posts was then completed in about a day. 

[RevengeFNF]

3 minutes ago, TSP said:

Neowin took close to or over a month I believe with their 11 million posts. 

I did a pretty much standard rebuild for a 3 million posts board with all the tasks completed in about 3 days. I  did however change the order the rebuilding was done in (so newest posts would be rebuilt first) and I decided to merge two tasks since it was a better use of time to have them work simultaneously in one task instead of two (because they both worked on the content column in the posts table and thus were both querying the posts table).

For a 15 million posts board I decided this was not sufficient at all. So I decided to create a script that would allow me to run the rebuild script with a 1 million interval on our 7 different production servers, and also made changes to the standard queue task to not try to rebuild a post twice if it should happen to "catch up". Rebuild of posts was then completed in about a day. 

Did you share your changes with IPS? 

[TSP]

I have shared the changes to merge the two tasks I merged earlier and also to change the direction of the rebuild, but it wasn't picked up. 

As for the interval thing I did to have it be able to run the process from more than one php process feels more "hackish" and I would assume they would need to work on a more allround/client-friendly solution than what I did, so it's really nothing to share.

[CheersnGears]

Could IPS change the rebuild process to run from newest to oldest?

[White Miku]

12 minutes ago, CheersnGears said:

Could IPS change the rebuild process to run from newest to oldest?

This is not a solution.

Big topics will be partially corrupted while rebuilding in process.

[TSP]

1 minute ago, CV01 said:

This is not a solution.

Big topics will be partially corrupted while rebuilding in process.

How come? 

I've not noticed any problem with this method. And as far as I can judge from the code, the rebuild process is not dependent on earlier posts being rebuilt correctly. It just rebuilds the current post, the state of other posts doesn't matter. 

[CheersnGears]

3 minutes ago, CV01 said:

This is not a solution.

Big topics will be partially corrupted while rebuilding in process.

That doesn't even make sense....  if that were true, then those same big topics are being corrupted during the current process, just in the opposite order.

[JordanRash]

Same issue and IPB was no help. We have around 500k posts and 1M members. It took us nearly a full day and the background processes have been running for 2 days while our server idles at 3% usage.

[Daniel F]

5 minutes ago, JordanRash said:

Same issue and IPB was no help. We have around 500k posts and 1M members. It took us nearly a full day and the background processes have been running for 2 days while our server idles at 3% usage.

Have you tried to run the background tasks yourself? This method will finish much faster 

[JordanRash]

1 minute ago, Daniel F said:

Havemyou tried to run the background tasks yourself? This method will finish much faster 

We have had 4 admins doing it at once. I think it should be looked into how to speed it up regardless of what I'm doing however. I think we are suggesting that IPB makes it faster, or an option to make it faster for larger servers. We don't want the blame to be put on us, it's slow, and we all know it.

[Tom_K]

Can you please advise how to speed up the process? I setup the cron to run the tasks. Is there anything else that can be done?

Judging by the replies there is a lot of negative feedback when upgrading to v4. IPB says the migration works fast but feedback from hosted clients says otherwise and I'm concerned about the feedback how users don't like v4. IPB are you listening to this? It seems many large boards are having trouble.

<mod edit: please don't post PMs>

[Tom_K]

On 6/9/2016 at 0:18 PM, Charles said:

I am surprised it took that long. I did a site of a similar size yesterday morning and it's at 63% complete on the rebuild tasks so I would expect it will be done by tomorrow morning which puts it around 48 hours. I do large site upgrades all the time and they never take 7 days to complete their tasks.

You are correct that the script does 50 posts at a time however you are missing that the script continues to loop 50 at a time for 45 seconds of execution. So it will do 50 cycles as many times as your server can do in 45 seconds. If you are using the cron running every minute then it's only idle for 15 seconds before it starts the next batch.

I am sorry it took your system 7 days to process but that is not the norm in my experience of doing them. Of course maybe we just have powerful servers processing these.

What is the weak spot for these tasks? Processor, memory, HD speed?

On 6/9/2016 at 0:21 PM, RevengeFNF said:

Did you speed up the conversion in the Admin CP? My board with 1,6 Million posts took less than a Day to complete. Your should took no more than 2 days. 

How can you speed up the conversion in Admin CP? I just setup the tasks to be run by cron.

[Lindy]

I'm looking into this. For performance reasons (the way MySQL handles data) - rebuilding content newest to oldest has been significantly slower in previous tests (some may recall this from 4.0) however, I think we could safely populate the search index newest to oldest. 

Please keep in mind, from a user standpoint, you expect this to be an upgrade and I completely understand that. From a technical perspective, however, it's a conversion. It's literally taking each post, some of which have used iterations of data from much older versions, even different software, and converting them to a consistent format. We also have to unfortunately account for the lowest common denominator and bottom barrel hosts, that's why we're fairly conservative with the number per cycle. It wasn't that way in early builds and the complaints were instead "the rebuild is crashing my server!!" We do our best at seeing what your server can handle, but we need some sort of baseline. If it's taking a week to process a couple million posts, it's because IPS4 has determined processing more quickly would tax your server and impact performance. We can look at a "do it anyway" kind of system that you can manually do, but we need to figure out a way to handle that from a support standpoint when you set your server on fire and the rebuild breaks. 

I do agree that handling newest to oldest would be ideal and you'd care less if it took several days for content from 2009 to appear. We'll look into the performance impact of this again. 

Some of the shared experiences just aren't typical. We've done NHL and NFL teams with millions of posts in 1-2 days. With rare exception, I'm not sure we've had any hosted site take that long. David's case was a bit unusual as he literally has thousands of forums, a wide array of plugins and custom development, etc. 

One thing we've done for the next release is add a notice while background tasks are running to let users know this is what's happening. That coupled with looking into newest to oldest should help immensely, I would think.

[TSP]

3 hours ago, Lindy said:

I'm looking into this. For performance reasons (the way MySQL handles data) - rebuilding content newest to oldest has been significantly slower in previous tests (some may recall this from 4.0) however, I think we could safely populate the search index newest to oldest. 

Not something I've experienced when doing this, but you obviously have to utilize and store the primary key you last rebuilt, like you do already. If you do that, I can't see any reason it should be slower than doing oldest to newest. As far as I remember, in your first versions you used offsets for the retrieval of the posts to rebuild, which obviously will be slow (and unreliable because you end up attempting to rebuild posts multiple times when new posts come in). So I suspect you're comparing apples and cars here. 

Here is the changes I did last time: 

You would probably need to add some code to account for cases where someone upgrades to this version with a new way to rebuild and they have an active process of rebuilding though. You could add some information to the queue item data after you've determined it based on previous versions and whether there is any queue taks for the rebuild, and then wrap the method it should use in an if/else block that checks the queue item data for whether it should use new or old method.