Doing Test Conversions

[The Dark Wizard]

Hey All,

I'm doing a test convert on a xenforo site that has 12 million posts and about 19 million PMs and the convert seems slow relative to the board size, so any convert tests or eventually the live site, will be a while.

I would appreciate any advise or conversation on possibly altering the convert application, we have the resources and hardware to spare to crunch this conversion.

 

[Marc]

Increasing timeouts on mysql. Ensuring memory_limit is set high on your PHP installation. With background tasks, you may speed the background tasks up by changing the rebuild options but be sure your server can handle before doing this. Other than that however, the conversion is doing quite a lot and can indeed take some time to complete

 

[The Dark Wizard]

6 hours ago, Marc Stridgen said:

Increasing timeouts on mysql. Ensuring memory_limit is set high on your PHP installation. With background tasks, you may speed the background tasks up by changing the rebuild options but be sure your server can handle before doing this. Other than that however, the conversion is doing quite a lot and can indeed take some time to complete

 

Thanks for your reply!

Seems like Constants.php and the rebuild options will be good at the end of the conversion when we get to that step. Currently it's been about 20ish hours, maybe a little more, here's what it has done.

I was looking at the convert application files and I noticed that in Forums.php and Core.php the batch is controlled by the per_cycle setting. If I alter those, will it do higher batches or is it controlled in other places as well?

[Marc]

It will do, however I wouldn't suggest changing these. You are best leaving it as they are and awaiting its completion. There are a significant number of PMs there, so it will indeed take quite some time to complete. 

[Ghan]

One thing I discovered during the import is that the database table statistics need to be updated. The query planner was using the wrong index for this query of the convert application:

SELECT ipb_id FROM `convert_link` WHERE ( type IN('core_members') ) AND foreign_id='1667' AND app=1 ORDER BY link_id DESC

It was trying to use the "app" index and the query was taking around 3 seconds to execute. After updating table statistics by running ANALYZE TABLE against convert_link, it now runs in 0.0002 seconds.

Likely the statistics got behind because of all the data being inserted quickly.

[Marc]

Just now, Ghan said:

One thing I discovered during the import is that the database table statistics need to be updated. The query planner was using the wrong index for this query of the convert application:

SELECT ipb_id FROM `convert_link` WHERE ( type IN('core_members') ) AND foreign_id='1667' AND app=1 ORDER BY link_id DESC

It was trying to use the "app" index and the query was taking around 3 seconds to execute. After updating table statistics by running ANALYZE TABLE against convert_link, it now runs in 0.0002 seconds.

Likely the statistics got behind because of all the data being inserted quickly.

That sounds more an issue with your mysql instance than anything else there

[The Dark Wizard]

3 minutes ago, Ghan said:

One thing I discovered during the import is that the database table statistics need to be updated. The query planner was using the wrong index for this query of the convert application:

SELECT ipb_id FROM `convert_link` WHERE ( type IN('core_members') ) AND foreign_id='1667' AND app=1 ORDER BY link_id DESC

It was trying to use the "app" index and the query was taking around 3 seconds to execute. After updating table statistics by running ANALYZE TABLE against convert_link, it now runs in 0.0002 seconds.

Likely the statistics got behind because of all the data being inserted quickly.

Made a night and day difference for us.

2 minutes ago, Marc Stridgen said:

That sounds more an issue with your mysql instance than anything else there

This is why it's conversion testing! It won't be the only one up until we are ready to do the live site.

[Marc]

14 hours ago, The Dark Wizard said:

This is why it's conversion testing! It won't be the only one up until we are ready to do the live site.

Of course. And the correct thing to do 🙂 

[The Dark Wizard]

We have completed our first migration test, I'm sure we will do several more before we even attempt the live site.

I've noticed the long list of background tasks that have appeared, but something stood out to me:

 

"Rebuilding Personal Conversations" was at an instant 100 percent, should I be concerned about that?

[Randy Calvert]

If your users have not sent a lot of PMs, it won't take long at all for it to rebuild.  There's a TON of things rebuilding...  some complete super fast and others will drag on for infinity.  🙂

I pulled up another tab and ran the task queue manually while I worked in other windows to help make me feel better about everything finishing faster.  lol

[The Dark Wizard]

1 minute ago, Randy Calvert said:

If your users have not sent a lot of PMs, it won't take long at all for it to rebuild.  There's a TON of things rebuilding...  some complete super fast and others will drag on for infinity.  🙂

I pulled up another tab and ran the task queue manually while I worked in other windows to help make me feel better about everything finishing faster.  lol

19 million pm replies.

That shouldn't have finished faster then anything else.

[Randy Calvert]

Well, you'll be able to find out how things went by looking at your PMs.  😄

There just may not be much to do with it and the job could blow through it quickly.  

[The Dark Wizard]

1 minute ago, Randy Calvert said:

Well, you'll be able to find out how things went by looking at your PMs.  😄

There just may not be much to do with it and the job could blow through it quickly.  

PMs seem to be fine.

However, it's also doing tasks in order, so I was just concerned is all.

[Randy Calvert]

It actually does not do every task in order.  It rotates through small amounts of each based on my past experience.  

For example, it might do 200 posts, then some topics, etc.  If you run the tasks manually you'll see what I mean how it will do some of each.  🙂 

[The Dark Wizard]

1 minute ago, Randy Calvert said:

It actually does not do every task in order.  It rotates through small amounts of each based on my past experience.  

For example, it might do 200 posts, then some topics, etc.  If you run the tasks manually you'll see what I mean how it will do some of each.  🙂 

Currently, in attachment hell:

 

I haven't seen it do any other ones yet. All the other dozen+ tasks are at zero since it's focusing ona ttachments.

[Randy Calvert]

It will eventually hop over to something else for a bit.  I've given up trying to figure it out to be honest.  

(By the way, the other thing I've seen is that even when the dashboard says "100%", it is not quite there.  It' rounded up and has a couple left until it actually disappears.  Once the item actually disappears, you know it's really finished.  🙂 

[teraßyte]

It does tasks depending on the priority (from 1 to 5). If attachments has a higher priority than anything else it will do only that first.

As for the 100% on PMs, given how many you have I doubt it has already rebuilt them only. Most likely a bug in how the complete % is calculated.

Edited July 8, 2022 by teraßyte

[The Dark Wizard]

4 minutes ago, Randy Calvert said:

It will eventually hop over to something else for a bit.  I've given up trying to figure it out to be honest.  

(By the way, the other thing I've seen is that even when the dashboard says "100%", it is not quite there.  It' rounded up and has a couple left until it actually disappears.  Once the item actually disappears, you know it's really finished.  🙂 

 

4 minutes ago, teraßyte said:

It does tasks depending on the priority (from 1 to 5). If attachments has a higher priority than anything else it will do only that first.

As for the 100% on PMs, given how many you have I doubt it has already rebuilt them only. Most likely a bug in how the complete % is calculated.

 

We will definitely see when it tries to run that task!

Something else I noticed is that my account didn't get my avatar, or my original join date, but it gave me my PMs and posts, I'll see if anything improves when these catch up.

[Jim M]

12 hours ago, The Dark Wizard said:

Something else I noticed is that my account didn't get my avatar, or my original join date, but it gave me my PMs and posts, I'll see if anything improves when these catch up.

Did you create your Invision Community admin account with the same details as your account from your converted forum? If so, that is intended. The system will do it's best to merge accounts that exist already. However, will keep existing items in place like join date, profile photo, etc.. but add in new data like posts, messages, etc...

[The Dark Wizard]

4 hours ago, Jim M said:

Did you create your Invision Community admin account with the same details as your account from your converted forum? If so, that is intended. The system will do it's best to merge accounts that exist already. However, will keep existing items in place like join date, profile photo, etc.. but add in new data like posts, messages, etc...

I did use the same details yes.

[Jim M]

32 minutes ago, The Dark Wizard said:

I did use the same details yes.

Then what you’re experiencing is intended with the profile photo, join date, etc… Please let us know if you have any further questions.

[The Dark Wizard]

Out of curiosity, which background task fixes the Last Poster Box Thread IRL

You'll notice in my next image, that while I have the thread in the above picture selected, that the URL/ID via the last poster box is zero.

So,  when I click on a thread via the last poster box, it shows no permissions as expected.

But if I actually enter the forum topic list:

I'll get the proper URL:

From the topic list, I can enter the topic properly, read its contents etc.

I assume one of the background tasks currently running will fix the last poster box?

Remaining Tasks:

[Jim M]

Search index is responsible for those. You'll want to ensure all that gets rebuilt as there's a lot of rebuilding there.

[The Dark Wizard]

1 minute ago, Jim M said:

Search index is responsible for those. You'll want to ensure all that gets rebuilt as there's a lot of rebuilding there.

Good to know, thank you.

Wanted to make sure there wasn't an issue from the start, as we prepare in the foreseeable future to wipe the conversion and start again.

[The Dark Wizard]

It seems that all quoted names in PMs mess up.

This does not seem to be the case for topics.

It also seems mentions break in PMs but correctly reformat in topics as well

In PMs, all things that are supposed to be mentions turned into uSER for some reason.

All mentions work correctly in posts: