How to speed up conversion?

[AaronP]

We're converting from VB and doing a test first to make sure the end result looks good. We started this process over a month ago and the conversion is still processing. Right now the "Processing RebuildContent" is on day 4 or so of close to ~4 million posts.

This conversion test is being done locally on XAMPP, with 64GB RAM and an i9-7960X, and neither are being taxed (CPU is at 10%, Memory at 16% usage). I can't imagine our server(s) would be much faster.

My question is, is there any way to speed this process up? It seems like hardware isn't really the problem. Any ideas are greatly appreciated!

Thank you

[bfarber]

You say you "started this process" over a month ago, but how long has the actual conversion been running? It definitely should not take anywhere near that long. I've seen sites with tens of millions of posts converted in just a day or two.

Note that this does NOT include background tasks (like "Processing RebuildContent"). The expectation is that you can turn your site back online while those continue to process and rebuild in the background. A few things may not work 100% or look 100%, but your site will be usable for your users and certain non-critical things like parsing bbcode can occur in the background after the site is back online.

[AaronP]

12 hours ago, bfarber said:

You say you "started this process" over a month ago, but how long has the actual conversion been running? It definitely should not take anywhere near that long. I've seen sites with tens of millions of posts converted in just a day or two.

Note that this does NOT include background tasks (like "Processing RebuildContent"). The expectation is that you can turn your site back online while those continue to process and rebuild in the background. A few things may not work 100% or look 100%, but your site will be usable for your users and certain non-critical things like parsing bbcode can occur in the background after the site is back online.

Thank you for the response. It has been running non-stop with the exception of having to restart my computer a couple times. I'd say at least 2 weeks of it was processing attachments (we have millions of them - mostly photos), and it would take about 20 seconds to go through 10 of them...

[jesuralem]

My site is not as big as yours so this is not an issue for me, but when I do my test migrations it is clear the the CPU and RAM are not fully used.

I would think convertion speed could be improved by having more threads going simultaneously, maybe letting the user choose how many thread to start.

But I don't really know how it works and if this is doable.

[bfarber]

Web software does not really give you that same level of control as if you were developing a computer program - we can't really control how much memory or CPU gets used (although you may have more control over this by adjusting settings for PHP, Apache, MySQL and so on).

A major performance improvement was added to 4.4.5 so it's possible upgrading to this version will help speed up conversions overall, however I'm not sure it'll be a "magic bullet" for you. Converting 2 million attachments can indeed be a slow process depending upon your configuration, although it shouldn't take "2 weeks", nor should everything else take "2 weeks". I would recommend upgrading to 4.4.5 for your conversion which will help either way, and ensuring your server configuration allows the server software to use more resources as a starting point.