Upgrade to v4 took more than 7 days to complete!

[TSP]

@Lindy: And here is what I did to merge two background tasks into one. I had then earlier mentioned this in a bug report which was closed.

PLEASE NOTE: This was only done to account for the forums application, as that was all I had. IPS would need to review whether it would be possible to then remove the corresponding RebuildContentImages-tasks for their other apps too.

Since this is in the same method though, it could use the same queue item data you would have to add based on my previous suggestion in my previous post here, to help determine inside the task php file whether it should use the new method or not. 

commit 
Author: 
Date:   Sun Aug 30 17:00:56 2015 +0200

    Merge two rebuild tasks

diff --git a/www/applications/core/extensions/core/Queue/RebuildPosts.php b/www/applications/core/extensions/core/Queue/RebuildPosts.php
index 8207c64..95def6a 100644
--- a/www/applications/core/extensions/core/Queue/RebuildPosts.php
+++ b/www/applications/core/extensions/core/Queue/RebuildPosts.php
@@ -191,6 +191,14 @@ class _RebuildPosts
                }
            }

+           /* HW_CUSTOM merged two tasks into one. Below code retrieved from RebuildContentImages.php */
+           $rebuilt = \IPS\Text\Parser::rebuildAttachmentUrls( $item->$contentColumn );
+
+           if( $rebuilt )
+           {
+               $item->$contentColumn = $rebuilt;
+           }
+
            $item->save();

            $last = $item->$idColumn;
diff --git a/www/applications/forums/setup/upg_100028/upgrade.php b/www/applications/forums/setup/upg_100028/upgrade.php
index e96d996..79c8617 100644
--- a/www/applications/forums/setup/upg_100028/upgrade.php
+++ b/www/applications/forums/setup/upg_100028/upgrade.php
@@ -33,9 +33,10 @@ class _Upgrade
    public function finish()
     {
        \IPS\core\Setup\Upgrade::repairFileUrls('forums');
-       \IPS\Task::queue( 'core', 'RebuildContentImages', array( 'class' => 'IPS\forums\Topic' ), 3, array( 'class' ) );
-       \IPS\Task::queue( 'core', 'RebuildContentImages', array( 'class' => 'IPS\forums\Topic\Post' ), 3, array( 'class' ) );
-       \IPS\Task::queue( 'core', 'RebuildContentImages', array( 'class' => 'IPS\forums\Topic\ArchivedPost' ), 3, array( 'class' ) );
+       /* HW_CUSTOM merged two tasks into one. Commented out RebuildContentImages queue items */
+       //\IPS\Task::queue( 'core', 'RebuildContentImages', array( 'class' => 'IPS\forums\Topic' ), 3, array( 'class' ) );
+       //\IPS\Task::queue( 'core', 'RebuildContentImages', array( 'class' => 'IPS\forums\Topic\Post' ), 3, array( 'class' ) );
+       //\IPS\Task::queue( 'core', 'RebuildContentImages', array( 'class' => 'IPS\forums\Topic\ArchivedPost' ), 3, array( 'class' ) );
        \IPS\Task::queue( 'core', 'RebuildNonContentImages', array( 'extension' => 'forums_Forums' ), 3, array( 'extension' ) );

         return TRUE;

 

[Lindy]

The search index and post rebuilding routines will now complete from newest to oldest. That should solve most of your concerns. 

[RevengeFNF]

30 minutes ago, Lindy said:

The search index and post rebuilding routines will now complete from newest to oldest. That should solve most of your concerns. 

Best today news regarding the Search Index

[Tom_K]

Is that modification already implemented in latest build available?

Can I speed up the post-transfer process somehow? I don't even know what the bottleneck on the server is. It has more than enough resources to do it faster. Migration from vBulletin to IPB3 was done in a few hours so this should take similar time?

[Jed Rosenzweig]

15 hours ago, Lindy said:

Some of the shared experiences just aren't typical. We've done NHL and NFL teams with millions of posts in 1-2 days. With rare exception, I'm not sure we've had any hosted site take that long. David's case was a bit unusual as he literally has thousands of forums, a wide array of plugins and custom development, etc. 

Yes, we basically have a forum for every TV show people want to talk about. If forum counts affect the conversion time then I'm surprised it isn't still converting.   We are an edge case in that respect to be sure.

The really tough part of this is that a site in the middle of a multi-day conversion can give a poor first impression. If there was a way to have copied the 3.x install to a private instance for a couple days while it converted to 4.x then do another conversion/addition of the posts made during that couple days, that would have been great as we could have just closed the forum for the hour or two the couple-day-conversion took place. I'm sure that's very complicated to do on the backend but if that was available, I would have taken advantage.

[Lindy]

3 hours ago, Tomzl said:

Is that modification already implemented in latest build available?

Can I speed up the post-transfer process somehow? I don't even know what the bottleneck on the server is. It has more than enough resources to do it faster. Migration from vBulletin to IPB3 was done in a few hours so this should take similar time?

I'm afraid not. It was just pushed to development this afternoon. It will still need to go through QA, etc. I'm uncertain if we could get you a patch to test (entirely at your own risk, yada yada, etc.) but I can check, if you're interested and comfortable applying it.

[Tom_K]

I updated the version on my test install for the first time and now it's reindexing posts which looks like will take 2-3 days. Does this mean every time I upgrade the version, these tasks will need to be run?

[marklcfc]

How long is a 4 million post board expected to take?

[AtariAge]

I would like to try this on my forum with 3.5 million posts.  I have a dedicated development server I can do a test conversion on to see how long it takes.  Having a conversion process run for a week (or even several days) in the background, leaving all "unconverted" posts in a broken state is unacceptable.  Is there no command-line version of this process that can be run, as opposed to this "50 posts at a time" in a browser business.  I'd really prefer to have whatever this conversion is doing complete before I put the converted forum back online, which would allow me to use the full power of the server to perform these changes. 

 ..Al

[Tom_K]

I'm still only at 36% of reindexing messages after the upgrade from 4.1.x to the latest version (4.1.12). While I wait for the reindexing to complete my activity stream shows only topics from 2009. Basically the board shows almost no activity prior to the upgrade (only topics from 2009 and those with changes after the upgrade). This seems to only affect listing of topics.

@Lindy This can't be the way upgrades work, right? Why do posts need to be reindexed after a minor version upgrade? This means several days downtime not just when making the big jump from v3 to v4 but for each minor upgrade?

[Sai5i]

@Tomzl @Lindy Is it really need to re-index all posts for future updates or just one time problem with v3 to v4 conversion?

I have been planning to upgrade my board with about 100K posts and i'm not sure if it's worth if i have to go through the trouble again and again.. But if later minor updates will take normal time similar to earlier versions.. Then i can go with initial conversion..

And i'm on a shared hosting package on dreamhost, which is running several sites.. All others are wordpress sites and have quite a bit of issues because of shared hosting but i never had any problem with IPB.. It's running very smoothly.. No complain whatsoever from any user.. I'm bit worried whether v4 will make this performance more better or it could cause issues which are unseen in v3... Would appreciate If anyone know server resources comparison between v3 and v4 or point me to right direction..

[AtariAge]

I'm bumping this topic up because I am now trying to convert my 3.4.8 forum to 4.4.2, and this business of rebuilding content in the background is driving me crazy.  I have over four million posts and it hasn't even gotten to the "Rebuilding Posts" task after a day.  As the original poster said, this is simply unacceptable.  I have now setup the cron job to run these tasks, but it seems my server is mostly idle while it's doing this and I rarely see MySQL actually doing anything.  And all the progress bars showing progress are barely eking along, with most of them not even started.   I've also received am periodically receiving this notice from my server: 

I seem to be seeing this exactly every 16 or 17 minutes.. 

 

 

Surely there has to be a better way of doing this.  There is no direct way for me to monitor what is going on.  I want to use the full power of the server to process these tasks in the foreground without pauses.  How can I make these tasks all run in the foreground, in sequential order?  At this rate I'll be lucky if it completes in a week, and I'm not even confident they are running properly since I am sometimes seeing that very vague, "Something went wrong. Please try again." message from Cron.  The task is setup as follows: 

* * * * * /opt/cpanel/ea-php72/root/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/atariage/public_html/forums/applications/core/interface/task/task.php d4d5b62ec318f6affa4b428af3057577

If I run the task on the command-line it returns immediately without an error.

I'd like to state this is a brand new, dedicated server that has no other sites running on it, only my IP.Board 4.4.2 test site.  The site and database are on a PCIe NVMe SSD, and the server has 32GB of RAM, of which a significant amount is dedicated to MySQL.  The UTF8 converter and the queries during the (lengthy) upgrade process ran fairly quickly, even the queries against the post stable, which has over four million rows.  

Thank you, 

..Al

[AtariAge]

As an example, here are tasks that are currently, slowly, making progress..

Rebuilding signatures is something that should take no time at all.  Same with reports.  I only have 50,000 members, and most people don't have signatures, so I'm just dumbfounded that these tasks take so long to run. There has to be a better way to do this. 

In the 13 minutes that past since I took the above image, not a single one of these tasks has progressed at all.

 

[maddog107_merged]

@AtariAge

So not sure if this will help you, but cron * * * * *  is once a minute as you know. So it does 50posts per minute which is super slow. We have it running every 10 seconds by doing this (hack).

* * * * * ( sleep 10; /opt/remi/php72/root/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /var/www/html/main/applications/core/interface/task/task.php abcdefghijksssaasasasas)

* * * * * ( sleep 20; /opt/remi/php72/root/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /var/www/html/main/applications/core/interface/task/task.php abcdefghijksssaasasasas )

* * * * * ( sleep 30; /opt/remi/php72/root/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /var/www/html/main/applications/core/interface/task/task.php abcdefghijksssaasasasas )

* * * * * ( sleep 40; /opt/remi/php72/root/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /var/www/html/main/applications/core/interface/task/task.php abcdefghijksssaasasasas )

* * * * * ( sleep 50; /opt/remi/php72/root/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /var/www/html/main/applications/core/interface/task/task.php abcdefghijksssaasasasas )

 

We were able to do our 4mil posts in ~1.5days. Hopefully it can help you. 

[AtariAge]

Thanks @maddog107_merged, is it safe to run these tasks like this?  What happens if one task does not finish before the next one starts?  

@Stuart Silvester did give me a useful heads up that Xdebug can greatly slow down PHP, so I disabled it for now.  I'm using that as I'm having to rewrite a significant amount of older code on another portion of my site so it'll run on PHP 7.2.  It was definitely slowing the site down, so disabling Xdebug did help.  The tasks queue is still churning pretty slowly, although better now.  Still, at this point I am only at 4.36% of my posts and 12.78% of my private messages, and there's still a ton to rebuild after that.  Content is still broken in posts, such as attachments, no gallery images are visible, and lots of other miscellaneous things. 

I noticed double-spaced content not just in posts, but blog comments, signatures, and elsewhere.  Not crazy that my choices here are to leave it as is, change the behavior of the return key to go to the "next line" instead of paragraph, or run a script(s) to fix these issues (which isn't 100% reliable, although it would likely fix most of the issues, while still allowing me to use the preferred new behavior of the return key).  How did you deal with this issue? 

Where I currently stand: 

 

[The Old Man]

Hi @AtariAge

Congrats on starting the test upgrade. I don't have a bazillion posts like yourself but have you seen the constants file options, just in case they can help? You can override for large communities.

 

Also, I wondered if making temporary use of cloud computing solutions from Amazon or Google Cloud could somehow help to power through the data processing tasks, even if you went back to the dedicated server setup after the conversion had finished?

One thing my cloud VPS Hosting provider pointed out (I have access to 48 unlocked cores but only 3GB/6GB burstable RAM) was that a few months back when I had 1.5GB (burstable when needed to 3GB), the IPS Cron task concerned them greatly, because by default it turns off the memory limit potentially not leaving enough memory for the actual server itself to run the OS.

Possibly turning off unnecessary logging may help a little too.

Good luck!

 

[AtariAge]

HI @The Old Man, thanks for your suggestions.  I had looked at the available constants.php values, and the ones relevant to this stag of the upgrade are: 

• Constant - REBUILD_SLOW
  Use - Number of items to be rebuilt per-cycle for routines that take a while (change only if you are 100% sure your system will cope)
  Example value - 50
   
• Constant - REBUILD_NORMAL
  Use - Number of items to be rebuilt per-cycle for most routines (change only if you are 100% sure your system will cope)
  Example value - 250
   
• Constant - REBUILD_QUICK
  Use - Number of items to be rebuilt per-cycle for routines that are fast (change only if you are 100% sure your system will cope)
  Example value - 500

I looked through all the tasks, and all three of these constants are used depending on the actual task.  The RebuildPosts task uses REBUILD_SLOW, for instance.  I don't know what might happen if I changed the value of REBUILD_SLOW (for instance) to 250 or higher, but, hey, this is a test server (at the moment) so I could always try it.  I could also just modify the value in specific files, such as RebuildPosts.php.  I don't fully understand how these tasks run, I've looked a bit at the code and it's somewhat involved.  What I wish I could do is for a few of these more important tasks just let them run full-bore on the command-line until they complete.  Especially for the posts table, messages, and attachments.  I know this is playing with fire a bit.  

Using a cloud-based VPS provider is an interesting premise, but seems more trouble than it's worth for possibly minimal benefit.  

 ..Al

[Sonya*]

@AtariAge, we have over 10 millions posts and conversion hat last almost 6 weeks. We have a dedicated server and a busy site. The rebuilding processes were running in the background after upgrade. So the server has to rebuilt and do normal work at the same time. There a little to nothing you can do about it. The only thing that is really annoying you will see BBCODE on the non-rebuilt content and it will disappear gradually till rebuilt is done. It means you and your user can use the website normal way after upgrade, but the content looks ugly and the images are not shown. It is annoying.

[bfarber]

Ok, it's important to clear up some misconceptions here.

Firstly - the fact you are/were getting an error every 15 minutes means some cron task is failing (I am guessing the queue) at some point, and that is almost certainly part of why the rebuild is moving slower for you. If the software attempts to rebuild posts and hits an error, it stops (and skips that post on the next run). This would significantly slow down the overall rebuild time. You can likely check the system logs to determine the cause.

Adding additional cron tasks will absolutely not help speed up the queue. First of all, the queue task does not strictly rebuild 50 posts per cycle. The queue task starts, grabs 50 posts to rebuild, and once done if there is enough time it will do it again (and again and again until we're running out of time). The task will process for up to 90 seconds, so on smaller posts and what not you may be able to rebuild 1000 in one cycle, it really just depends upon the power of the server. Secondly, when the task starts processing it locks itself, and then unlocks itself once complete, so if you have a second cron trying to execute the same background task while the first cron is already executing it, it simply won't do anything since the task is locked.

The constants that were pointed out can be redefined in constants.php to increase the number per cycle, but honestly you're probably not going to see much difference there.

Finally - rebuilding posts is absolutely the slowest part of upgrading. Unfortunately, it takes time to do this, and many other things simply can't be done until this is complete (for instance, the content stored in the search index requires posts to be "normal" or rebuilt first, so your search index won't even start building until the rebuild posts is done. Basically, once rebuilding the posts completes those other tasks are going to move along a bit quicker...they don't all take the same amount of time to complete.

 

And then of course, something like XDebug which is entirely outside of our control will have an impact too.

 

My recommendation is that if you think there's a problem (the emails might indicate as such) to follow up via a ticket, but there's not really any way for us to speed this process up at this time. And it's also worth remembering that this happens one time going from 3.x to 4.x - your posts are never fully rebuilt again afterwards, as there is no need.

[AtariAge]

Thank you Brandon for your detailed response.  I've just been running the single Cron job, I haven't tried any of the other suggestions, as tantalizing as they sound.  

Yeah, Xdebug was obviously my fault entirely, as I didn't realize it had such a big hit on performance.  Lesson learned there.  I had it installed for reasons unrelated to my forum, and I'll make sure it's not enabled when I convert the live forum (hopefully reasonably soon).  

I may experiment with the REBUILD_SLOW constant just to see what happens.  I can watch the core_queue table to see how quickly it's churning through rows.  My expectations are low, but since this is a test server (at the moment), I can afford to play around. 

I'll probably do another test conversion start to finish just to make sure there are no issues, given the various hiccups I had this first time.  In particular, if one of the queue tasks was generating the error I was seeing, I'd like to dig into that to see what was happening.  So I'll go through the logs and see if I can find anything that corresponds with the time those events were occurring.  I also want to time how long everything takes so I have reasonable expectations when I do the live conversion (which I will be watching like a hawk).  Fortunately I won't be doing that work on the current production server, so if something goes awry, I can easily put the forum back online.  

Thanks again for your feedback. 

 ..Al

[AtariAge]

My posts table finally finished rebuilding overnight.  The rather lengthy list of tasks following it also seem to have completed (the only one I saw it building this morning when I awoke was "Building Posts Index" or something along those lines, and that went fairly quickly).  Is there any way to view a log of the tasks that were completed?

I need to spend some time looking through the site for any problems.  Right off the bat I can see that no Gallery images are visible at all. 

I'm also seeing some broken avatars.  And some other broken things, such as this: 

Also, it looks like the forum completely nuked my unread posts pointers, as all the forums on the main index are showing as read.  Viewing the "Unread Content" link confirms that this was all nuked for me. This is disappointing if that's going to happen for all members.

[bfarber]

Yeah unfortunately from here I couldn't really say what would cause those issues. I would test on the default theme (if your custom theme doesn't support lazy loading properly it could potentially cause the image issues you are describing) but otherwise you'd need to submit a ticket so we can investigate if you don't see the causes on your own.

[AtariAge]

Stuart already took care of the Gallery problem and said that should be fixed in an upcoming version.  Not sure why the link in the description doesn't work -- I removed it from my 3.4.8 forum so I don't have to deal with it when I upgrade the live forum.  And Stuart confirmed that the upgrade from 3.4.x to 4.x does not preserve the topic read markers.  

Now it's time for me to read through the development docs so I can properly code some existing modifications I have in 3.4.8.  The existing mods were done by directly modifying the code, but I want to avoid doing that going forward.  :) 

I'm going to start another upgrade in a few days, this time using the actual database server, which is a separate, dedicated box in the same datacenter with a private connection to the web server.  Curious to see if that makes any difference in the upgrade speed (and background tasks) since now there's network traffic involved.  The database server is still using spinning metal disks, unlike the new web server where the OS and accounts are sitting on a shiny NVMe SSD.  :D  However, the database server does have 64GB of RAM, and most of that is allocated to MySQL.  

Thanks, 

 ..Al