Very Slow Rebuilding Posts and Messages After 4.5 Upgrade

[bizman]

Hello, we recently upgraded our forum from V3 to V4.5. The upgrade was flawless, lasted for around 3 days. We thought we could finally turn the board online but then the software started rebuilding everything! Believe me when I say SLOW. The rebuilding process is progressing at 5% in a 24 hour period! That's 20 days offline! The good thing we copied everything first to another VM and made a subdomain classic.website.com. Our members are there now... the problem will happen when we turn on the main URL, we will have missing registrations and content for a month! hahaha

My question. How do we possibly speed this up? The server loads on the machines are 0.1 for the DB Server and 0.2 on the web server.

[Nathan Explosion]

What do you have set as your task method? To find out, type 'task' in the ACP search box and click on the relevant result.

[SeNioR-]

use cron, and set it to 30 seconds.

[CoffeeCake]

36 minutes ago, bizman said:

The server loads on the machines are 0.1 for the DB Server and 0.2 on the web server.

Consider adjusting the rebuild constants. See:

 

[Dean_]

Could also run it manually in a separate browser to speed it all up, this is what I do 90% of the time so it's done and out the way.

 

[bizman]

13 hours ago, CoffeeCake said:

Consider adjusting the rebuild constants. See:

 

<?php
\define('REBUILD_SLOW','50');
\define('REBUILD_NORMAL','250');
\define('REBUILD_QUICK','500');

Is this how it's supposed to be done? I just save this as a PHP file? I tried uploading this to the root but the server load remains very low. Thank you very much!

[CoffeeCake]

That documentation will tell you how to use constants.php, which is saved at the root of your IPS install as a file called constants.php. To see what the default values are, look at init.php in the root of your install. You can increase these values to sizes that make more sense for your environment.

You can change the value which will increase the number of items to process per iteration, yet you will want to test this on your test environment first before making a change in production.

It sounds like your server load would be able to handle larger increments from your statement that the load is very low, however making these changes without understanding them is dangerous, especially in a production environment. You'd want to ensure you had backups of everything first.

[Stuart Silvester]

Personally, I would recommend leaving those values as they are and making sure that you use cron to run the tasks. The tasks when ran via cron will process as much data as they can within the allowance of server resources.

Increasing these values may result in less data being processed over the same amount of time.

[Terry Ellison]

@Stuart Silvester I have my forum running on a 4-core VPS at v4.4 and about to cut over to a 6-core VPS at current 4.6.10.  My reason for getting stuck at 4.4 was historic (one of the then mods had customised our default template in a way that was dropped with 4.4, so we would have a look-and-feel bump.) I want to minimise actual cut-over The forum isn't that large (< ½M posts), but we are still looking at a ~1 day downtime. 

My forum LAMP stack is all docker containered, so spinning it up various test rigs is easy.  I am currently trial testing on the 6coreVPS and on a 4 core Ubuntu laptop which I use as a local dev machine. 

On 6/16/2021 at 7:08 PM, Stuart Silvester said:

The tasks when ran via cron will process as much data as they can within the allowance of server resources.

1. You application dashboard includes the following statement in the Background Processes pane when doing the conversion: "These processes are performed in the background in batches and may take a long time to complete. Alternatively, you can manually run them now and wait until they all complete. This wording implies that running manually will help speed the conversion, but so long as the cronjob has been correctly configured, This is terrible advice as doing so will slow down the conversion maybe 10×. It needs rewording.
2. Even when correctly configured, the conversion process is neither CPU nor IO bound.  There seems to be some back-off time throttling within the task.php architecture.  The forum will be offline during the conversion presenting a DoS to user community, so why not do as MediaWiki does and offer a batch script so do this, for example a "run at full rate until complete" option on task.php?

Edited February 9, 2022 by Terry Ellison

[Stuart Silvester]

22 hours ago, Terry Ellison said:

You application dashboard includes the following statement in the Background Processes pane when doing the conversion: "These processes are performed in the background in batches and may take a long time to complete. Alternatively, you can manually run them now and wait until they all complete. This wording implies that running manually will help speed the conversion, but so long as the cronjob has been correctly configured, This is terrible advice as doing so will slow down the conversion maybe 10×. It needs rewording.

Thanks for the feedback, I'll log an internal issue for consideration.

22 hours ago, Terry Ellison said:

Even when correctly configured, the conversion process is neither CPU nor IO bound.  There seems to be some back-off time throttling within the task.php architecture.  The forum will be offline during the conversion presenting a DoS to user community, so why not do as MediaWiki does and offer a batch script so do this, for example a "run at full rate until complete" option on task.php?

Don't forget, background processes (tasks) are designed to work in the background, they are designed to run when your community is open and available. Cron is the quickest, but depending on your knowledge you could also call the same path in a bash script if you wish. YMMV with that though.

What kind of background tasks do you have running, you shouldn't really have that many resource intensive tasks after an upgrade from 4.4. Although you do mention a 'conversion' so I'm not really sure if these are tasks from an upgrade, or tasks from a conversion from another software.

[Terry Ellison]

@Stuart Silvester you might be interested in the following:

 

[Marc Stridgen]

15 hours ago, Terry Ellison said:

@Stuart Silvester you might be interested in the following:

 

I'm not sure that really answers my colleagues question there. While I understand what you are doing, that will not change how many tasks there are from 4.4 to 4.6. What we are unsure of is if you have other conversion tasks outside of that upgrade process. 

[Terry Ellison]

3 hours ago, Marc Stridgen said:

I'm not sure that really answers my colleagues question there.

Sorry Marc, I am not sure what and where  your "there" refers to, though I will happily to any Qs that the IC team asks of me.

As I explained in my Migration Notes wiki page, migrating from 4.4 to 4.6 on a new server is complicated because of LAMP stack version and caching options incompatibility between these two IC versions.  This all gets messy if you have a native LAMP stack on the hosting server, because the standard (Ubuntu) packages don't comfortably support multiple version of this Stack installed or worse running at the same time without a lot of faffing around in the configuration, and so you end up doing serial installs and removal of packages, and these tend to /etc garbage.  Using docker addresses this whole issue as it is trivial to spin up then bring down separate docker-specific stacks.  I realise that IC-hosted forums are the commercially preferred option for IC, but in our community and forum are UK-based so want a UK hosted option; we also can't afford a five-fold increase in hosting fees (because we would pay a fair commercial price for such a managed service) on our no-advertising funding model.  I am sure that many self-hosting customers have similar drivers, which why I added this Self-Hosting topic.

After quite a few trial migrations, I think it entirely fair to describe your upgrade process as both unduly long and worse fragile in term of runtime.  For our forum, the best I've achieved is a 3½ hr elapse for our conversion time on a dedicated 6-core VPS with NVMe SSD, and that is working to a tight script.  Do the wrong click in one of the ACP screens and this can go up three-fold.  Using a Docker approach under GitHub makes the whole end-to-end more determined and maintainable.

[Marc Stridgen]

It was in reference to what I quoted of yours. My colleague asked what background tasks you have running that you are having issues with. That doesnt appear to have been answered

The items regarding your docker setup and the likes are really something for yourself to look at. We cannot assist with a custom setup such as what you have there. We do not support docker. You have stated you have an issue with the upgarde an dit taking a long length of time, but I have never seen an upgrade take 3.5 hours from 4.4 to 4.6. Hence the question from my colleague

[Terry Ellison]

@Marc StridgenLet me properly instrument my next trial conversion run for you.  Any feedback needs be supported by clear and hard data for it to be informative for you.  😊

[Terry Ellison]

@Marc Stridgen, I just did another dress rehearsal of my Migration from  v4.4.9 to v4.6.10. 

My host server is a 6-core VPS with 16Gb RAM and I've 4Gb allocated to InnoDB cache.  The migration took about 2 hr 46 mins, and the average CPU utilisation was under 1% for most of this period. The I/O rates were under 10 Mb/s writes which, given that the FS is on NVMe  is maybe 20× less than maximum throughput.  After about 30 mins of the conversion seeming to be stalled, I temporarily replaced the crontab job by a custom poll loop to execute task.php:

for ((i=1;i<1000;i=i+1)); do 
  declare -i start=$SECONDS
  php $opts $tasker $TASK_KEY 
  declare -i t=$(($SECONDS - $start))
  echo -n "$t:" 
  sleep 10
done

 which was an adhoc scripting hack, but  this showed the throttling nature. This was the output during the conversion (with CRs added for formatting):

0:0:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:
0:0:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:1:0:2:0:0:0:0:0:0:0:1:1:0:0:0:0:0:0:
0:0:0:0:0:0:0:0:0:87:90:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:1:0:0:0:0:0:0:0:
0:0:0:0:0:0:0:0:0:0:0:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:1:0:0:0:0:0:0:0:0:0:
0:0:0:0:0:0:0:0:0:0:0:0:1:0:0:88:89:89:98:86:89:90:177:87:87:89:65:0:0:0:

In other words, even though there was a conversion queue stacked for the cronjob,  task.php typically executes under 1 sec elapsed processing every 2-3 mins. The exception was a short burst of 3 mins batch processing maybe 45 mins in and then a solid burst of 19mins to complete the conversion another 45 mins later.  This started about a third of the way through the post re-indexing so the step-up didn't occur on a functional boundary.

BTW, this why you really don't want to click the "manually run them now"  option because seems to lock you into this trickle mode for the entire conversion.  I do have to wonder if your developers have ever done timed conversions on representative customer databases.

I am going to draw a line under this because a ~3hr migration downtime is good enough for us.  Better than the ~30hrs in our early trials on a test laptop.

Thanks for your attention, Marc 😊

[Marc Stridgen]

You may actually want to take a look here, as it may be worth you having a look at the rebuild constants for your purposes 🙂

 

 

[Terry Ellison]

@Marc Stridgen,

Just an update.  The live cut-over went smoothly.  One extra step that I did do was to revalidate the licence key immediately after relocating to the new server.  I also upped the REBUILD constants by 5× in the config.  Likewise I pause the cron task and used a bash loop on my FPM container:

for ((i=1;i<1000;i=i+1)); do 
>   ((t=SECONDS))
>   php -d memory_limit=-1 -d max_execution_time=0 \
>       /var/www/ipb/applications/core/interface/task/task.php \
>       9e0ff925b30af547068652c6ee9929c7
>   ((t=SECONDS-t))
>   echo -n "$t:" 
>   sleep 10
> done

179:86:85:85:87:91:88:86:89:89:84:78:76:78:160:82:77:91:80:79:80:84:87:84:86:81:80:166:83:84:83:82:82:74:74:86:181:74:71:24:0:0:...

You can see the solid PHP task script running with no time metering.  Still about 15× faster than my earlier trials. 

[Marc Stridgen]

Glad to hear you are getting to where you wish to be there