Jump to content

Community

How to speed up the Background Processes?


Recommended Posts

  • Replies 175
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

We are going to build a degree of intelligence to the rebuild routine that will gauge how much your server can handle based on how fast it can run a cycle. This will be completed next week. 

I've gone ahead and incorporated this for the next release - the indexes were not correct anyway (carry-over from the 3.x converters schema), so I have re-done them.

​All your forum are belong to Lindy. His expression is perfect.... too perfect...

Posted Images

I'll be choosing to keep it offline (thankfully, my busy periods are tied into the UK football season, so a week offline after end of season isn't going to be an issue) so that I don't have to constantly field questions as to why posts look strange. However, is it possible for the number of posts that are rebuilt each time to be increased? If yes, then great....if no, then fine (by me) Would just like to know either way.

Link to post
Share on other sites

The background processes are designed to do just that - run in the background. There should be no need to leave your site offline until they complete.

​What will be unavailable when these processes are not yet finished ?  Just statistics or also topics/messages ?  And if so, will the newer topics handled first ?

Link to post
Share on other sites

Since members will be complaining about their posts, signatures, about me fields in profiles and personal massages not being formatted as they intended it to be and some of them might get to changing it in order to be presented in a way they want it. From our perspective, this is unnecessary stress for both us running the community and the members who have no idea why that's happening. So we'll be offline up until that gets rebuilt.

In our case around 640.000 posts get rebuild in about 24 hours (we run it manually right away), it's not a big deal, but does it really have to take that long? 

Link to post
Share on other sites

Nothing is "unavailable", content is just not formatted properly.

It takes a long time because of all the overhead of actually performing the content rebuilding and parsing it.  Rebuilding content is an intensive process, which is why we try to avoid doing it if possible (going from 3.4 to 4.0 is a big jump and requires it)

Link to post
Share on other sites

How many bugs have been filed because the background process was working and they cleared up after it was complete.

It's not benign as users can't search until its complete. 

Admin's can't really get into redecorating or doing any sort of spring cleaning while they are operating. (deleted forums or other ACP settings)

adding or removing apps / add ons - your mileage may vary - i have had some refuse to install or not function as intended because background processes were not complete

probably not good idea to attempt to install sphinx  or other server side settings ( caching - or moving of files to different locations ) until the system is complete. 

 

 

Link to post
Share on other sites

We are going to build a degree of intelligence to the rebuild routine that will gauge how much your server can handle based on how fast it can run a cycle. This will be completed next week. 

​Great, I'll wait till then to upgrade my forums.

Edited by simonle
Link to post
Share on other sites

2.5 million (and counting) posts to rebuild here - on my last test upgrade, it took 8 days on my dedicated server. That question I asked above, if answered at any time, will help me further with this one.

​So some good news - the above test was performed with RC6. I'm 1 hour into the background tasks on a test upgrade (to 4.0.0) of my forum now, and the post rebuild is already 4.5% through....so I'm already looking at an improvement on the 8 days.

Edited by Nathan Explosion
Link to post
Share on other sites

I don't think people should get their hopes up about any major time improvements here - whatever tweaks are made. It's an intensive process. I modded my board to update 500 posts at a time (which I imagine will be way higher than any official tweaks were mentioned earlier.).  It really didn't change the time at all - a few seconds here and there. In 500 post chunks the rebuild speed was 100k posts per hour.

When I upgrade for real and given that the best rebuild time for my board is projected to be 12 hours I'm just going to reopen it straight away. I'll pre-warn members the day before about what is going to happen.

What would help me though is if the rebuild process could be made to go backwards. This would mean that the active posts are converted first so that most people wouldn't notice there is an issue unless they went hunting for archive threads.

Edited by AutoItScript
Link to post
Share on other sites

What would help me though is if the rebuild process could be made to go backwards. This would mean that the active posts are converted first so that most people wouldn't notice there is an issue unless they went hunting for archive threads.

​Funny you should mention that, as the same thing occurred to me earlier - have made a file change on my test site to see if it has the effect I think it will.

EDIT: Yeah...that didn't have the effect I thought...oops. :D

Edited by Nathan Explosion
Link to post
Share on other sites

I found a way around to resolve the stack in rebuilding background tasks without turning on the cron in ACP:

in the dashboard below all background processes there is a link

"Alternatively, you can manually run them now and wait until they all complete. "

which didn't work for 2 days showing only 92% stack in rebuilding posts since we upgraded last Friday, and I opened it many times in the current tab in FireFox, showing "starting...." and then just a blank page in that tab, a few minutes ago I opened it with a right mouse button in a New Window AND - voila! it's working right now and the progress is on! Yahooooo!

All tasks took about 3 hours to complete

Link to post
Share on other sites
  • Management

For those of you who are experiencing slow task execution please try uploading this file. Simply upload it and then wait for your tasks to run as normal.

applications/core/tasks/ queue.php

Unsupported, use at your own risk, backup, and so on. This is designed to work on 4.0.1. If it helps then we will release it in 4.0.2. Please report your experience here.

Link to post
Share on other sites

Already seeing a slight speed up - prior to putting the new file in place, the % indicator for rebuilding posts was 'climbing' at a rate of .01% each task run (I've got 3 cron jobs in place running every minute, but using "sleep xx;" to get a 20 second interval between each job) but seeing that change to a .04% incremenet with the new file.

Link to post
Share on other sites
  • Management

Already seeing a slight speed up - prior to putting the new file in place, the % indicator for rebuilding posts was 'climbing' at a rate of .01% each task run (I've got 3 cron jobs in place running every minute, but using "sleep xx;" to get a 20 second interval between each job) but seeing that change to a .04% incremenet with the new file.

​Would you mind doing a normal cron setup to simulate how clients would normally do it? Just a normal one running once a minute. I fear your setup may cause problems with the new queue logic.

But either way you're already seeing an improvement of 4 times per cycle which is good :) 

Link to post
Share on other sites
  • Recently Browsing   0 members

    No registered users viewing this page.


×
×
  • Create New...

Important Information

We use technologies, such as cookies, to customise content and advertising, to provide social media features and to analyse traffic to the site. We also share information about your use of our site with our trusted social media, advertising and analytics partners. See more about cookies and our Privacy Policy