Jump to content

Community

How to speed up the Background Processes?


sound

Recommended Posts

on a test site its taking a long time to re-build content via the background process

talking a couple of million posts 

large numbers of gallery images, calendar events etc

 

do have cron running as well as hitting the manual option 

 

How can I  speed these 'background process' up as currently can't see how I can upgrade the 'live' site if its going to take days to rebuild?

 

Edited by mikesound
Link to comment
Share on other sites

I too have been vexed by this process, I did a test upgrade last Monday and it's still going. 

- And I have hardly as many posts as you. 

I have even upgraded my server with more RAM and SSD to help this process and it has not made this perform faster.

There have been submitted tickets - and bug reports and have not seen any improvement or resolutions this past several RC -

The other issues is that you can't do a whole lot of "back end" configuration while this stuff is being processed - like 'delete a forum' it queues up and won't go away until it gets around to it. Same is true for some third party add apps until the background processes are complete you can't do too much with configuring them.

This is a serious issue in the fact that a community can't be expected to be "offline" until its complete. And no it's not something that can be done in the background while the site is open to the end users - because there are things that just don't work (search) for example.

 

 

Link to comment
Share on other sites

Are you guys running your sites on virtual private servers or dedicated hardware?   I'm seeing this behavior too on my 1.3M post forum and it's taking well over 30 hours so far to get to about 71% complete.

I am thinking very seriously about spinning up a Linux VM on one of my home VMware ESXi hosts and throwing a ton of vCPU and vRAM at it, and do my conversion THERE, then dump the MySQL database and move it over to my hosted VPS when the time comes to do this for real.  Hoping that would speed it up some.

 

Link to comment
Share on other sites

  • Management

I believe content rebuild is based on number of content items per cycle. I'll ask that someone chime in with specifics on how to increase that limit if you feel your environment can support it. Unfortunately, we have to, by default, account for the lowest common denominator hosts -- but I think we can improve and better document this for users with more powerful environments. 

Thanks for your patience.

Link to comment
Share on other sites

Sometime overnight the Rebuilding Post Content (or at least I think that's what it was called) completed along with a few of the smaller tasks.  Now I'm watching a 0.12% completion status bar on the Reindexing Posts task and suspect that one is going to take at least a week to complete.  :(

Anything you guys can to do help speed those things up would be appreciated.  It sounds like what you were saying, @Lindy, is that the code is throttled somewhat to make it work on the widest range of server configurations possible.  If so, my idea of doing my conversion locally on a dedicated "monster VPS" wouldn't really help me any would it?

 

Link to comment
Share on other sites

I believe content rebuild is based on number of content items per cycle. I'll ask that someone chime in with specifics on how to increase that limit if you feel your environment can support it. Unfortunately, we have to, by default, account for the lowest common denominator hosts -- but I think we can improve and better document this for users with more powerful environments.

Thanks for your patience.

​I am concerned slightly about this because while the lowest common denominator hosts are great...some of us may not know how to set up cron jobs properly if that is what it takes to do this better/faster. I certainly do understand(and appreciate) meeting the lowest common denominator because not everyone here is a Big board and the little guy matters just as much as the guy with 20 million posts.

But the current implementation on IPB3.X line for background processes and all that stuff is far superior to the soon to be current implementation on IPB 4. I haven't played with my test install in weeks/ages(I believe I am on RC4) but I do plan to upgrade to RC6 probably Tuesday. But I do like the way that we were able to control how many cycles for each type of item.

I just logged in at my test install and the new configuration apparently has the queue locked. I tried to manually do it and it locked again within a few seconds.

Link to comment
Share on other sites

I sent in a support ticket for this, but we didn't get any useful response.

Haven't yet dug through the specifics of the task code to do this ourselves but if I discover anything useful I'll share.

This really is integral for big forums, I'm sure a lot of us have a custom setup which could deal with processing a lot more per cycle.

Edited by G17 Media
Link to comment
Share on other sites

I believe content rebuild is based on number of content items per cycle. I'll ask that someone chime in with specifics on how to increase that limit if you feel your environment can support it. Unfortunately, we have to, by default, account for the lowest common denominator hosts -- but I think we can improve and better document this for users with more powerful environments. 

Thanks for your patience.

​A while back you (or Charles, not sure) mentioned a script that IPS was going to provide to allow those of us who are power users to run the processes at a much faster rate (aside from the already provided cron task).  Any update on that?

Link to comment
Share on other sites

  • Management

​A while back you (or Charles, not sure) mentioned a script that IPS was going to provide to allow those of us who are power users to run the processes at a much faster rate (aside from the already provided cron task).  Any update on that?

​I don't think a script will be necessary and believe it's a matter of simply updating a variable, but please don't quote me on that. I'm trying to get more information for you guys - we'll be back to full operation tomorrow and @bfarber or @Mark should be able to point you in the right direction as I know we did this on our own site and unless something has changed between then and now, you should be able to as well. 

Link to comment
Share on other sites

You can set up the cron job per the instructions in the ACP, and you can run the background processes via the ACP manually if you choose.  We have been testing a command line script that effectively does the same thing as the ACP "run background processes manually" command but it is not ready to be distributed - it is still undergoing initial testing, and I am unsure when/if it will be released.

Link to comment
Share on other sites

The script merely recreates what is already possible.  Initial testing showed some memory issues which we have been testing.  Just to be clear, I'm not saying this standalone script won't be released - just that it hasn't been deemed ready for release yet is all. :)

Really though, it basically just runs the "run background processes manually" that you see in the ACP now.  It's something like 30 lines of code to initialize the suite and run the right library calls in a loop.  I will try to follow up on where the testing of it is.

Link to comment
Share on other sites

Deleting the forum has to be queued if there are topics to delete.  The main problem is that you have a lot of posts to rebuild so if you do not set up a cron or manually run the background tasks it can take quite a while (although typically once posts rebuild the rest of the queue diminishes quickly).

Link to comment
Share on other sites

For what it's worth - my background processes on a 23,000 member forum with ~1.2 Million posts completed after about four days.  I have a mid-weight VPS running the community.  I could probably have sped it up by doing all of the conversion offline on a dedicated local VM.

 

Link to comment
Share on other sites

I started 3 days ago manually

0fd43d879b8b58864c739fa29636e680.png

4ae4d4851d8daf0b2c59f2b3a6b7507a.png

My test site is running on clean Dedicated Server, Intel Xeon E3-1225v3, 32GB DDR3 RAM, SSD. 

Site statistics: Threads 273,452 Posts 760,854 Members 285,480

I hope that soon there will be a solution how to speed up the process. I really do not want to turn off my board for one week.

Edited by Afternova
Link to comment
Share on other sites

In IPB 3 I had about 4 million posts years ago, it use to take forever to rebuild but after I created some indexes in the db and tweaked the my.cnf (added a tmp directory) things ran in 15 minutes instead of days.  I don't remember exactly but I know there are things a sysadmin can tell you I just don't know, most of what I did was shear trial and error.

Link to comment
Share on other sites

×
×
  • Create New...

Important Information

We use technologies, such as cookies, to customise content and advertising, to provide social media features and to analyse traffic to the site. We also share information about your use of our site with our trusted social media, advertising and analytics partners. See more about cookies and our Privacy Policy