sound Posted April 4, 2015 Share Posted April 4, 2015 on a test site its taking a long time to re-build content via the background processtalking a couple of million posts large numbers of gallery images, calendar events etc do have cron running as well as hitting the manual option How can I speed these 'background process' up as currently can't see how I can upgrade the 'live' site if its going to take days to rebuild? Link to comment Share on other sites More sharing options...
Ocean West Posted April 4, 2015 Share Posted April 4, 2015 I too have been vexed by this process, I did a test upgrade last Monday and it's still going. - And I have hardly as many posts as you. I have even upgraded my server with more RAM and SSD to help this process and it has not made this perform faster.There have been submitted tickets - and bug reports and have not seen any improvement or resolutions this past several RC -The other issues is that you can't do a whole lot of "back end" configuration while this stuff is being processed - like 'delete a forum' it queues up and won't go away until it gets around to it. Same is true for some third party add apps until the background processes are complete you can't do too much with configuring them.This is a serious issue in the fact that a community can't be expected to be "offline" until its complete. And no it's not something that can be done in the background while the site is open to the end users - because there are things that just don't work (search) for example. Link to comment Share on other sites More sharing options...
Whiskey Bizness Posted April 4, 2015 Share Posted April 4, 2015 Are you guys running your sites on virtual private servers or dedicated hardware? I'm seeing this behavior too on my 1.3M post forum and it's taking well over 30 hours so far to get to about 71% complete.I am thinking very seriously about spinning up a Linux VM on one of my home VMware ESXi hosts and throwing a ton of vCPU and vRAM at it, and do my conversion THERE, then dump the MySQL database and move it over to my hosted VPS when the time comes to do this for real. Hoping that would speed it up some. Link to comment Share on other sites More sharing options...
Management Lindy Posted April 5, 2015 Management Share Posted April 5, 2015 I believe content rebuild is based on number of content items per cycle. I'll ask that someone chime in with specifics on how to increase that limit if you feel your environment can support it. Unfortunately, we have to, by default, account for the lowest common denominator hosts -- but I think we can improve and better document this for users with more powerful environments. Thanks for your patience. Link to comment Share on other sites More sharing options...
Whiskey Bizness Posted April 5, 2015 Share Posted April 5, 2015 Sometime overnight the Rebuilding Post Content (or at least I think that's what it was called) completed along with a few of the smaller tasks. Now I'm watching a 0.12% completion status bar on the Reindexing Posts task and suspect that one is going to take at least a week to complete. Anything you guys can to do help speed those things up would be appreciated. It sounds like what you were saying, @Lindy, is that the code is throttled somewhat to make it work on the widest range of server configurations possible. If so, my idea of doing my conversion locally on a dedicated "monster VPS" wouldn't really help me any would it? Link to comment Share on other sites More sharing options...
Management Matt Posted April 5, 2015 Management Share Posted April 5, 2015 You can run the processes from the ACP dashboard too. You can just leave the tab open and let it process. You can close the tab and it'll carry on with the crons fine. Link to comment Share on other sites More sharing options...
Izaya Orihara Posted April 5, 2015 Share Posted April 5, 2015 I believe content rebuild is based on number of content items per cycle. I'll ask that someone chime in with specifics on how to increase that limit if you feel your environment can support it. Unfortunately, we have to, by default, account for the lowest common denominator hosts -- but I think we can improve and better document this for users with more powerful environments.Thanks for your patience.I am concerned slightly about this because while the lowest common denominator hosts are great...some of us may not know how to set up cron jobs properly if that is what it takes to do this better/faster. I certainly do understand(and appreciate) meeting the lowest common denominator because not everyone here is a Big board and the little guy matters just as much as the guy with 20 million posts.But the current implementation on IPB3.X line for background processes and all that stuff is far superior to the soon to be current implementation on IPB 4. I haven't played with my test install in weeks/ages(I believe I am on RC4) but I do plan to upgrade to RC6 probably Tuesday. But I do like the way that we were able to control how many cycles for each type of item.I just logged in at my test install and the new configuration apparently has the queue locked. I tried to manually do it and it locked again within a few seconds. Link to comment Share on other sites More sharing options...
Ocean West Posted April 5, 2015 Share Posted April 5, 2015 I started one last Monday and it is still going and I don't have nearly as many records as others. Also there are duplicate task listed in the dashboard. Link to comment Share on other sites More sharing options...
G17 Media Posted April 5, 2015 Share Posted April 5, 2015 I sent in a support ticket for this, but we didn't get any useful response.Haven't yet dug through the specifics of the task code to do this ourselves but if I discover anything useful I'll share.This really is integral for big forums, I'm sure a lot of us have a custom setup which could deal with processing a lot more per cycle. Link to comment Share on other sites More sharing options...
Ocean West Posted April 5, 2015 Share Posted April 5, 2015 i tried logging in from a few different computers and attempted to manually run the background process but that didn't work.Guess the old saying is true....Nine women can not make a baby in one month. Link to comment Share on other sites More sharing options...
CheersnGears Posted April 6, 2015 Share Posted April 6, 2015 I believe content rebuild is based on number of content items per cycle. I'll ask that someone chime in with specifics on how to increase that limit if you feel your environment can support it. Unfortunately, we have to, by default, account for the lowest common denominator hosts -- but I think we can improve and better document this for users with more powerful environments. Thanks for your patience.A while back you (or Charles, not sure) mentioned a script that IPS was going to provide to allow those of us who are power users to run the processes at a much faster rate (aside from the already provided cron task). Any update on that? Link to comment Share on other sites More sharing options...
Management Lindy Posted April 7, 2015 Management Share Posted April 7, 2015 A while back you (or Charles, not sure) mentioned a script that IPS was going to provide to allow those of us who are power users to run the processes at a much faster rate (aside from the already provided cron task). Any update on that?I don't think a script will be necessary and believe it's a matter of simply updating a variable, but please don't quote me on that. I'm trying to get more information for you guys - we'll be back to full operation tomorrow and @bfarber or @Mark should be able to point you in the right direction as I know we did this on our own site and unless something has changed between then and now, you should be able to as well. Link to comment Share on other sites More sharing options...
MadMikeyB Posted April 7, 2015 Share Posted April 7, 2015 Do you mean that there actually may be a "Go Faster" button, @Lindy? Link to comment Share on other sites More sharing options...
bfarber Posted April 7, 2015 Share Posted April 7, 2015 You can set up the cron job per the instructions in the ACP, and you can run the background processes via the ACP manually if you choose. We have been testing a command line script that effectively does the same thing as the ACP "run background processes manually" command but it is not ready to be distributed - it is still undergoing initial testing, and I am unsure when/if it will be released. Link to comment Share on other sites More sharing options...
Ocean West Posted April 7, 2015 Share Posted April 7, 2015 and I am unsure when/if it will be released.In this one statement you have thrown out every plan and schedule we may have. This effectively indefinitely suspends further investment of time and energies for us to upgrade. Link to comment Share on other sites More sharing options...
bfarber Posted April 7, 2015 Share Posted April 7, 2015 The script merely recreates what is already possible. Initial testing showed some memory issues which we have been testing. Just to be clear, I'm not saying this standalone script won't be released - just that it hasn't been deemed ready for release yet is all. Really though, it basically just runs the "run background processes manually" that you see in the ACP now. It's something like 30 lines of code to initialize the suite and run the right library calls in a loop. I will try to follow up on where the testing of it is. Link to comment Share on other sites More sharing options...
Ocean West Posted April 7, 2015 Share Posted April 7, 2015 But it should not take a week to complete. Operations like "DELETE FORUM" should just be deleted and not queued up behind other tasks I started a test on Saturday - today is Tuesday. Link to comment Share on other sites More sharing options...
bfarber Posted April 7, 2015 Share Posted April 7, 2015 Deleting the forum has to be queued if there are topics to delete. The main problem is that you have a lot of posts to rebuild so if you do not set up a cron or manually run the background tasks it can take quite a while (although typically once posts rebuild the rest of the queue diminishes quickly). Link to comment Share on other sites More sharing options...
Nathan Explosion Posted April 7, 2015 Share Posted April 7, 2015 Simple question - each time the tasks.php file is run, via cron, how many posts will be rebuilt? Is that number of posts configurable? Link to comment Share on other sites More sharing options...
Ocean West Posted April 7, 2015 Share Posted April 7, 2015 But my site is not as big as others (<450k posts < 29k member).Cron is enabled, and I have pressed the run now button. Link to comment Share on other sites More sharing options...
sound Posted April 8, 2015 Author Share Posted April 8, 2015 1st april upgradedthe rebuild background processes has just ended a hour or so ago that's 7 days.... Link to comment Share on other sites More sharing options...
Whiskey Bizness Posted April 8, 2015 Share Posted April 8, 2015 For what it's worth - my background processes on a 23,000 member forum with ~1.2 Million posts completed after about four days. I have a mid-weight VPS running the community. I could probably have sped it up by doing all of the conversion offline on a dedicated local VM. Link to comment Share on other sites More sharing options...
Starship Posted April 8, 2015 Share Posted April 8, 2015 I started 3 days ago manually My test site is running on clean Dedicated Server, Intel Xeon E3-1225v3, 32GB DDR3 RAM, SSD. Site statistics: Threads 273,452 Posts 760,854 Members 285,480 I hope that soon there will be a solution how to speed up the process. I really do not want to turn off my board for one week. Link to comment Share on other sites More sharing options...
Nathan Explosion Posted April 8, 2015 Share Posted April 8, 2015 2.5 million (and counting) posts to rebuild here - on my last test upgrade, it took 8 days on my dedicated server. That question I asked above, if answered at any time, will help me further with this one. Link to comment Share on other sites More sharing options...
John 3:16 Posted April 8, 2015 Share Posted April 8, 2015 In IPB 3 I had about 4 million posts years ago, it use to take forever to rebuild but after I created some indexes in the db and tweaked the my.cnf (added a tmp directory) things ran in 15 minutes instead of days. I don't remember exactly but I know there are things a sysadmin can tell you I just don't know, most of what I did was shear trial and error. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.