How to speed up the Background Processes?

[Prank]

9 hours ago, Ulrich said:

We used the trick of changing all the rebuild numbers in the classes in order to maximise the ressources usages (increase it until around all ajax request was arround 30seconds ).

Any more info on this? What classes in particular did you edit? Thanks for the info

5 hours ago, Lindy said:

Not at all. I think you're misunderstanding. I'd be frustrated too, but did your post make your rebuild go any faster? There could be some set of unique circumstances that we could potentially resolve for you. We'll never know unless you ask for help via the client area so we can look at your site. a post on here isn't going to change anything at all for your individual circumstance; if you need help - please, ask for it via the client area. You and Tom seem to imply there's some sort of epidemic that we "can't be bothered with" to give credibility to your concerns and out of the thousands of people that have upgraded, hundreds that we've upgraded directly, only a very small percentage have performance issues with the rebuild and in nearly all cases, it's an environment issue, or there's a missing index, database issue, etc. There's no single "oh sure, do XYZ, that will fix you up!" Magic bullet that wouldn't have an impact on others. 

I'll repeat - you have every right to be frustrated and I would be too. Under ideal circumstances, it should take nowhere near that long for the rebuild tasks to complete. Your frustration and snippy comments, however, are doing nothing to solve your concerns, whereas, contacting us might (and for the record, Tom's ticket was regarding a test rebuild.) No, we're not able to chase you down and beg to help you, just like we're not going to remain fully operational on what you refer to as "ridiculous" holidays (and I would note, we do staff support 24/7/365 in limited capacity) - your expectations are frankly unreasonable. 

Edit: I see you submitted a ticket earlier today - my apologies, I should have checked. Thank you for doing so and I assure you we'll give it full attention. Thanks for your patience. 

Thanks.

Thanks for the reply Lindy, I'll bite my tongue here. Hopefully we can get this sorted through Client Area.

[Lindy]

2 hours ago, Prank said:

Any more info on this? What classes in particular did you edit? Thanks for the info

Thanks for the reply Lindy, I'll bite my tongue here. Hopefully we can get this sorted through Client Area.

I've made a note in the ticket to spend some time profiling this and figuring out what the issue may be - whether it's on our end or not. I'm hopeful we can get an answer for you before the weekend. Thanks for your patience.

[Baptiste&Ulrich]

6 hours ago, Prank said:

Any more info on this? What classes in particular did you edit? Thanks for the info

This is already expressed earlier in this thread, mainly: 

applications/core/extensions/core/Queue/RebuildPosts.php / Line 29

public $rebuild = 600; //orig: 50, 1000 works also ;

applications/core/extensions/core/Queue/RebuildSearchIndex.php / Line 29 ->

public $index   = 1000;

And so on ... each time the ajax request is a bit too short.

It's sad that all this process are not run in parallel, only one request a time, that mean only one thread on my ...  24 thread machine.

 

[Prank]

Thank you Ulrich, I missed the earlier post.

[Charles]

I replied in your ticket but for everyone's info:

Do not run the queue tasks both in the browser (using the manually run now link) and via the cron at the same time. The queue task locks when it is running (so only one thing has control at a time) and, when you do both the cron and the browser, only the browser task will execute and it is much slower than the cron task.

When you do both of them the cron is useless and does nothing because the browser keeps the queue task always locked. The cron will execute, see the browser has the queue locked, and instantly exit therefore doing nothing. Looping via the browser is much, much slower than letting the cron take care of it.

I think trying to do both at the same time is the root of your slow rebuilds.

The cron system has logic in it where it will run at least 45 seconds per execution. There is no need to edit anything in the system, write fancy looping scripts, change the number of posts it does per loop, and so on because it will just keep looping as long as it can with in that 45 second limit. It will then continue on the next execution.

I personally upgrade large sites quite often and I know from experience the cron can process well over a million posts per day without any special interaction. Just set the cron and forget it  if anything goes wrong you will see a notice at the top of the AdminCP dashboard saying there is something wrong with the queue. If that is the case let us know and we will of course help you.

 

[BN_IT_Support]

That is interesting because I had assumed that running the tasks manually was always going to be faster than letting it just run by itself.  So when would you recommend using running the tasks manually?

Is running tasks through cron always faster than running them manually; and is running them manually faster than running them automatically with site traffic?

[Charles]

44 minutes ago, Edward Shephard said:

That is interesting because I had assumed that running the tasks manually was always going to be faster than letting it just run by itself.  So when would you recommend using running the tasks manually?

Is running tasks through cron always faster than running them manually; and is running them manually faster than running them automatically with site traffic?

In order of speed.

Cron -> Run Manually -> Site Traffic

When I upgrade a large site from 3 to 4 I usually put the cron up for a day or two until it's all done with queues then switch back to site traffic. I do that because I don't want to keep up with hundreds of cron running

If you are curious, the "Run Manually in Browser" mode is slower because it has to refresh to go to next loop whereas cron just does it internally and it has to deal with memory limits that the cron, being at command line, doesn’t have to worry as much about. Of course day to day operations do not have the huge volume that a 3 -> 4 upgrade has to contend with so switching to Site Traffic is just fine after it's all done.

[Prank]

Thanks @Charles - thats helpful info.

 

[Tom_K]

On 7. 7. 2016 at 10:09 PM, Lindy said:

You and Tom seem to imply there's some sort of epidemic that we "can't be bothered with" to give credibility to your concerns and out of the thousands of people that have upgraded, hundreds that we've upgraded directly, only a very small percentage have performance issues with the rebuild and in nearly all cases, it's an environment issue, or there's a missing index, database issue, etc. There's no single "oh sure, do XYZ, that will fix you up!" Magic bullet that wouldn't have an impact on others. 

I'll repeat - you have every right to be frustrated and I would be too. Under ideal circumstances, it should take nowhere near that long for the rebuild tasks to complete. Your frustration and snippy comments, however, are doing nothing to solve your concerns, whereas, contacting us might (and for the record, Tom's ticket was regarding a test rebuild.)

Why haven't all those admins responded to these topics and said how fast their upgrade process was? Instead I've heard from @David T. Cole (enterprise customer who had the same experience as me - how do you explain his case since IPS was doing the upgrade?), @TSP (said Neowin website with 11 million posts took 1 month to get their background tasks finished), @JordanRash (500k posts and more than 2 days with background tasks still running), user @cwir (contacted me via private message and said his board with 1 million posts took over a week to upgrade as well).

In this topic alone users @sound, @Ocean West, @Dolphin., @Nathan Explosion, @Drew D , @SecondSight , @Prank have complained about slow running of background tasks as well.

That's a long list of people that have had the same problem. I'm sure many boards have been upgraded but my guess is that not a very large percentage of those are large boards.

I don't know why I even started listing all the people with problems since the main problem here is that IPS is not willing to give assistance in diagnosing the reason for slow execution of background tasks.

I'm on a dedicated server with i7-2600 Quad Core processor, 32 GB DDR3 RAM and 2x 3 TB SATA III HDD. Am I right to assume that these specs should be enough to get the job of converting 2 million posts executed sooner than 7-8 days since the whole server has only one active site on it?

When I asked support (request ID 950772) how to speed up the process, I was told: "If you already have this setup on a cron, there isn't anything more you can do in this case, if you tried in increase the amount per cycle, it would likely cause many errors due to resources." When I said I'm on a dedicated server with no other active sites and asked if setting a cron would speed up the process, I received this reply: "running it manually from the acp is actually faster than a cron, so that is the quickest method".

It is my understanding that this is not true and cron is actually faster than manually running the tasks. Conflicting information from IPS.

Lindy, you hinted that since my ticket was regarding a test re-build it would have been given more attention, if I was doing a live upgrade. This is silly. The whole point of doing test upgrades is to identify problems and avoid them when doing things live. IPS policy to only assist with live upgrades is pretty useless to me. Chances are that between the time I report the ticket and you manage to find a solution, a week will already have gone by and the tasks will have been completed without your help but in the meantime all my users would have been pissed and reporting problems for a week.

I'm the head of customer support in our company and deal with customer complaints like this on a daily basis so it's even harder to understand why agents aren't given more freedom to asses when these policies should be bypassed. Since Lindy thinks this is not a problem it probably means noone thinks anything should be done differently to avoid such conflicts with clients.

Try to see this situation from my POV. I'm told I can't get support regarding a problem I know will occur, until it is present on my live board. When I ask questions that would help me fix the problem with my own resources (e.g. which resource on the server causes slow background task execution), I receive no reply. When I upgraded my test board to 4.1.12.3 I had to rebuild the posts index which again took 2-3 days (that's on top of the 7-8 days for the initial v3 to v4 upgrade). When I and others asked if this will happen on all minor upgrades, no reply was given. Taking all this into consideration my opinion is that IPS is handling this poorly.

Perhaps my frustration is stronger because from past experience over the years of being an IPS client and following this board I know that IPS seems very rigid concerning some issues and it doesn't matter how many people complain, nothing changes. There are things people have been complaining about or suggesting for years and nothing has happened. But countless hours have been spent posting replies negating customers opinions.

So since I don't expect anything to improve, the only other thing to do is to let my frustration out here on the board. At least it will make me feel better.

 

On 8. 7. 2016 at 0:58 PM, Charles said:

I personally upgrade large sites quite often and I know from experience the cron can process well over a million posts per day without any special interaction. Just set the cron and forget it  if anything goes wrong you will see a notice at the top of the AdminCP dashboard saying there is something wrong with the queue. If that is the case let us know and we will of course help you.

I'll try this one last time since I've received no reply the last time. Which server resource is the key to make the tasks run faster? Is it processor speed, RAM quantity, HDD speed, caching, MySQL settings etc.? Simply saying your server doesn't have enough resources is a BS answer. Tell me which resource on my server I have to upgrade and I'll do it. If you can't tell me what it is that I have to upgrade then my only conclusion is that you don't know really know the reason but just found a convenient/easy excuse.

I'm sure shared hosting clients have all sorts of limits on their packages but VPS and dedicated server clients should not be in the same situation or can change the settings of their servers. Help them out and we can avoid tens of pages of writing to each other.

If I were IPS, I would see more sense in sacrificing a few hours to try and diagnose the reasons for slow background tasks execution than writing long posts trying to convince others that they are only a small percentage like that makes anyone feel any better. You lose time in both cases but at least you've shown some effort to actually solve the issue, if you invest some effort into troublehooting the problem. Chances are you'll also improve your product in the process of doing it. My 0.05 $.

[Charles]

Someone here finally submitted a ticket on their issue and as I posted we sorted out the problem. It turned out there was nothing wrong other than the client trying to do two things at once which caused the whole process to slow to a crawl.

So I might suggest that you simply ask for our assistance in looking at what is going on. Posting long posts here do not solve problems but short support tickets do  

[Tom_K]

I did exactly what you suggest Charles, long before posting here. Please see request ID 950772 and tell me what else I could have done to get better support? I'm not being cynical but really want to know because I'm often left feeling frustrated after dealing with IPS support. Most of the time it's because it takes many posts to even explain what the problem is and in many cases the problem is not solved but I'm told it's just the way it is.

It would save me and IPS a lot of time to find a better way to communicate.

[BN_IT_Support]

1 hour ago, Tomzl said:

Why haven't all those admins responded to these topics and said how fast their upgrade process was?

Perhaps because that would just be annoying!  My upgrade went fine, but just telling you how quick mine was would do nothing to help assuage your frustration.

[Tom_K]

4 minutes ago, Edward Shephard said:

Perhaps because that would just be annoying!  My upgrade went fine, but just telling you how quick mine was would do nothing to help assuage your frustration.

Why would that be annoying? It would actually be very useful since you could give feedback about your server specs which would help rule out or confirm server hardware as the main reason/excuse for slow running background tasks. Simply an elimination process of possible causes.

If possible please share how many users and posts your board has and how much time all the background tasks took when you upgraded from v3 to v4.

[Nathan Explosion]

1 hour ago, Tomzl said:

 

In this topic alone users @Nathan Explosion, .....have complained about slow running of background tasks as well.

 

Whoa there!!! Take my comments in the context of when they were being made....early 4.0.0 betas, and very early 4.0.x releases. I've not 'complained' on this subject for a while and that is because I don't have a complaint to make. My first test on an early beta had a 7-day processing of my then ~2.5 million posts. Now? When I'm testing, it's down to less than a day for nearly ~3 million posts.

[Charles]

If anyone is having issues feel free to submit a support ticket. We are happy to assist. As far as I know, @Prank is the only person in this topic who has submitted a ticket recently and we were able to resolve the issue.

If you had submitted a ticket in the past and are still experiencing an issue you can always start a new ticket as a follow up.

[Tom_K]

 

20 minutes ago, Nathan Explosion said:

Now? When I'm testing, it's down to less than a day for nearly ~3 million posts.

Great to hear. Please share your server/hosting specs. Did you use cron or did you manually run the tasks?

3 minutes ago, Charles said:

If anyone is having issues feel free to submit a support ticket. We are happy to assist. As far as I know, @Prank is the only person in this topic who has submitted a ticket recently and we were able to resolve the issue.

If you had submitted a ticket in the past and are still experiencing an issue you can always start a new ticket as a follow up.

Does this mean I will receive support if I have the same issue on my next test upgrade or only for live upgrade?

[Charles]

As always, we only support live sites. We cannot support test installs.

[Nathan Explosion]

4 minutes ago, Tomzl said:

 

Please share your server/hosting specs.

No, if you don't mind.

Quote

Did you use cron or did you manually run the tasks?

As you know I've commented before, you will already know that I use cron.

[JordanRash]

I first created a support ticket and THEN was told to post here in the ticket.... which is it. Do we post here or do we create a support ticket.

 

@Charles @Tomzl

[Lindy]

5 hours ago, JordanRash said:

I first created a support ticket and THEN was told to post here in the ticket.... which is it. Do we post here or do we create a support ticket.

 

@Charles @Tomzl

You submitted a ticket and we looked and investigated. You were getting 502 errors, which is a gateway error and is often caused by the server overloaded to the point it can't serve further requests. It was deemed at the time your server was choking and that was the cause of the slow rebuild performance. We suggested because of the size of your site and the fact you were using MyISAM that you switch to InnoDB. Beyond that, I'm sorry we weren't able to further troubleshoot your server environment for you. That's why you were referred to the community to get help from other sysadmins in regards to working with your server performance. 

[Prank]

I was so happy when this one hit 100% but then it just kept going!

[mfahl]

Can someone please give me cron to run.

I have tried

Quote

/usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/***/public_html/applications/core/interface/task/task.php 1d1a46caf559609a0dc9229b4008e344

But getting following error

Quote

Incorrect key

Any help much appreciated.

 

[BN_IT_Support]

I think it's just the standard one under system > advanced configuration > server environment > use cron, and it should tell you the code and key that you need.

[mfahl]

Many thanks,found it now.

Cannot understand why Ips have made this version so counterintuitive to use.

Seems to be going backwards to me.

Just hoping board will run faster when tasks have all run otherwise i will be looking elsewhere.

[AuroraEN]

What is up with this? I cannot unlock the 'queue' task, it immediately gets relocked. I have a cron setup running every minute.

Attempting to run it manually gives me this error: