How to speed up the Background Processes?

[rllmukforum]

Ah, there already is an index on those two columns, called first_post

But MySQL (5.6.18 win64) doesn't choose to use it.

It picks post_date instead. If I force the query to use index first_post then the query takes 0.0 seconds instead of 70+ seconds.

Logging a support call with IPS.

Still interested why we don't seem to have the same schema as Omri though!

[SecondSight]

Hello !

I also have this speed problem while my server is very powerful :

RebuildPosts - IPS\forums\Topic\Post - 56736 / 8388702

 

On 7/12/2015 at 0:00 PM, maddog107_merged said:

So a couple of things, first I increased the values of the content per cycle its doing in the applications/core/extensions/core/Queue folder.

Please, can you explain how to achieve this exactly ?

 

Thank you 

[MADMAN32395]

15 minutes ago, SecondSight said:

Hello !

I also have this speed problem while my server is very powerful :

RebuildPosts - IPS\forums\Topic\Post - 56736 / 8388702

 

Please, can you explain how to achieve this exactly ?

 

Thank you 

have you setup a cron? or are you still using the default 'By Traffic' ? 

[SecondSight]

No, I've not setup a cron. I've just had a look and I suppose I need to login to the ACP, then go to the advanced configuration and choose to use cron. Then it gives me a line to add under ssh which looks like this one :

/usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/mysite/public_html/forums/applications/core/interface/task/task.php 11856d90a47c84a484a5839f1c1deabb

I can do this if it's needed to increase the value of the content per cycle.

[MADMAN32395]

1 hour ago, SecondSight said:

No, I've not setup a cron. I've just had a look and I suppose I need to login to the ACP, then go to the advanced configuration and choose to use cron. Then it gives me a line to add under ssh which looks like this one :

/usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/mysite/public_html/forums/applications/core/interface/task/task.php 11856d90a47c84a484a5839f1c1deabb

I can do this if it's needed to increase the value of the content per cycle.

I'd say enable and setup the cron. But don't mess with the default value of cycles. The cron will do its job.

[SecondSight]

OK. I'm going to setup the cron and see if it's faster.

[SecondSight]

I've setup the cron and saw it's running (tail -f /var/log/cron under ssh).

Regarding the content rebuilt, is there a way to see the percentage done and left when using the cron ?

Thank you !

[BomAle]

On 17/2/2016 at 10:03 PM, Ryan Ashbrook said:

I've gone ahead and incorporated this for the next release - the indexes were not correct anyway (carry-over from the 3.x converters schema), so I have re-done them.

Hi

in actual stable release (4.1.9) the issue was fixed? Thanks

 

[Ryan Ashbrook]

On 3/26/2016 at 8:08 PM, BomAle said:

Hi

in actual stable release (4.1.9) the issue was fixed? Thanks

 

This was for the converters, specifically.

[Prank]

Just throwing my 2c in here. We have almost 8m posts and 500k topics. Upgrade/conversion completed almost 24 hrs ago and we're at 1.9% of posts rebuilt. 

A command line script with configurable 'rows per run' or similar should be available to us. Leaving a browser tab open for days (or weeks?) is not a viable solution and feels quite antiquated. 

[Nathan Explosion]

6 hours ago, Prank said:

Upgrade/conversion completed almost 24 hrs ago and we're at 1.9% of posts rebuilt. 

Are you utilising site traffic (which is the default) or a cron to run tasks?

[Prank]

 

Just now, Nathan Explosion said:

Are you utilising site traffic (which is the default) or a cron to run tasks?

Cron is configured but I have clicked on the 'run tasks now' link to have them process faster in the browser.

[Prank]

This is pretty dismal reading after ~30hrs; 

Processing RebuildPosts - IPS\forums\Topic\Post - 206861 / 7753634

[AutoItScript]

The cron task command returns immediately if it's already running and it generally is coded so that it takes around 45 seconds to run. Which means that you are missing out on about 10-15 seconds worth of processing. You can do a script to run from cli that attempts re-running the cron command every second so that you minimise the downtime. It only slightly speeds things up, but I think it's faster than using the browser option as that adds http delays in as well.

#!/bin/bash
while :
do
	/usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /var/www/vhosts/......./task.php a0b84115c9f762de7e1f33466cd5e389
	sleep 1
done

Replace the command with your own cron string.

[Prank]

Thanks AutoItScript, I was thinking of doing something similar, this will do the trick. Thanks.

[Lindy]

15 hours ago, Prank said:

This is pretty dismal reading after ~30hrs; 

The rebuilder tries to employ a degree of intelligence... it will increase the number per cycle automatically based on how much your server can handle. If it's dismal, it's because IPS4 has determined your server can't handle more without significantly impacting performance. We recommend, if you're using a VPS, increasing the specs while the rebuild runs. 

[Prank]

1 hour ago, Lindy said:

The rebuilder tries to employ a degree of intelligence... it will increase the number per cycle automatically based on how much your server can handle. If it's dismal, it's because IPS4 has determined your server can't handle more without significantly impacting performance. We recommend, if you're using a VPS, increasing the specs while the rebuild runs. 

Well, thats not really 'intelligent' is it, if you have to increase server specs to run it at a decent rate. I have 2 quite large servers (app / db) and both are using very little resources at the moment. Theres not many people on the boards atm (10:42am in Sydney) but still the processes are painfully slow. After ~36hrs the posts rebuilding is at 4%!! 

Surely there is a script I can run to speed this up and process 1000+ rows at a time (looks like maybe 100 atm?) - You mentioned earlier that there was a script that was in development? 

[Prank]

Ok, so, end of day 4 and i'm a smidge over 5% for posts rebuilding.

If I multiple those two values by 20, I get 100% in eighty days!  Or, some time around the end of September. 

This seems like an astonishing oversight for a product that appears pretty solid and quality. 

I'm going to rip apart the code and make a command line script to process more, quicker. 

[Tom_K]

Please share your results. IPS doesn't appear to be bothered with this problem but a lot of admins of big boards are facing problems since background tasks take so long.

[Prank]

5 hours ago, Tomzl said:

Please share your results. IPS doesn't appear to be bothered with this problem but a lot of admins of big boards are facing problems since background tasks take so long.

 

Will do. 

[Lindy]

18 hours ago, Prank said:

Ok, so, end of day 4 and i'm a smidge over 5% for posts rebuilding.

If I multiple those two values by 20, I get 100% in eighty days!  Or, some time around the end of September. 

This seems like an astonishing oversight for a product that appears pretty solid and quality. 

I'm going to rip apart the code and make a command line script to process more, quicker. 

That's not normal - we have a dedicated team of people who do IPB to IPS4 upgrades all day, every day and our managed services team are finishing up our v3 migrations and even with millions of posts, 5% in 4 days just doesn't happen under normal circumstances. The rebuild will run as fast as it's able to while trying not to bring your site down for live traffic. You could be missing an index, you could have something else going on, but if you are using cron-based rebuilds or running them manually and your server is as robust as you believe it to be, we should take a quick look. 

Getting frustrated on the forum (and I understand it is frustrating, to be clear) isn't going to make your rebuild process any faster, but contacting us and saying "hey, I'm doing everything you said and I'm getting nowhere" can allow us to look and provide further guidance as necessary. 

Also note, 4.1.13 has rebuild improvements, including the order (newest to oldest) - you may wish to inquire about continuing the process under 4.1.13 - I don't know specifically how that would work or if it's advised. 

[Tom_K]

54 minutes ago, Lindy said:

Getting frustrated on the forum (and I understand it is frustrating, to be clear) isn't going to make your rebuild process any faster, but contacting us and saying "hey, I'm doing everything you said and I'm getting nowhere" can allow us to look and provide further guidance as necessary.

When my test upgrade from v3 to v4 took more than a week I contacted support and I was told it is the way it is. So no, contacting support will not help either.

In this thread and the one I started, it was requested many times that a more optimal way to run background proccesses is provided.

So far there has been little to no effort from IPS to help customers diagnose what is the bottleneck on otherwise powerful dedicated servers when performing background tasks.

Some admins said they "tweaked" the script themselves to make it do the work faster.

I even asked if upgrading every minor version will cause the need for the post index to be rebuilt (it took several days on my board when I did the upgrade to 4.1.12) but no answer was given in that topic.

I think there are a lot of people here (including me) who are willing to give their feedback and help with testing ways to speed up these background tasks, if IPS had interest in solving this issue which is obviously a problem for large boards.

[Prank]

6 hours ago, Lindy said:

Getting frustrated on the forum (and I understand it is frustrating, to be clear) isn't going to make your rebuild process any faster, but contacting us and saying "hey, I'm doing everything you said and I'm getting nowhere" can allow us to look and provide further guidance as necessary. 

Are you really telling me I'm not allowed to express my frustration as a disgruntled customer (even in your diplomatic way)? I'm terribly sorry but thats not how things work. I am having the same problem as several other people on here and posted in the same thread as them, not starting a new one, and joined the already frustrated people. I appreciate you guys had your 4th of July holiday off (which is ridiculous! If you offer support it should be year round) and I got a few cranky posts in while you guys were away. Since my cranky posts I've had @Tomzl come forward and ask for any updates on my hacking. Clearly another user in the same thread who is frustrated by the same issue. 

Rather than respond the way you did, your support could have contacted me directly if that is what is required to fix the problem, ie offering a helping hand, but no. (Even if the problem is being experienced by more than me and you could support all of us in the same place!)

I guess I'll raise a ticket then eh? Maybe we can post the fix in here so that we can help others who are frustrated and cranky?

We hit the big 7-percenter today. Exciting times.

[Baptiste&Ulrich]

It took us 36h on high-end gears for 500k members, 3.5 millions messages to rebuilds things completely. (after 3h of conversion).

We used the function of the dashboard to make it by hands with a computer that stay awake. We used the trick of changing all the rebuild numbers in the classes in order to maximise the ressources usages (increase it until around all ajax request was arround 30seconds ).

[Lindy]

9 hours ago, Prank said:

Are you really telling me I'm not allowed to express my frustration as a disgruntled customer (even in your diplomatic way)?

Not at all. :) I think you're misunderstanding. I'd be frustrated too, but did your post make your rebuild go any faster? There could be some set of unique circumstances that we could potentially resolve for you. We'll never know unless you ask for help via the client area so we can look at your site. a post on here isn't going to change anything at all for your individual circumstance; if you need help - please, ask for it via the client area. You and Tom seem to imply there's some sort of epidemic that we "can't be bothered with" to give credibility to your concerns and out of the thousands of people that have upgraded, hundreds that we've upgraded directly, only a very small percentage have performance issues with the rebuild and in nearly all cases, it's an environment issue, or there's a missing index, database issue, etc. There's no single "oh sure, do XYZ, that will fix you up!" Magic bullet that wouldn't have an impact on others. 

I'll repeat - you have every right to be frustrated and I would be too. Under ideal circumstances, it should take nowhere near that long for the rebuild tasks to complete. Your frustration and snippy comments, however, are doing nothing to solve your concerns, whereas, contacting us might (and for the record, Tom's ticket was regarding a test rebuild.) No, we're not able to chase you down and beg to help you, just like we're not going to remain fully operational on what you refer to as "ridiculous" holidays (and I would note, we do staff support 24/7/365 in limited capacity) - your expectations are frankly unreasonable. 

Edit: I see you submitted a ticket earlier today - my apologies, I should have checked. Thank you for doing so and I assure you we'll give it full attention. Thanks for your patience. 

Thanks.