How to speed up the Background Processes?

[media]

15 minutes ago, Rhett said:

Yes just select the cron option and add it on your server/hosting etc.

But I am right here:

Processing RebuildPosts - IPS\forums\Topic\Post - 470915 / 1150652 

Would I lose that 470915 ?

[Rhett]

Hello, no you won't lose anything, these run on their own in the background, or you can process them from the admincp, or you can use a cron, all good with either method, just use what suits you best.   

[heartbreakers]

If selected to manually run the background processes can I still close my tab without switching to cron? Will the processes still run?

[media]

Yes...

I did that way and it ran in the background and finished...

[heartbreakers]

5 hours ago, media said:

Yes...

I did that way and it ran in the background and finished...

Hmm, that doesn't seem to work for me, so I think I'll run a cron job.

On 10/20/2015, 1:17:34, Rhett said:

Hello, no you won't lose anything, these run on their own in the background, or you can process them from the admincp, or you can use a cron, all good with either method, just use what suits you best.   

Using the Cron Jobs application from cPanel, is it sufficient to use the command given to me in the ACP without having to adjust any of the directories? For example, if this is the command

/usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/***/public_html/***/applications/core/interface/task/task.php 1d1a46caf559609a0dc9229b4008e344

is it okay if I can't find a folder for /usr/, /local/, /bin/, or /php/ -- assuming those are folders/directories?

[media]

22 minutes ago, heartbreakers said:

Hmm, that doesn't seem to work for me, so I think I'll run a cron job.

Using the Cron Jobs application from cPanel, is it sufficient to use the command given to me in the ACP without having to adjust any of the directories? For example, if this is the command

/usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/***/public_html/***/applications/core/interface/task/task.php 1d1a46caf559609a0dc9229b4008e344

is it okay if I can't find a folder for /usr/, /local/, /bin/, or /php/ -- assuming those are folders/directories?

Sorry, I thought you are talking about cronjob...

I meant cronjob. You need to run the cronjob and you do not need to do anything... It keep running in the background until it finishes...

[heartbreakers]

21 hours ago, media said:

Sorry, I thought you are talking about cronjob...

I meant cronjob. You need to run the cronjob and you do not need to do anything... It keep running in the background until it finishes...

Also, are the posts in forums supposed to be funky before your run the background processes and while the process runs? For example, looking through some posts in my test install, I notice that the quoted post isn't indicated as such. I'm assuming the background process will fix this?

Thanks for your help!

[media]

1 hour ago, heartbreakers said:

Also, are the posts in forums supposed to be funky before your run the background processes and while the process runs? For example, looking through some posts in my test install, I notice that the quoted post isn't indicated as such. I'm assuming the background process will fix this?

Thanks for your help!

Well, remember your data needs to be utf8 ready... Other than that background process fixes all that funky stuff

[heartbreakers]

1 hour ago, media said:

Well, remember your data needs to be utf8 ready... Other than that background process fixes all that funky stuff

Yeah, all my data was formatted to UTF8 even before the upgrade, so I'm hoping the rebuilding process fixes all the broken quotes and posts.

[heartbreakers]

Do I need to set permissions to the task.php file back to 644 after the cron job is finished?

[Omri Amos]

Hi,

This is completely CRAZY SLOW - I have a board with ~5M posts (500K topics) and after a few HOURS it's only 0.15% done!! Running from CRON and/or manually, I'm trying everything I can but it's just SUPER SUPER SLOW.

Please provide a modification that will at least allow this to run backwards - posts from 10 years ago and not that important and can be fixed within a few weeks/months, but I need all the posts from the last month or so to be fixed right away.

I was thinking about modifying the 'offset' value of the task in the hw_core_queue table to (total posts - X latest posts), and then after it's finished to start it again from 0.
Does that make sense?

 

[Dll]

Are you sure the task isn't locking for some reason. For instance we had a problem with the tasks being run for blogs and that was locking everything, fixing that made it all work much more quickly. 

[Omri Amos]

Yes I'm sure...

I was actually able to modify the task in order to work backwards:

In "/applications/convert/extensions/core/Queue/RebuildContent.php" , Line 108, I changed ">" to "<" and "ASC" to "DESC".
And in Line 130 I changed the 'if' statement so that it will not stop in the last (now first) id but instead on the first id (id=1).

I started again (with a fresh backup of the unconverted 'posts' table, and a manual offset set to the same as 'count'+1) and it seems to be working fine. Still extremely slow (~1 post per second?!) but at least within a few hours I will have the posts from the last few days fixed. All the older posts can take their time, I don't mind...

 

[maddog107_merged]

5 hours ago, Omri Amos said:

Yes I'm sure...

I was actually able to modify the task in order to work backwards:

In "/applications/convert/extensions/core/Queue/RebuildContent.php" , Line 108, I changed ">" to "<" and "ASC" to "DESC".
And in Line 130 I changed the 'if' statement so that it will not stop in the last (now first) id but instead on the first id (id=1).

I started again (with a fresh backup of the unconverted 'posts' table, and a manual offset set to the same as 'count'+1) and it seems to be working fine. Still extremely slow (~1 post per second?!) but at least within a few hours I will have the posts from the last few days fixed. All the older posts can take their time, I don't mind...

 

I too have 5mil posts and upgraded last week. So a couple of things, first I increased the values of the content per cycle its doing in the applications/core/extensions/core/Queue folder. Also in MySQL ensure that you have query cache enabled as it does the same queries over an over. Same with PHP opcache (same script is executed each cycle). And finally just make sure you run it via Cron and not the UI. 

Took exactly a week to finish rebuilding everything (during which my users were pretty pissed off, and some still are as there are some issues to iron out) 

[Izaya Orihara]

I am not happy about this personally.  We are converting back to IPB and I am on 3.X for the moment and strongly considering staying on it(delaying the inevitable). Not because 3.X is better(it isn't) but it is so baffling and frustrating that it took me literally an hour to convert 550,000 posts, 12,000 topics, 2200 members from a completely entirely different software to IPB3.

However(both times we have had to deal with background content), it has taken DAYS to just upgrade from 3.X to 4.X

I still have no idea why they removed the amazing functionality of the background process in 3.X and replaced it with this.

[Omri Amos]

OK, it seems to be extremely slow because this query (with different id's) called all the time and each one is taking ~10 seconds (!!) to complete...

SELECT * 
FROM  `hw_convert_link_posts` 
WHERE  `ipb_id` =4977531
AND  `type` =  'forums_posts'
AND  `app` =2

(This is a 5M rows table)

So... I added a new unique index:

And now these queries take ~0.0001 sec each!!!! OMG!!

Now instead of ~200 posts per HOUR, it goes ~200 posts per SECOND.

[stoo2000]

9 hours ago, Omri Amos said:

Yes I'm sure...

I was actually able to modify the task in order to work backwards:

In "/applications/convert/extensions/core/Queue/RebuildContent.php" , Line 108, I changed ">" to "<" and "ASC" to "DESC".
And in Line 130 I changed the 'if' statement so that it will not stop in the last (now first) id but instead on the first id (id=1).

I started again (with a fresh backup of the unconverted 'posts' table, and a manual offset set to the same as 'count'+1) and it seems to be working fine. Still extremely slow (~1 post per second?!) but at least within a few hours I will have the posts from the last few days fixed. All the older posts can take their time, I don't mind...

 

Yeah, don't do this. What you're now doing is altering the offset of data to process each time a post is added, you're actually going to end up rebuilding posts multiple times over instead of 1 time per post. Also bear in mind that the parser is intended to be ran on legacy data (3.4) running it again on 4.x data may very well break it.

[Omri Amos]

56 minutes ago, stoo2000 said:

Yeah, don't do this. What you're now doing is altering the offset of data to process each time a post is added, you're actually going to end up rebuilding posts multiple times over instead of 1 time per post. Also bear in mind that the parser is intended to be ran on legacy data (3.4) running it again on 4.x data may very well break it.

The forum is closed so no new posts are added, I double checked - no post is being parsed more than once (which will indeed break it).
Either way it is no longer really needed because now after I added the missing index (see my prev reply) I can finish the entire 5M table in just about ~12-24 hours - instead of 10+ MONTHS.....

[stoo2000]

2 minutes ago, Omri Amos said:

The forum is closed so no new posts are added, I double checked - no post is being parsed more than once (which will indeed break it).
Either way it is no longer really needed because now after I added the missing index (see my prev reply) I can finish the entire 5M table in just about 2-3 hours - instead of 10+ MONTHS.....

You may have thought that through (good job!). But others will read this topic and blindly follow what you're saying without thinking about this.

[Ryan Ashbrook]

On 12/7/2015 at 6:18 AM, Omri Amos said:

OK, it seems to be extremely slow because this query (with different id's) called all the time and each one is taking ~10 seconds (!!) to complete...

SELECT * 
FROM  `hw_convert_link_posts` 
WHERE  `ipb_id` =4977531
AND  `type` =  'forums_posts'
AND  `app` =2

(This is a 5M rows table)

So... I added a new unique index:

And now these queries take ~0.0001 sec each!!!! OMG!!

Now instead of ~200 posts per HOUR, it goes ~200 posts per SECOND.

I've gone ahead and incorporated this for the next release - the indexes were not correct anyway (carry-over from the 3.x converters schema), so I have re-done them.

[rllmukforum]

That would have been handy for us, Ryan

Of about 10 million post rows, the manual job has rebuilt 55% in 14 days. Something that allows you to take the site offline and bring to bear the full resources of the server would have been useful, too.

[chilihead]

Nice catch @Omri Amos!

[media]

Grate job Amos....

[rllmukforum]

Hi @Omri Amos

I can't find that table hw_convert_link_posts and I searched the schema for any column called ipb_id and came up blank. What am I doing wrong?

[rllmukforum]

The query that seems to be running slowly most often on our server is

SELECT pid FROM `forums_posts` WHERE topic_id=263152 ORDER BY post_date ASC LIMIT 0,1;

I just ran this manually and it took 79 seconds!

There is an index on topic_id+queued+post_date+author_id but I don't think it's being used for this because if I run the same query without the ORDER BY, it takes 0.0seconds. Some of these threads are long - this one has 36000 replies.

Does anyone think creating a new index on topic_id+post_date is worth doing? Does MySQL's optimiser work best like that? 10 million posts so I expect it will have an impact on performance when I kick off creation of that so I'm not rushing into it!