media Posted October 20, 2015 Share Posted October 20, 2015 15 minutes ago, Rhett said: Yes just select the cron option and add it on your server/hosting etc. But I am right here: Processing RebuildPosts - IPS\forums\Topic\Post - 470915 / 1150652 Would I lose that 470915 ? Link to comment Share on other sites More sharing options...
Rhett Posted October 20, 2015 Share Posted October 20, 2015 Hello, no you won't lose anything, these run on their own in the background, or you can process them from the admincp, or you can use a cron, all good with either method, just use what suits you best. Link to comment Share on other sites More sharing options...
heartbreakers Posted November 30, 2015 Share Posted November 30, 2015 If selected to manually run the background processes can I still close my tab without switching to cron? Will the processes still run? Link to comment Share on other sites More sharing options...
media Posted November 30, 2015 Share Posted November 30, 2015 Yes... I did that way and it ran in the background and finished... Link to comment Share on other sites More sharing options...
heartbreakers Posted November 30, 2015 Share Posted November 30, 2015 5 hours ago, media said: Yes... I did that way and it ran in the background and finished... Hmm, that doesn't seem to work for me, so I think I'll run a cron job. On 10/20/2015, 1:17:34, Rhett said: Hello, no you won't lose anything, these run on their own in the background, or you can process them from the admincp, or you can use a cron, all good with either method, just use what suits you best. Using the Cron Jobs application from cPanel, is it sufficient to use the command given to me in the ACP without having to adjust any of the directories? For example, if this is the command /usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/***/public_html/***/applications/core/interface/task/task.php 1d1a46caf559609a0dc9229b4008e344 is it okay if I can't find a folder for /usr/, /local/, /bin/, or /php/ -- assuming those are folders/directories? Link to comment Share on other sites More sharing options...
media Posted November 30, 2015 Share Posted November 30, 2015 22 minutes ago, heartbreakers said: Hmm, that doesn't seem to work for me, so I think I'll run a cron job. Using the Cron Jobs application from cPanel, is it sufficient to use the command given to me in the ACP without having to adjust any of the directories? For example, if this is the command /usr/local/bin/php -d memory_limit=-1 -d max_execution_time=0 /home/***/public_html/***/applications/core/interface/task/task.php 1d1a46caf559609a0dc9229b4008e344 is it okay if I can't find a folder for /usr/, /local/, /bin/, or /php/ -- assuming those are folders/directories? Sorry, I thought you are talking about cronjob... I meant cronjob. You need to run the cronjob and you do not need to do anything... It keep running in the background until it finishes... Link to comment Share on other sites More sharing options...
heartbreakers Posted December 1, 2015 Share Posted December 1, 2015 21 hours ago, media said: Sorry, I thought you are talking about cronjob... I meant cronjob. You need to run the cronjob and you do not need to do anything... It keep running in the background until it finishes... Also, are the posts in forums supposed to be funky before your run the background processes and while the process runs? For example, looking through some posts in my test install, I notice that the quoted post isn't indicated as such. I'm assuming the background process will fix this? Thanks for your help! Link to comment Share on other sites More sharing options...
media Posted December 1, 2015 Share Posted December 1, 2015 1 hour ago, heartbreakers said: Also, are the posts in forums supposed to be funky before your run the background processes and while the process runs? For example, looking through some posts in my test install, I notice that the quoted post isn't indicated as such. I'm assuming the background process will fix this? Thanks for your help! Well, remember your data needs to be utf8 ready... Other than that background process fixes all that funky stuff Link to comment Share on other sites More sharing options...
heartbreakers Posted December 1, 2015 Share Posted December 1, 2015 1 hour ago, media said: Well, remember your data needs to be utf8 ready... Other than that background process fixes all that funky stuff Yeah, all my data was formatted to UTF8 even before the upgrade, so I'm hoping the rebuilding process fixes all the broken quotes and posts. Link to comment Share on other sites More sharing options...
heartbreakers Posted December 6, 2015 Share Posted December 6, 2015 Do I need to set permissions to the task.php file back to 644 after the cron job is finished? Link to comment Share on other sites More sharing options...
Omri Amos Posted December 6, 2015 Share Posted December 6, 2015 Hi, This is completely CRAZY SLOW - I have a board with ~5M posts (500K topics) and after a few HOURS it's only 0.15% done!! Running from CRON and/or manually, I'm trying everything I can but it's just SUPER SUPER SLOW. Please provide a modification that will at least allow this to run backwards - posts from 10 years ago and not that important and can be fixed within a few weeks/months, but I need all the posts from the last month or so to be fixed right away. I was thinking about modifying the 'offset' value of the task in the hw_core_queue table to (total posts - X latest posts), and then after it's finished to start it again from 0. Does that make sense? Link to comment Share on other sites More sharing options...
Dll Posted December 6, 2015 Share Posted December 6, 2015 Are you sure the task isn't locking for some reason. For instance we had a problem with the tasks being run for blogs and that was locking everything, fixing that made it all work much more quickly. Link to comment Share on other sites More sharing options...
Omri Amos Posted December 7, 2015 Share Posted December 7, 2015 Yes I'm sure... I was actually able to modify the task in order to work backwards: In "/applications/convert/extensions/core/Queue/RebuildContent.php" , Line 108, I changed ">" to "<" and "ASC" to "DESC". And in Line 130 I changed the 'if' statement so that it will not stop in the last (now first) id but instead on the first id (id=1). I started again (with a fresh backup of the unconverted 'posts' table, and a manual offset set to the same as 'count'+1) and it seems to be working fine. Still extremely slow (~1 post per second?!) but at least within a few hours I will have the posts from the last few days fixed. All the older posts can take their time, I don't mind... Link to comment Share on other sites More sharing options...
maddog107_merged Posted December 7, 2015 Share Posted December 7, 2015 5 hours ago, Omri Amos said: Yes I'm sure... I was actually able to modify the task in order to work backwards: In "/applications/convert/extensions/core/Queue/RebuildContent.php" , Line 108, I changed ">" to "<" and "ASC" to "DESC". And in Line 130 I changed the 'if' statement so that it will not stop in the last (now first) id but instead on the first id (id=1). I started again (with a fresh backup of the unconverted 'posts' table, and a manual offset set to the same as 'count'+1) and it seems to be working fine. Still extremely slow (~1 post per second?!) but at least within a few hours I will have the posts from the last few days fixed. All the older posts can take their time, I don't mind... I too have 5mil posts and upgraded last week. So a couple of things, first I increased the values of the content per cycle its doing in the applications/core/extensions/core/Queue folder. Also in MySQL ensure that you have query cache enabled as it does the same queries over an over. Same with PHP opcache (same script is executed each cycle). And finally just make sure you run it via Cron and not the UI. Took exactly a week to finish rebuilding everything (during which my users were pretty pissed off, and some still are as there are some issues to iron out) Link to comment Share on other sites More sharing options...
Izaya Orihara Posted December 7, 2015 Share Posted December 7, 2015 I am not happy about this personally. We are converting back to IPB and I am on 3.X for the moment and strongly considering staying on it(delaying the inevitable). Not because 3.X is better(it isn't) but it is so baffling and frustrating that it took me literally an hour to convert 550,000 posts, 12,000 topics, 2200 members from a completely entirely different software to IPB3. However(both times we have had to deal with background content), it has taken DAYS to just upgrade from 3.X to 4.X I still have no idea why they removed the amazing functionality of the background process in 3.X and replaced it with this. Link to comment Share on other sites More sharing options...
Omri Amos Posted December 7, 2015 Share Posted December 7, 2015 OK, it seems to be extremely slow because this query (with different id's) called all the time and each one is taking ~10 seconds (!!) to complete... SELECT * FROM `hw_convert_link_posts` WHERE `ipb_id` =4977531 AND `type` = 'forums_posts' AND `app` =2 (This is a 5M rows table) So... I added a new unique index: And now these queries take ~0.0001 sec each!!!! OMG!! Now instead of ~200 posts per HOUR, it goes ~200 posts per SECOND. Link to comment Share on other sites More sharing options...
stoo2000 Posted December 7, 2015 Share Posted December 7, 2015 9 hours ago, Omri Amos said: Yes I'm sure... I was actually able to modify the task in order to work backwards: In "/applications/convert/extensions/core/Queue/RebuildContent.php" , Line 108, I changed ">" to "<" and "ASC" to "DESC". And in Line 130 I changed the 'if' statement so that it will not stop in the last (now first) id but instead on the first id (id=1). I started again (with a fresh backup of the unconverted 'posts' table, and a manual offset set to the same as 'count'+1) and it seems to be working fine. Still extremely slow (~1 post per second?!) but at least within a few hours I will have the posts from the last few days fixed. All the older posts can take their time, I don't mind... Yeah, don't do this. What you're now doing is altering the offset of data to process each time a post is added, you're actually going to end up rebuilding posts multiple times over instead of 1 time per post. Also bear in mind that the parser is intended to be ran on legacy data (3.4) running it again on 4.x data may very well break it. Link to comment Share on other sites More sharing options...
Omri Amos Posted December 7, 2015 Share Posted December 7, 2015 56 minutes ago, stoo2000 said: Yeah, don't do this. What you're now doing is altering the offset of data to process each time a post is added, you're actually going to end up rebuilding posts multiple times over instead of 1 time per post. Also bear in mind that the parser is intended to be ran on legacy data (3.4) running it again on 4.x data may very well break it. The forum is closed so no new posts are added, I double checked - no post is being parsed more than once (which will indeed break it). Either way it is no longer really needed because now after I added the missing index (see my prev reply) I can finish the entire 5M table in just about ~12-24 hours - instead of 10+ MONTHS..... Link to comment Share on other sites More sharing options...
stoo2000 Posted December 7, 2015 Share Posted December 7, 2015 2 minutes ago, Omri Amos said: The forum is closed so no new posts are added, I double checked - no post is being parsed more than once (which will indeed break it). Either way it is no longer really needed because now after I added the missing index (see my prev reply) I can finish the entire 5M table in just about 2-3 hours - instead of 10+ MONTHS..... You may have thought that through (good job!). But others will read this topic and blindly follow what you're saying without thinking about this. Link to comment Share on other sites More sharing options...
Ryan Ashbrook Posted February 17, 2016 Share Posted February 17, 2016 On 12/7/2015 at 6:18 AM, Omri Amos said: OK, it seems to be extremely slow because this query (with different id's) called all the time and each one is taking ~10 seconds (!!) to complete... SELECT * FROM `hw_convert_link_posts` WHERE `ipb_id` =4977531 AND `type` = 'forums_posts' AND `app` =2 (This is a 5M rows table) So... I added a new unique index: And now these queries take ~0.0001 sec each!!!! OMG!! Now instead of ~200 posts per HOUR, it goes ~200 posts per SECOND. I've gone ahead and incorporated this for the next release - the indexes were not correct anyway (carry-over from the 3.x converters schema), so I have re-done them. Link to comment Share on other sites More sharing options...
rllmukforum Posted February 25, 2016 Share Posted February 25, 2016 That would have been handy for us, Ryan Of about 10 million post rows, the manual job has rebuilt 55% in 14 days. Something that allows you to take the site offline and bring to bear the full resources of the server would have been useful, too. Link to comment Share on other sites More sharing options...
chilihead Posted February 25, 2016 Share Posted February 25, 2016 Nice catch @Omri Amos! Link to comment Share on other sites More sharing options...
media Posted February 25, 2016 Share Posted February 25, 2016 Grate job Amos.... Link to comment Share on other sites More sharing options...
rllmukforum Posted February 25, 2016 Share Posted February 25, 2016 Hi @Omri Amos I can't find that table hw_convert_link_posts and I searched the schema for any column called ipb_id and came up blank. What am I doing wrong? Link to comment Share on other sites More sharing options...
rllmukforum Posted February 25, 2016 Share Posted February 25, 2016 The query that seems to be running slowly most often on our server is SELECT pid FROM `forums_posts` WHERE topic_id=263152 ORDER BY post_date ASC LIMIT 0,1; I just ran this manually and it took 79 seconds! There is an index on topic_id+queued+post_date+author_id but I don't think it's being used for this because if I run the same query without the ORDER BY, it takes 0.0seconds. Some of these threads are long - this one has 36000 replies. Does anyone think creating a new index on topic_id+post_date is worth doing? Does MySQL's optimiser work best like that? 10 million posts so I expect it will have an impact on performance when I kick off creation of that so I'm not rushing into it! Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.