suggestion: ssh converter/importer

[ttech]

there is a feature that i've seen in woltlab burning board and would like to see in IPB
it can be very useful for big boards
in the admin panel converter section: after entering the source database information you have 2 choices: either run it on the web like most converters or generate a ssh script that does it
if you choose the latter a command will appear and all you have to do is run it in ssh
i have tested it and i'm confident its made my import at least 10 times faster

[Mark]

It's being worked on :)

[TCattitude]

May i ask when?

I ask that, because we are a big-board with our new invision's licence in hand (ready to install), and one of our (two) biggest concerns it is in fact the database size (10GB now).
Being able to use some sort of ssh script to import from vBulletin to Invision will be definitely a plus for us.

Greetings.

[ttech]

any updates on this one?

[Akabane]

The post import / recount thingy, ssh or not, is a joke.... :(

24h (a bit more actually) for 715 000 msg on a good server. :x

[TCattitude]

Akabane... and guest how much time for 16M posts.
We did it. A holy PITA. But we did it.

[Akabane]

Ouchhhh. 3x12h for a better server hosting 3M msg so I can't tell but I guess you had fun. :P

IPB is the best product of its kind atm but I would love better scripts for that. :P

[Mark]

You could bump up the amount it does per cycle. It's only as slow as your server ;)

[ttech]

still an ssh option could make it much more faster
wouldn't it?

[TCattitude]

Mark, sure, you can bump the amount of data per cycle, it's not hard if you have server access. We did it, and instead of 1000 or 2000 per cycle, we did cycles of 10000 and 15000.
But if, for example, every 2000 cycle takes 30 seconds to complete, bumping it to 4000 per cycle does not mean 30 seconds for that too. It increase, maybe to 45 or 50 seconds per cycle, but increace the time too. And with 16M posts... that extra seconds become minutes, and then hours... and days. A lot of days.

Until now, i'm really happy with Invision. Our servers (load balanced) are running much more lighter than we was using vB.

The only big problem it's that the importer/converter it's not really thinked and prepared to run on big-boards with a lot of data. Really (and that was not the only problem we faced with the converter).
It's not a problem only affect you, guys. vB does not have a ssh converter/impex also, just the web based.

Having 1M posts or more can make the conversion process a really tricky and not pleasant thing.
If the idea it's to bring here new not-happy-people-with-vB-InternetBrands-thing, like me, it's definitely worth thinking how to improve the conversion process for big-boards.

Greetings ;)

[bfarber]

Just as a quick note, we also offer a conversion service. Anyone with a really big board (16M posts definitely counts ;) ) may be interested in just paying us to do the conversion for you. Then you don't have to worry about any of it. :)

I'm just putting that out there in case anyone wasn't aware we offer this service.

[ttech]

It's being worked on :)

hello Mark, can you give us an update

[accyroy]

I'd also love to see an updated convertor available from the command line. I am struggling importing a forum with just 250,000 posts, the server canny take no more and load just skyrockets!
Is there any way of delaying the refresh of the cycle page in the current convertor? I actually need to slow down the convertor to give the server chance do do other things as well as the conversion.

[Connor T]

I used SSH to convert my posts and PMs. It asked me what ID to start at, and how many I wanted to do per cycle.

Were you not given those options?

[accyroy]

Nope, not given any SSH options.. Maybe I did something different. I've got it going now, had to change the cycle page refresh from 2 seconds to 10 seconds.. Taking its time but at least its not overloading the server! I don;t know what I'm going to do with my bigger forums though...

[Connor T]

ooooo.

Re-download whatever version your using and look in the "Tools" folder.

Upload the needed files (rebuildPMs.php or rebuildPosts.php) to the root folder path (area that contains conf_global)

Then visit it on ssh like

php path to file

[Cap'n Refsmmat]

Aren't those (rebuildPMs and rebuildPosts) the same as the Rebuild Content features in the ACP? If so, they're not the same as the full converter. You can't run those two scripts to import a vBulletin install, for example.