Jean-Philippe Posted January 18, 2016 Posted January 18, 2016 Hi all, I have to convert (very, very) old forums for which there is no available converter. So I decided to write my own converter for the members and forums database, using REST API . It is easy to import members (or topic, or posts) to IPS 4 with REST API ... BUT ... I have a big problem : calls to REST API (with a PHP script) are very slow : a call to core/members to create a member, or a call to forums/create take 0,5s (on a big server) For 35.000 members creation using members/create it will take >10 hours ! For 2 millions posts creation using forums/create it will take an eternity (or more ...) My questions : Q1. Is it stupid to use the REST API to import custom forums ? Q2. is there a solution to reduce execution time of REST API calls ? Q3. Is there a php library offering functions to create members/topics/posts ? ... (and if yes where may I found documentation ?) Q4. who may I contact for a custom import of our forums ? Thank you for your help.
Jean-Philippe Posted January 18, 2016 Author Posted January 18, 2016 Our forums are 19 years old ... they where built on a old version of UBB 5.38 with many customisations. It's time for us to convert to modern forums like IPS 4
MADMAN32395 Posted January 18, 2016 Posted January 18, 2016 12 minutes ago, Jean-Philippe said: Our forums are 19 years old ... they where built on a old version of UBB 5.38 with many customisations. It's time for us to convert to modern forums like IPS 4 could contact sales and see how painful the conversion would be for you for them to do it. might be easier and more beneficial.
Jean-Philippe Posted January 18, 2016 Author Posted January 18, 2016 I would like to make conversion by myself I have foudn answer to Q2 : asynchronous calls to REST API. With 100 asynchronous calls to REST API, creation of 100 members takes only 1s But, I think the best solution would be to use php scripts ... and answer to my Q3 question would be a great help (if an API for IPS4 exist).
MADMAN32395 Posted January 18, 2016 Posted January 18, 2016 2 hours ago, Jean-Philippe said: I would like to make conversion by myself I have foudn answer to Q2 : asynchronous calls to REST API. With 100 asynchronous calls to REST API, creation of 100 members takes only 1s But, I think the best solution would be to use php scripts ... and answer to my Q3 question would be a great help (if an API for IPS4 exist). the REST API is for IPS4. this is the help reference that is in the ACP. https://drive.google.com/file/d/0B0mBb9QuGv84aHFsaU9XRlVKZTQ/view?usp=drivesdk and this is where the help button brings you. https://community.invisionpower.com/4guides/developing-plugins-and-applications/rest-api/
Stuart Silvester Posted January 18, 2016 Posted January 18, 2016 The REST API is really not intended for conversions.... We have a converter for that. You can build your own custom converter within the framework of the Converter app for IPS4.
Jean-Philippe Posted January 19, 2016 Author Posted January 19, 2016 7 minutes ago, Stuart Silvester said: The REST API is really not intended for conversions.... We have a converter for that. You can build your own custom converter within the framework of the Converter app for IPS4. Build my own converter is what I would like to do. It's easy to convert members with the REST API, but to slow to convert millions posts Where can I found the "framework of the Converter app for IPS4" and the associate documentation ? Thank you for your help !
Daniel F Posted January 19, 2016 Posted January 19, 2016 13 hours ago, Jean-Philippe said: Build my own converter is what I would like to do. It's easy to convert members with the REST API, but to slow to convert millions posts Where can I found the "framework of the Converter app for IPS4" and the associate documentation ? Thank you for your help !
Jean-Philippe Posted January 19, 2016 Author Posted January 19, 2016 @Daniel F : There is not link to the framework on your reply ... and I already had a look to the page concerning conversions, but I need a "custom" conversion. I can convert members with REST API (with asynchronous http requests to REST API I can create 10.000 members in less than 10 minutes) I only have to convert Topics and Posts with many posts, which are in a very simple structure, in 2 tables : Table Topics : IdForum, IdTopic, Author, Title, Date, Content in html Table Posts : IdTopic, Author, Date, Content in html It's easy to make the conversion with REST API through HTML requests ... but not realistic because too slow (approx. 150.000 topics, and 2 millions posts) What would be perfect for me, would be an example of complete php code equivalent to the 2 REST API entries : forums/posts and forums/topics And ... may be there is a php library equivalent to the REST API ? Thank by advance for help ...
Daniel F Posted January 19, 2016 Posted January 19, 2016 The linked app includes the "framework" Also it's not recommended to use the REST API for conversions because of all the overhead and additional operations like sending notifications, inserting into search index, rebuilding likes count etc for each record. We had an internal discussion about this and we ended with: Quote Converters should, generally speaking, insert data directly into the database and then rebuild things at the end.
Jean-Philippe Posted January 19, 2016 Author Posted January 19, 2016 @Daniel F : I'm sorry, I don't understand. In this topic, I can see your 3 replies (and I thank you for your help), but : I don't see any link to the "framework of the Converter app for IPS4" ... What is the "linked app" ? ... I don't see any link to any app in your replies You mention an "internal discussion" in your last reply ... what do you mean by "internal discussion" ? (is it a discussion between members of IPS team ? or a discussion on another topic ?)
Stuart Silvester Posted January 19, 2016 Posted January 19, 2016 3 minutes ago, Jean-Philippe said: @Daniel F : I'm sorry, I don't understand. In this topic, I can see your 3 replies (and I thank you for your help), but : I don't see any link to the "framework of the Converter app for IPS4" ... What is the "linked app" ? ... I don't see any link to any app in your replies You mention an "internal discussion" in your last reply ... what do you mean by "internal discussion" ? (is it a discussion between members of IPS team ? or a discussion on another topic ?) The converter app itself is the framework, you can add new converters to it in exactly the same way that we do. I would recommend having a look at the code for converters we already include. You can find these in Sources/Software/Core/*.php and Sources/Software/Board/*.php Daniel posted a link to the converter app above ( https://community.invisionpower.com/topic/425461-rest-api-calls-slow-forums-convertion-php-library/?do=findComment&comment=2611654 ) Yes, Daniel is referencing a staff discussion. If you're not comfortable with doing this yourself (this isn't directed at you Jean), our sales team will be able to provide a quotation for a custom conversion. Email sales@invisionpower.com for more information.
Jean-Philippe Posted January 19, 2016 Author Posted January 19, 2016 Thank you very much for your help !
Recommended Posts
Archived
This topic is now archived and is closed to further replies.