GreenLinks Posted June 14, 2014 Share Posted June 14, 2014 Hello , Convertor may work for small websites in current state but for big boards it is not feasible. Current format risks too long down times for any big board. The tables are being forced to be created on same database. So if anything goes wrong big boards need to drop their database , reimport the database and start from scratch. Exporting a database takes around : 15 min Importing a database takes around : 1 hour Conversion is completed around : 2 hour 30 minute Collation change is taking around : 5 hours to compete You need to make sure to eliminate the need of exporting and importing the databases so we will not have to be forced to take down our boards longer then it is needed. If we assume upgrade is going to take around 2 hour ( table changes etc. ) IPB teams main focus should be on making sure no additional senario's can cause downtimes. So allowing a separate database should be the first target atm... Link to comment Share on other sites More sharing options...
Aiwa Posted June 14, 2014 Share Posted June 14, 2014 We talking about the converter that converts you over from another software, or the UTF8 converter? I'm assuming the utf8 converter, as I saw you were testing it the other day.... The utf8 converter creates NEW tables in your database so it's not messing with your live site. Then you simply update your conf_global sql prefix to test the conversion... Once it's good, you switch it back and finalize the utf8 conversion. If the conversion doesn't go well, you only need to drop the NEW tables, and ensure your conf_global sql prefix is pointing to your original tables, not the whole database... Why would you have import your database? Link to comment Share on other sites More sharing options...
GreenLinks Posted June 14, 2014 Author Share Posted June 14, 2014 We talking about the converter that converts you over from another software, or the UTF8 converter? I'm assuming the utf8 converter, as I saw you were testing it the other day.... The utf8 converter creates NEW tables in your database so it's not messing with your live site. Then you simply update your conf_global sql prefix to test the conversion... Once it's good, you switch it back and finalize the utf8 conversion. If the conversion doesn't go well, you only need to drop the NEW tables, and ensure your conf_global sql prefix is pointing to your original tables, not the whole database... Why would you have import your database? If anything goes wrong within the convertor example ( on our last test conversion we found some table renaming missing ) , you basically need to start from scratch or manually have to rename all those tables. Which is another major time consuming thing UTF8 Convertor should work as simple as possible and should not interfere with the main database. That shall always be the principle. Link to comment Share on other sites More sharing options...
Aiwa Posted June 14, 2014 Share Posted June 14, 2014 If it's not creating all new tables, with a new prefix, that's a bug, and needs to be reported. You should test the conversion before you finalize it... update your conf_global.php sql prefix to test the new conversion before you finalize and all the converted tables get renamed to take the place of your original tables... If that finalize isn't renaming the tables properly, that's also a bug and needs to be reported... The UTF8 converter must interface with your main database to get the data for conversion. Other than the data residing on new tables, your live tables and converted tables are mutually exclusive until you finalize the conversion. So it shouldn't affect your live site until your finalize and make the switch... You should ALWAYS test the conversion before you finalize it. Link to comment Share on other sites More sharing options...
GreenLinks Posted June 14, 2014 Author Share Posted June 14, 2014 If the team doesn't make necessary changes to assist big board owners within their tests , then highly possible we will stop reporting the bugs Aiwa. This conversion process needs to be flawless, thats why it shall include the most effective test practises which lays on having all changes on separate databases. I am sorry but i seriously don't see a reason how you can arguing against this. Math is extremely simple here and the outcome is again extremely simple. If anything goes wrong : If conversion done on a separate database : Dropping the secondary database - creating a new database takes : 2 min max If conversion done on same database : Multiple choices Clean the current database -> 60-90 min manual work Drop your test database, create a new database and first import your backup -> At least 60-70 minute wait time Link to comment Share on other sites More sharing options...
Aiwa Posted June 14, 2014 Share Posted June 14, 2014 I'm not arguing against it. I'm explaining how it's supposed to work... The way I'm reading what you're saying, something didn't work that way... which means a bug needs reported. The test data, while still on the same database, should still be mutually exclusive and shouldn't require a dump of your whole database and a restore.. I understand your want for all of the data to be converted into a new database, and I've got no problem with that suggestion, but if it's because something didn't work, that something that didn't work needs fixed... Link to comment Share on other sites More sharing options...
GreenLinks Posted June 14, 2014 Author Share Posted June 14, 2014 Sure lets report some bugs which are mainly highlighted in the ticket already : Timeout after huge table conversions Script fails to remove orig. named tables which doubles database sizes ( requires manual work , unacceptable ) Indexes are deleted during the conversion These are explained in the ticket ( expect indexes ) but we don't know if they are fixed or not . I did a final test after yesterdays upgrade ( all bugs unfortunately still exist ) Each test takes around 8 hours so if you are expecting us to go on with these tests , then you shall first act and implement improvements that will ease our job. Link to comment Share on other sites More sharing options...
Aiwa Posted June 14, 2014 Share Posted June 14, 2014 Let me first say that I respect you taking as much time as you have to run such tests. I'm sure IPS appreciates it more than you know... Ok, great, you have a ticket open about the issues. I wasn't aware of the specific issues you were having. Link to comment Share on other sites More sharing options...
Rhett Posted June 14, 2014 Share Posted June 14, 2014 The last post from you I saw was that your site only took 35 minutes to convert with 19 mil post... that was great news... now I see this, I'll check on your ticket and I'm sure we will address any issues though and do the best we can to adapt for larger sites. Nay , close to 19 mil post conversion took around 35 minute max. Link to comment Share on other sites More sharing options...
DevNTG Posted June 14, 2014 Share Posted June 14, 2014 The conversion took well over 6 hours for me, with 150,000 posts. Not only did it take that long, it frequently stopped on the posts table saying zend_mm_heap corrupted.That means, the board will have to be offline for at least 6 hours so that it converts all updated rows. Running it on a live site would be pointless. Link to comment Share on other sites More sharing options...
chromaniac Posted June 14, 2014 Share Posted June 14, 2014 I am on a dedicated with around 800,000 posts. It was pretty fast for me. But then the server has very little load at nights which is when I did the conversion. I guess having a server with enough resources to hand a forum with 10 times the traffic helped. Link to comment Share on other sites More sharing options...
elnomio Posted June 14, 2014 Share Posted June 14, 2014 This converter is totally catastrophic! Link to comment Share on other sites More sharing options...
GreenLinks Posted June 14, 2014 Author Share Posted June 14, 2014 The last post from you I saw was that your site only took 35 minutes to convert with 19 mil post... that was great news... now I see this, I'll check on your ticket and I'm sure we will address any issues though and do the best we can to adapt for larger sites. Yes , however that was when we first noticed the time out issues. Another thing i noticed is those timeouts specifically occur when you use 4-byte UTF-8 Encoding. On my mac pro total time to complete the conversion is around 5 hours vs 8 hours on test server but the difference was Use 4-byte UTF-8 Encoding Merts-Mac-Pro:GcForum lizardking$ php utf8convert/cli.php ------------------------------------------ Welcome to the IPS UTF8 Conversion utility ------------------------------------------ 303 table(s) are not UTF-8 and need converting. Use 4-byte UTF-8 Encoding (utf8mb4)? Some non-common symbols (such as historical scripts, music symbols and Emoji) require more space in the database to be stored. If you choose 'no', users will not be able to use these symbols on your site. If 'yes', these characters will be able to be used, but the database will use more disk space. [y] Enter 'y' for Yes [n] Enter 'n' for No n This utility will convert all the tables in this database to UTF-8. The converted data is inserted into new tables prefixed with x_utf_ and the original data kept. Press enter to continue Processing: 100% Conversion has completed in 2 hour(s), 24 minute(s) and 47 seconds. [f] Enter 'f' to finish [x] Enter 'x' to reset and start the conversion again. f You can now test the conversion by: Editing conf_global.php to add $INFO['sql_charset'] = 'UTF8'; Change $INFO['sql_tbl_prefix'] to: $INFO['sql_tbl_prefix'] = 'x_utf_'; If you are happy with the conversion, you can have it permanently rename the tables so that you can restore your original 'sql_table_prefix' and delete the original tables. [y] Enter 'y' to rename your tables [x] Enter 'x' to reset and start the conversion again. y Renaming tables, this may take a short while. Checking and fixing collations. You can now complete the conversion by editing conf_global.php to add $INFO['sql_charset'] = 'UTF8'; and to change $INFO['sql_tbl_prefix'] to: $INFO['sql_tbl_prefix'] = ''; I still have all the orig named tables which is clearly a sign that the convertor only received minor corrections. Link to comment Share on other sites More sharing options...
GreenLinks Posted June 14, 2014 Author Share Posted June 14, 2014 One more info Rhett , Non converted tables ( ones that has x_utf_ prefix are usually related to IP.Convertor related ) Link to comment Share on other sites More sharing options...
Rhett Posted June 14, 2014 Share Posted June 14, 2014 Matt was going to add in a method to remove the orig_tables I believe, I'll check on this as I do agree having to do it manually is a bit of work. Link to comment Share on other sites More sharing options...
elnomio Posted June 14, 2014 Share Posted June 14, 2014 Wait! What? What exactly converter does with all those tables? As I understood it creates both orig_ and x_utf_ prefixed tables and then delete orig_ and rename x_utf_. Is that correct? Or I should rename all orig_ tables manually? Link to comment Share on other sites More sharing options...
stoo2000 Posted June 14, 2014 Share Posted June 14, 2014 Wait! What? What exactly converter does with all those tables? As I understood it creates both orig_ and x_utf_ prefixed tables and then delete orig_ and rename x_utf_. Is that correct? Or I should rename all orig_ tables manually? orig_ tables are the original encoded tables, it doesn't affect that. The final step of the converter renames the x_utf_ tables to your defined prefix, so they take the names that the original tables did have. Link to comment Share on other sites More sharing options...
elnomio Posted June 14, 2014 Share Posted June 14, 2014 My defined prefix - no prefix) Just ''. Using 1.1.7 I got both orig_ and x_utf_ tables, using 1.1.8 I got orig_ yables that does not work right. I don`t know what should I do to make it. Link to comment Share on other sites More sharing options...
stoo2000 Posted June 14, 2014 Share Posted June 14, 2014 My defined prefix - no prefix) Just ''. Using 1.1.7 I got both orig_ and x_utf_ tables, using 1.1.8 I got orig_ yables that does not work right. I don`t know what should I do to make it. Ok, so those tables with orig_ are the ones that were there before the converter was used, x_utf_ was a temporary prefix for the 'testing' step of the converter. The final step of the converter removes the x_utf_ prefix and replaces it with your original prefix (if you had one). So, in summary you're supposed to have two sets of tables now, one with (or with your original prefix) and one set with the orig_ prefix. Link to comment Share on other sites More sharing options...
elnomio Posted June 14, 2014 Share Posted June 14, 2014 I have no any orig_ tables in my database before conversion. They appear right after I convert my database. Link to comment Share on other sites More sharing options...
stoo2000 Posted June 14, 2014 Share Posted June 14, 2014 I have no any orig_ tables in my database before conversion. They appear right after I convert my database. Yes, that is correct, they are the original unconverted tables, renamed to preserve their content. Link to comment Share on other sites More sharing options...
elnomio Posted June 14, 2014 Share Posted June 14, 2014 Yeah, but Converter does not rename them. That`s the problem. They still orig_ tables. Link to comment Share on other sites More sharing options...
stoo2000 Posted June 14, 2014 Share Posted June 14, 2014 Yeah, but Converter does not rename them. That`s the problem. They still orig_ tables. No, they originally were your tables without a prefix, they HAVE been renamed to have an orig_ prefix. This is perfectly normal. Link to comment Share on other sites More sharing options...
elnomio Posted June 14, 2014 Share Posted June 14, 2014 So why does my forum does not work correct with them? Link to comment Share on other sites More sharing options...
stoo2000 Posted June 14, 2014 Share Posted June 14, 2014 So why does my forum does not work correct with them? Your forum config shouldn't be point to these tables, as per the instructions your prefix should be set to '' (since you don't use a custom prefix) How it worksThe converter will create new tables with the prefix "x_utf_". The converter will read your existing data, convert it to UTF-8 and then save it into the "x_utf_" prefixed tables. Once the data conversion has taken place, you can test the conversion by editing your conf_global.php file to alter the sql_prefix. The exact instructions for this are shown when the conversion has finished. Once you are happy that the conversion is correct, you are prompted to finish the conversion. This will rename your existing database tables with the prefix "orig_" and the UTF-8 tables will be renamed to your existing prefix. You are then prompted again to edit your conf_global.php file to alter the sql_prefix. You can then delete the "orig_" prefixed tables when you are happy that the conversion is complete and the original unconverted data is no longer required. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.