Charset Conversion Libraries

[media]

[quote name='bfarber' date='26 March 2009 - 07:27 AM' timestamp='1238074039' post='1792594']
I'm afraid you're simply missing something. For instance, your site might be using the class_db_mysqli_client.php file - in which case the edit wouldn't do anything.

You really need to post in the peer to peer forums to get help with this sort of thing.



in my config file i have this



I will move this to pear section... :) thanks

$INFO['sql_driver'] = 'mySQL';

Can someone help to place this code mysql_query( "SET NAMES utf8", $this->connection_id ); into this function, maybe i am doing something wrong on placing it into the function 

 else

{ 

if ( IPS_MAIN_DB_CLASS_LEGACY )

{

$this->connection_id = @mysql_connect( $this->obj['sql_host'] ,

$this->obj['sql_user'] ,

$this->obj['sql_pass']

);

}

else

{

$this->connection_id = @mysql_connect( $this->obj['sql_host'] ,

$this->obj['sql_user'] ,

$this->obj['sql_pass'] ,

$this->obj['force_new_connection']

);

}

}

[media]

[quote name='Valtasar' date='07 February 2009 - 09:38 PM' timestamp='1234064300' post='1784420']
Hi,

I have converted the script to support batch queries, and uploaded it as a Resource (and here).
I still get some "MySQL Server has gone away", that I have to fix probably by retrying the query after reconnecting, but otherwise the conversion seems to be quite successful (of course extensive testing of the new IPB setup is now needed, to actually use this).


Valtasar,

I would like to thank you on this matter…

I have converted two of my Live IPB board to UTF-8 Charset completely by using your conversion Script….
I have tested everything and ALL LOOK GOOD AND DANDY…

You can post your script in resource section that someone can use it in the future…

I am truly appreciated you for your effort on this matter….

Thank you again…

[Valtasar]

[quote name='media' date='03 April 2009 - 04:48 PM' timestamp='1238777323' post='1794108']
Valtasar,

I would like to thank you on this matter…

I have converted two of my Live IPB board to UTF-8 Charset completely by using your conversion Script….
I have tested everything and ALL LOOK GOOD AND DANDY…

You can post your script in resource section that someone can use it in the future…

I am truly appreciated you for your effort on this matter….

Thank you again…

Bir şey değil my friend-I'm happy you finally made it! Indeed the "SET NAMES UTF-8" query IS required, and you have to change both files:
mysqli_client.php and mysql_client.php.

I have already posted this script on the Resources (category: IPB Tools/Generic Tools), and I also managed to convert my database successfully
to UTF-8 by modifying some mysql setting, as I already told you before (although it is not used yet in our live forum).
Also, as I already mentioned, one small problem with greek are a few differences (2-3 characters only) between some iso-8859-7 (used by IPB) and
the microsoft windows Greek encoding that is very common in Greece. Otherwise, everything seems to be fine, but still you MAY discover some issues (e.g.
files you need to encode in UTF-8), but I guess it's not something that is impossible to solve.
used

[osman84]

[quote name='media' date='24 March 2009 - 04:27 AM' timestamp='1237858031' post='1792106']
Hello,

I have finally got the database converted to UTF-8, but my problem is;

In PhpMyAdmin I can read my string right with utf-8, but on the site in IPB view all the strings from database are broken (I have already set charset in CP to utf-8 - Yes i have recreated skins in CP)

What would be the problem....???? (I see the browser is showing that site is using UTF-8 - Also I have cleared browser cache)

Do i need to change anything else in the script or ????? any suggestion or idea....

Thanks

I'm getting this same issue. I've converted to UTF8, and in PhpMyAdmin I can read my strings fine with UTF-8 encoding, but my IPB 3.0.0 displays these strings as question marks.

I tried putting:
$INFO['sql_charset'] = 'utf8'

But this gives me an error:

Fatal error: Uncaught exception 'Exception' with message 'Could not initiate the registry, the settings cache is empty or missing' in /home/ismaily/public_html/testupgrade/admin/sources/base/ipsRegistry.php:1627 Stack trace: #0 /home/ismaily/public_html/testupgrade/admin/sources/base/ipsRegistry.php(498): ipsRegistry->setUpSettings() #1 /home/ismaily/public_html/testupgrade/admin/sources/base/ipsController.php(75): ipsRegistry::init() #2 /home/ismaily/public_html/testupgrade/admin/sources/base/ipsController.php(62): ipsController->init() #3 /home/ismaily/public_html/testupgrade/index.php(24): ipsController::run() #4 {main} thrown in /home/ismaily/public_html/testupgrade/admin/sources/base/ipsRegistry.php on line 1627

[media]

[quote name='osman84' date='13 July 2009 - 02:44 AM' timestamp='1247474647' post='1825314']
I'm getting this same issue. I've converted to UTF8, and in PhpMyAdmin I can read my strings fine with UTF-8 encoding, but my IPB 3.0.0 displays these strings as question marks.

I tried putting:
$INFO['sql_charset'] = 'utf8'

But this gives me an error:




Hello,
First thing first...
If you upgrading your UTF-8 site to 3.0.0,
YOU HAVE TO PUT THAT LINE ($INFO['sql_charset'] = 'utf8' ) INTO CONFIG FILE BEFORE STARTING UPGRADE PROCESS
Otherwise some of your database will be broken...


Then that will set the name utf8 and you are good to go... :)

Good luck

Fatal error: Uncaught exception 'Exception' with message 'Could not initiate the registry, the settings cache is empty or missing' in /home/ismaily/public_html/testupgrade/admin/sources/base/ipsRegistry.php:1627 Stack trace: #0 /home/ismaily/public_html/testupgrade/admin/sources/base/ipsRegistry.php(498): ipsRegistry->setUpSettings() #1 /home/ismaily/public_html/testupgrade/admin/sources/base/ipsController.php(75): ipsRegistry::init() #2 /home/ismaily/public_html/testupgrade/admin/sources/base/ipsController.php(62): ipsController->init() #3 /home/ismaily/public_html/testupgrade/index.php(24): ipsController::run() #4 {main} thrown in /home/ismaily/public_html/testupgrade/admin/sources/base/ipsRegistry.php on line 1627

[Norki]

hi valtasar! thanks for your tools it half-saved my life.

let's my explain why "half" :P

well i've got a database of 400MB of posts and they are all in ISO-8995-1. i want them on UTF-8.

well, i've got fully converted the topics, the signature... but not the posts! i don't know why, the script reach the end with no errors but posts are always in ISO-8995-1.

i haven't chage the charset to the database, not locally and not live.

what can i do? thanks.

[Developer]

[quote name='Valtasar' date='05 February 2009 - 12:45 AM' timestamp='1233776730' post='1784018']
Many thanks to all of you for your nice words!
Yes, I can probably upload the script to resources, after a few improvements.

Well, you seem to want to try this on a backup copy of your database at your server - I think this is possible,
although I did it at my copy at home.
I do not know the current charset of your database, but you should try to run the script, on a database copy having exactly
the same content, however both the database copy and its tables should be created with utf8 encoding.
I did this by:
1) Exporting the database (with create tables commands) to an sql script
2) Replacing latin1 to utf8 everywhere in the script (create tables commands and client charset)
3) Creating a new utf8 database
4) Importing the sql script in this new utf8 database

Good luck and please tell us if it worked!
I hope it works better in your case - as I said in my case some posts/fields were not converted, and I am still checking what went wrong and if I find something I will post it here.

Hello,

I have a forum with Win1251 forum encoding and latin1_swedish_ci collation database.
Can you please outline step by step instructions here?
I am a bit lost with so many posts a few mentioned scripts.

First I will need to make a backup and export my database as SQL file and then run your script from command line?

Also could you please comment about these parameters?

Note two important issues:
- the "set names" command to the old encoding to read correctly from the database and
-the "_utf8" modifier before the utf8 string at the UPDATE command to write to the database. without it, the script did NOT work.
how?

$old_char_set = "iso-8859-7";
In my case it will be Win1251?

$alter_database=false; // if true,database and its tables (structure) will be converted to utf8 as well. NOTE: May alter length/type of some fields
Should this be set to true?

$autodetect_encodings="ISO-8859-7,UTF-8";
In my case Win1251, UTF-8?

[quote name='aljareh4ever' date='20 February 2009 - 07:12 AM' timestamp='1235095942' post='1785984']
finally after 3 months of searching the web for solution .

finally i found good script.

DataBase Charset Converter v2.2 done by AL3NDALEEB :thumbsup:

i think this will help i did not test it yet becouse

i'm away from my computer soon i'll test it and provide feedback her .


it's only one file that will convert your Db even the large one becouse it's done 500 per cycle .

never try it on live forum

also when you export your DB via phpmyadmin export it with the same Charset you use in your forum .

if you use latin1 in your forum export your DB in latin1 form because this script will mainly depend on that

to 100% convert your DB into utf-8 without problems .

i hope this will help :whistle: .

c22.zip

Has anyone tried this script yet?

[quote name='media' date='13 July 2009 - 06:35 PM' timestamp='1247492158' post='1825412']
Hello,
First thing first...
If you upgrading your UTF-8 site to 3.0.0,
YOU HAVE TO PUT THAT LINE ($INFO['sql_charset'] = 'utf8' ) INTO CONFIG FILE BEFORE STARTING UPGRADE PROCESS
Otherwise some of your database will be broken...


Then that will set the name utf8 and you are good to go... :)

Good luck

May it be that if I put it in config file and upgrade from 2.x to 3.x it will convert the database content to UTF-8 and the forum with UTF-8 encoding will start working just fine without running above script?

[Superman123]

In which file should I add mysql_query("SET NAMES 'utf8'"); for v3.0.5?

Thanks!

[bfarber]

Please use the peer to peer forums for support. The modification is no longer needed in 3.0.x. You just need to add a config parameter to your conf_global.php file.

[La Jobe]

I have just used the Valtasar's script that run slow (+6h to complete) but smooth. Thanks, for this great Job !

To fix the "mysql has gone away" error message (that happened opening my 260.000+ posts on ibf_posts table), i modified the php.ini with "mysql.connect_timeout = 600"

After a successful conversion (+ modify the class_db_mysql… 2 files) i now have 2 problems :

1/ The display looks like this : <{F_NAV}> <{F_NAV_SEP}>Forum Category<{F_NAV_SEP}>Forum name

2/ I can't log in in the PCAdmin because of a "Email address or password incorrect" error. Of course i know my log/pass by heart (and it's in automatic completion)

How can i rebuild caches without login in the PCA ?
How can i fix the login problem ?

Thanks for your help

[Edit] : i'm using IPB 2.3.6, not Converge and all this running on a local copy (with a hosts file targeting the real domain name on 127.0.0.1)

[bfarber]

Please use the peer to peer forums for help with such things.

[MadBox]

the script has delete all my database, converting 3.1.3 iso to utf8, edit the script put the database info, run an nothing happend, go to my forum and dont letme login go to phpmyadmin and my 10gigs database only has 200kb

what the hell i do wrong

[MadBox]

forget the last, either way don work form mi i have this error

			<div id='branding'>

				<h1>SQL Error</h1>

			</div>

		</div>

		<div id='content'>

			<div class='message error'>

				An error occured with the SQL server:

				<p class='message unspecific'>mySQL query error: INSERT INTO ibf_cache_store (`cs_key`,`cs_value`) VALUES('mail_processing',0) ON DUPLICATE KEY UPDATE cs_key=VALUES(cs_key),cs_value=VALUES(cs_value)</p>

				<p class='desc'>

					This is not a problem with IP.Board but rather with your SQL server. Please contact your host and copy the message shown above.

				</p>

			</div>


			<p class='message unspecific footer'>

				&laquo;<a href='/index.php' title='Go to home page'>Return to the index</a>

			</p>

		</div>

	</body>

[bfarber]

[quote name='bfarber' timestamp='1267025430' post='1917408']
Please use the peer to peer forums for help with such things.