IPB 3.1.x + Sphinx + non-latin, non-english, greek, UTF8 characters

[titos]

Took me a while to figure it out, so I thought I may as well share how I managed to make my IPB 3.1.x installation work with Sphinx, when most of my content is in Greek.

Some information is taken from this topic (and hence from here too).

Note that my DB, character encoding, everything is UTF8. I use Sphinx 2.0.1-beta.

Edit your sphinx.conf file, and

[*]in every source construct, add a sql_query_pre directive at the beginning, for example:

source core_search_main : ipb_source_config

{

sql_query_pre	= SET CHARACTER SET utf8

...

Note that some source constructs may already have an sql_query_pre directive, in which case simply add another one just above it. For example:

source core_search_main : ipb_source_config

{

sql_query_pre	= SET CHARACTER SET utf8

sql_query_pre	= REPLACE INTO ibf_cache_store VALUES( 'sphinx_core_counter', (SELECT max(id) FROM ibf_faq), '', 0, UNIX_TIMESTAMP() )

...

[*]In every index construct, change all charset_type = sbcs instances to charset_type = utf-8 and add just below:

charset_table 	= 0..9, A..Z->a..z, _, -, a..z,U+370->U+371, U+371, 

U+372->U+373, U+373, 

U+374->U+375, U+375, 

U+376->U+377, U+377, 

U+37a, 

U+3fd->U+37b, U+3fe->U+37b, U+3ff->U+37b, U+37b, 

U+3fe->U+37c, U+37c, 

U+37e, 

U+386->U+3b1, 

U+388->U+3b5, 

U+389->U+3b7, 

U+38a->U+3b9, 

U+38c->U+3bf, 

U+38e->U+3c5, 

U+38f->U+3c9, 

U+390->U+3b9, 

U+3aa->U+3b9, 

U+3ab->U+3c5, 

U+3ac->U+3b1, 

U+3ad->U+3b5, 

U+3ae->U+3b7, 

U+3af->U+3b9, 

U+3b0->U+3c5, 

U+3ca->U+3b9, 

U+3cb->U+3c5, 

U+3cc->U+3bf, 

U+3cd->U+3c5, 

U+3ce->U+3c9, 

U+3cf->U+3d7, U+3d7, 

U+3d0->U+3b2, 

U+3d1->U+3b8, 

U+3d2->U+3c5, 

U+3d3->U+3c5, 

U+3d4->U+3c5, 

U+3d5->U+3c6, 

U+3d6->U+3c0, 

U+3d8->U+3d9, U+3d9, 

U+3da->U+3db, U+3db, 

U+3dc->U+3dd, U+3dd, 

U+3de->U+3df, U+3df, 

U+3e0->U+3d1, U+3d1, 

U+391..U+3a1->U+3b1..U+3c1, U+3b1..U+3c1, 

U+3a3..U+3a9->U+3c3..U+3c9, U+3c3..U+3c9

so you end up with something like:

index XXX

{

	...

	charset_type	= utf-8

	charset_table 	= 0..9, A..Z->a..z, _, -, a..z,U+370->U+371, U+371, 

U+372->U+373, U+373, 

U+374->U+375, U+375, 

U+376->U+377, U+377, 

U+37a, 

U+3fd->U+37b, U+3fe->U+37b, U+3ff->U+37b, U+37b, 

U+3fe->U+37c, U+37c, 

U+37e, 

U+386->U+3b1, 

U+388->U+3b5, 

U+389->U+3b7, 

U+38a->U+3b9, 

U+38c->U+3bf, 

U+38e->U+3c5, 

U+38f->U+3c9, 

U+390->U+3b9, 

U+3aa->U+3b9, 

U+3ab->U+3c5, 

U+3ac->U+3b1, 

U+3ad->U+3b5, 

U+3ae->U+3b7, 

U+3af->U+3b9, 

U+3b0->U+3c5, 

U+3ca->U+3b9, 

U+3cb->U+3c5, 

U+3cc->U+3bf, 

U+3cd->U+3c5, 

U+3ce->U+3c9, 

U+3cf->U+3d7, U+3d7, 

U+3d0->U+3b2, 

U+3d1->U+3b8, 

U+3d2->U+3c5, 

U+3d3->U+3c5, 

U+3d4->U+3c5, 

U+3d5->U+3c6, 

U+3d6->U+3c0, 

U+3d8->U+3d9, U+3d9, 

U+3da->U+3db, U+3db, 

U+3dc->U+3dd, U+3dd, 

U+3de->U+3df, U+3df, 

U+3e0->U+3d1, U+3d1, 

U+391..U+3a1->U+3b1..U+3c1, U+3b1..U+3c1, 

U+3a3..U+3a9->U+3c3..U+3c9, U+3c3..U+3c9

	...

}

[*]Rebuild your indexes.

[Valtasar]

Many thanks for your good guide - nice to see that Sphinx 2.0.1-beta works fine with IPB.
Just one addition (that bfarber forgot in his charset table):

Add:

						U+3c2->U+3c3, \

anywhere in (2) to map "final sigma" to "small sigma".