bfarber

It's timing out, simply.

Are you running php from command line? If not, on a medium or larger database it's going to be very hard to try to run this script through your browser.

Firstly, you should run the script from command line. If you do, your .htaccess changes will have no impact because you're not using apache. Additionally, you'll need to get rid of memory limit and max execution times to give it enough power to finish.

Secondly, I do agree, processing in batches might prove more reliable. What will happen if someone has 1M posts?

Valtasar,

Thank you for this, but your post is too technical for us....

What do you mean by "LOCAL COPY ONLY"????

If i make a back up from my database with PHPMYADMIN and run this script on my backup on the server, CAN I DO THIS?

Yes, I think he's just saying not to use this on your live site as of yet. ;) Could/probably will break something until more testing is done.

Impressive Valtasar. :) This isn't the easiest thing to deal with by any means.

Hi,

I followed this thread (and the corresponding IPB3 thread) and I am also very much interested in this issue.

I attempt to migrate an existing IPB2.3.5 database from iso-8859-7 to utf8.

I will try to write such an automatic conversion php script and post it here for everybody to try, based on the IPB3 script you provided and the i18n charset library.

However, before I do this, I need to ask you some important questions:

1) Since I haven't studied any IPB3 code, will the IPB2 function:

$post = $this->ipsclass->txt_convert_charsets($r['post'] , 'old_char_set', 'utf-8' );

do exactly the same thing as the IPB3 function you used:

$post = IPSText::convertCharsets( $r['post'], 'old_char_set', 'utf-8' );

or not?

Yes, it is effectively the same function.

2) I will automatically get a list of IPB fields to convert. I plan to convert the fields:

char varchar text enum set tinytext mediumtext longtext

Should I convert some of them? Or maybe there are more to convert? Can you please specify the proper set?

We do not use any enum, set, or longtext fields, to my knowledge, but you can look and convert them if there are any. The list is fine otherwise.

3) In the database I see a lot Greek characters converted to html entities for some reason (as I said iso-8859-7 was used as the charset in IPB) that I assume cause problems in searches. Is this handled by txt_convert_charsets() function?

Should I try to convert these entities to utf8 e.g. using a standard PHP function like html_entity_decode() or mb_convert_encoding(), or not?

This is most likely from AJAX functionality. html_entity_decode won't generally be able to convert these character entities, however mb_convert_encoding might be able to using the special HTML-ENTITIES "character set".

4) A terrible headache could be text fields written with the serialize() function in IPB, where I guess conversion will fail, since string lengths may change after converting to utf8. Is this true? If so, what is the best way to deal with this, or at least is there a way to automatically identify serialized fields and leave them out of the conversion (maybe only latin characters really exist in such fields)?

Many thanks in advance.

Indeed this is a big problem, and I'm afraid there's no easy solution. They are not solely latin characters (e.g. polls and poll data is serialized), so they cannot just be left as is, unless that's a compromise in creating the script from the start ("your polls will not convert") that the user is ok with.

The proper solution would be to select the serialized data, unserialize it, iterate recursively through the array, convert each value and store the converted value back into the array over itself, reserialized, then the db value updated.

We do not presently have any tools that allow you to convert your database character sets. I posted a sample of how such a library could be written for ipb3 earlier but there is no final script, no.

The charset conversion libraries from IPB2 are the same ones that will be used for IPB3.

With IPB3 we are going to provide them as a separate download - practically none of our users have a need for them, and they consume something like 10MB of space, so we'll just offer them as a separate download for those who do need them. In the mean time, you can just use the folder from IPB2.

I would rather add a copy topic feature. You would lose the ability to have replies to one of those topics show in the other one, of course, but you would be able to place a topic in more than one forum. Alternatively, you can move a topic from one forum to the other, which would show the redirect link (if you allowed).

Oh, and to further what Josh said, lofi links that are indexed in Google *will* be 301 redirected to their proper location in 3.x. You won't simply start getting 404 errors from them.

I'm just going to state for the record that on a personal level, I'd love to get more integration with third party software. That includes additional login methods and integration with larger software packages like Drupal and Wordpress.

I'm considering looking into them (independently/not here at work) later this coming year.

We support AddonChat and ParaChat.

We have a shoutbox as a community project as well.

Facebook earlier in the year just leased about 86,000 sq ft server space lol

Oh that's irrelevant. We should be able to add any feature someone dreams up and it should run on that Pentium 2 shared server user xyz is using that has 1 GB of RAM and 5 GB hard drive space, with 50 accounts on the box.

/end sarcasm

People don't consider these sorts of things you know. ;)

Time will come and you will include it anyway )

Perhaps. There are a lot of features I don't personally see the point of that I/we add. ;)

I was just saying it's not in 3.0.

See, there are times when AJAX can be very useful. It's an interesting, and often useful technology. But as with any technology, a lot of people go overboard and want this feature and that feature to use AJAX, just because it's "cool".

Submitting a form, in general, does not normally require AJAX. There are cases where it is useful and can be done, but replying to a topic, in my mind, is simply not one of them.

I'm sure there were will be addons for this, due to the demand/requests for such a feature. But at this time it is not a 3.0 feature.

What I've had to do is modify a kernel library (ips_kernel/class_file_management.php) and print out the ACTUAL response from feedburner, which includes the Location header.

I've "fixed" this for 3.0 by adding support for curl, and verified it follows the url properly.

Feedburner redirects the request for some reason behind the scenes. If you get the url it redirects to you can use that url.

Might take a look at having the RSS importer support CURL, which can handle redirects.

I'll say right now, while "proper database design theory" calls for no comma-separated strings, in reality it's not always easy or the most efficient way. While we recognize that some areas of the database design can *always* be improved, I'll state right now that it's not likely IPB3 will have no comma-separated strings in the database.

As you can just from the replies here, which don't even encompass all of the functionality necessary and desired for tagging, it's not as simple as whipping up a tagging system and throwing it on top of the forum engine. ;) Serious thought needs to be put into how this would work for the most people. Some people only want certain members to be able to use tags. Do view and submit permissions need to be split in that case, or should everyone be able to view? Of course you need a tag cloud as well, but it should be able to support multiple products to be efficient. There's just a ton of things that goes into this, we didn't want to rush a half-baked version for 3.0 just to "get it out there".

Go take a look at TAZ

(The Admin Zone)

Most of the groups don't give you access to any additional forums, they just show your interest in a specific topic (in their case different forum software) in your member profile (eg:

My profile there

and also as a tag under your username in posts.

This is all done by the user, save for a select few groups that need a group leader approval to join, and requires no interaction from the admin.

+1 from me as well.

That's social groups, which is different. The OP is specifically stating they aren't looking for this sort of thing.

Let me just state right now that while we have discussed internally group enhancements (along the lines of both joinable and social groups) it takes a lot of effort to "get it right". There have to be proper administrator controls, proper front-end moderation controls for group managers, ability to make new users group managers, and then there has to be functionality useful enough for this sort of thing.

This functionality will not be in 3.0, however it is something on drawing board that we come back to and are still discussing. That is to say, it may make it into a future release, it just won't be in the next release.

I don't think it will reduce serverload by any noticable factor. All you are really doing is loading half a page instead of an entire page when you do it via AJAX, but the main load-bearing processes (parsing the posts, for instance) still have to be done.

Further to that, if you did this via an AJAX request, not only do you have to run the pre_db_parse routines (parse before saving), you have to then select all new posts, and run pre_display_parse routines on them to send back via the AJAX (as opposed to running only pre_display_parse routines, not both).

Granted I don't think that would affect much, but at the end of the day you likely won't save any server resources with this feature whatsoever.

I've never seen any real benefit to an AJAX quick reply other than to sit and say "ohhh, look it has ajax".

Submitting a form to add new content should *generally* reload the page. Too much ajax breaks normal browser back/forward functionality.

IF (that's if, mind you) we were to do this, we would support the standardized ical format over CSV (but most calendars, including Outlooks, support this too).

I know exactly which query you are referring to (we see it pop up in slow query logs even on moderate-sized boards). I don't think your fix includes the overall exact functionality, but is something I'll look into. :)

In all honesty, a huge bulk of the messenger was already rewritten, so the query may not even be there anymore...

I agree with groups. Not sure about the others. Flashchat is great and I've asked for a built in IM system for a while, but it seemed like the thoughts were that it would be toor esource intensive and not enough payoff. I personally think it would be cool, but judging by how staff reacted... I don't think they will do it. Although they also didn't like the idea of a rep system, so who knows.

As far as the icons when people are on different IM services... some of that can be done now as bb codes, but would be nice to be built in. Also Michael said in the past there was some kind of IM functionality in IPB. I think buttons you clicked to IM people? But for some reason IPS stopped it?

The thing you are thinking of that used to be in IPB was links using custom protocol handlers. I think it was something like

aim://screename?msg=something

or something to that effect, but as protocols and accessibility changed it was removed. They could be skinned back in relatively easily if you looked up the right urls.

Building in an ajax-based communication system is indeed quite resource intensive. We've seen the results first hand, and I have a feeling if we were to build something like that in we'd be overwhelmed with people reporting their server caught on fire.

Uh, everyone....

Do you not realize we already offer support for 2 different third party chat clients (and I've seen freely available modifications that support FlashChat as well)?

Yes, both of those could. Essentially, any tables that won't get huge and do not hold critical data you can't afford to lose (since they are cleared when mysql restarts).