Jump to content

Invision Community Blog


Managing successful online communities

Mark
Sign in to follow this  
 

4.0 - Rethinking XML Handling

The IPS Social Suite stores skin and language (and some other stuff) in xml files which are imported into the database at installation and upgrade. The reason we do it this way is so of course, you can export skins and languages and install them on other sites or distribute them via the IPS Marketplace.

I'm not the biggest fan of PHP's XML handling at the best of times (it would seem whoever wrote the SimpleXML class and I would disagree on the definition of "simple") and had already changed some of the places we use XML to use JSON instead, but these skins and languages are particularly difficult because they can get huge. Handling these large files, especially when combined with sub-par servers can lead to memory exhaustion or maximum execution time timeouts. Our current method was taking both too much memory, and too much time in a single HTTP request.

So we had to come up with something better. At first we considered splitting the XML file into several, but that would mean either requiring people installing a new skin/language to upload multiple files (which we deemed not acceptable) or compressing them in some way (which is another of PHP's weaknesses, and would have just created another problem). We also researched different markup languages, however found similar problems with everything. We also had to make sure that any solution didn't step on the toes of any of our other 4.0 goals, in particular we want to completely eliminate writing cache files to disk so as to better support cluster environments.

The solution was two-fold. To resolve the memory problem, we decided to use the XMLReader and XMLWriter classes - these are non-cached, forward-only classes for reading and writing XML. Rather than store the document in memory, you can only read/write one node at a time, and can only move forward, not backwards. Even with this approach though, we needed to account for the fact that importing is an intensive operation anyway, and one that needs to be staggered. To accommodate this, we wrote a simple AJAX-based "redirector", which continuously fires HTTP requests at a PHP controller until the controller reports the import is finished (it displays a fancy progress bar to the user while this is happening).



We decided to make the AJAX redirector a helper class so that it can be used elsewhere and by third-party developers (rather than our current approach in some areas of making the user sit through loads of physical redirects). The code is really simple:

IPSOutput::i()->output = new IPSHelpersMultipleRedirect(
	/* Query string which will take us back to this controller */
	'app=foo&module=bar&controller=baz&do=process',
	/* Code to run for each HTTP request */
	function( $data )
	{
		// If we're not done yet...
		return array(
			$data,				// Data to get passed back to this function,
			"Processing...",	// Message to show (via AJAX) to the user
			50,					// [Optional] Number between 1 and 100 to indicate progess for progress bar
		);
		
		// If we're done
		return NULL;
	},
	/* Code to run when we're done */
	function()
	{
		// Code that runs when we're done
	}
);

Sign in to follow this  

Comments

Recommended Comments

Sounds great. Does this mean the skin import / export process now only requires one XML file?

I think it's best for it to remain as two files.  After all, what if you want to use the same image set for multiple skin sets or vice-versa?  Instead of being able to pick and choose smaller skin and image XML's to import, you have to pick and choose a single file among multiple large XML files.  Though if it were made to do it in a single file as an option (could still export into individual template, image, replacements XML files, and thus import separately as well), then I could see that being a benefit.  Simple for installing a paid for skin, while allowing the import of just the templates or images, to use stuff that already exists within the community.

 

I would definitely like to see the ability to overwrite an existing image folder or being able to remove an existing image folder, from within the ACP.  But that's not XML related.

Share this comment


Link to comment
Share on other sites

 

Awesome. What about attaching a setting group like when exporting a hook? (which would really make things seamless.) 

 

We'll have a blog entry explaining the details soon, but basically yes, you can do that too.

Share this comment


Link to comment
Share on other sites

I suppose it would be pushing our luck to want to be able to import an application via XML...  Practically eliminate the need for FTP and such, just import everything. :X  (Don't hurt me, you know it would be awesome.)

Share this comment


Link to comment
Share on other sites

I suppose it would be pushing our luck to want to be able to import an application via XML...  Practically eliminate the need for FTP and such, just import everything. :X  (Don't hurt me, you know it would be awesome.)

 

It would be pushing your luck to get that much out of me from this blog entry ;)

Share this comment


Link to comment
Share on other sites

Summary translation: Some servers time out or throw memory errors while importing language packs and skins in 3.x.  That will no longer happen in 4.x.

 

That's fighting talk. :p 

 

I think the support folk would agree that your sentence should end "That will no longer happen in 4.x (except when it does.)"  GoDaddy and XTREME MEGA HOSTING will make sure of it.

Share this comment


Link to comment
Share on other sites

 

That's fighting talk. :tongue:

 

I think the support folk would agree that your sentence should end "That will no longer happen in 4.x (except when it does.)"  GoDaddy and XTREME MEGA HOSTING will make sure of it.

 

Fortunately, we have a solution for that too :p

Share this comment


Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...