Jump to content
Mark
 Share


4.0 - File Storage

Introduction

The IPS Social Suite needs to store lots of different files - there's attachments and profile photos uploaded by members, CSS and JavaScript files, emoticons, etc.

In IP.Board 3.x, various images got stored in different places:

  • Files uploaded by users get put in the /uploads directory. If you have a complicated setup, it's difficult to handle these. If you have a load-balanced cluster you need to set up an environment whereby all files are stored on a single server, or all uploaded files are synched between servers, but serving these files over a high-performance CDN can be difficult.
  • CSS, JavaScript files, images and emoticons get put in /style_* directories. If you want to serve these over a CDN, you can do so, but you need to copy the files over yourself.
  • Other pieces of data are written to disk as a caching mechanism. This has the same issue with load-balanced environments as file uploads.
  • Some applications had other methods - for example, IP.Downloads allows you to store files on a remote server using FTP.


In 4.0, we wanted to pull this all together and build a much better system for storing files and build the whole system with high-performance environments in mind.


File Storage

In 4.0, you have several different ways to store files:
  • On a local server
  • On a remote server using FTP (which you can use to upload files to many CDN services)
  • As binary data in the database
  • On Amazon S3

You can set up different configurations and choose which configuration to use for different types of files. For example, if you want to store user's profile photos on Amazon S3, but you want attachments to be on the local server, or even a different Amazon S3 bucket - 4.0 can handle that. And if at any point you change your mind about which storage method you want to use, the system will automatically handle moving all the files for you.

Everywhere that writes a file will use this central system - so IP.Downloads and IP.Gallery are included too.




Caching

There are lots of places throughout the suite where the same stuff needs to be retrieved or calculated over and over - for example, certain configuration settings, language data, information about the installed applications, etc. If this data can be cached, not only does it alleviate database load, it means the PHP code doesn't need to re-process the data.

In IP.Board 3.x, some of this was stored in a particular database table and could be cached using a proper caching system - but it was difficult to configure, and not everything used it - compiled HTML templates, language strings and more were saved as files in the /cache directory, which causes difficulties for load-balanced cluster environments.

In 4.0, we've overhauled all of this. For things that need cold storage (like compiled HTML templates) - you can choose either the file system or the database for storage. The data can then cached, along with anything else which might benefit from caching (like settings, application data, etc.) using one of 5 supported caching methods:
  • APC
  • eAccelerator
  • Memcached
  • Wincache
  • XCache

 Share

Comments

Recommended Comments



So now which cache should we use? Which one is better?

 

The simple answer is that Memcached is best.

 

The longer answer would depend on the specifics of your site and the hardware available to you.

Link to comment
Share on other sites

Will the cache support tags/events?

 

 

Is there only 1 Cache instance? (e.g. it wouldn't be the best choise to put EVERYTHING into memcached,right?;)

ATM we're using 2 different cache mechanisms for our page => memcached and filecache(filecache for big, not often changing data, which we need to fetch daily from other pages

Will this be easy to implement?

Link to comment
Share on other sites

 

The simple answer is that Memcached is best.

 

The longer answer would depend on the specifics of your site and the hardware available to you.

 

If you have multiple server , i can agree that Memcached is the best but if you are on single server on most of our benchmark tests Memcached is outperformed by APC & xCache 

 

couple questions :

Will we be able to define different cache for apps , like store core on Memcached , store board cache on APC

 

And will there be an option to define custom cache prefix ?

Link to comment
Share on other sites

  • Management

@GreenLinks: You can determine different storage locations for each application (or more specifically for each item group that needs storing) but you cannot specify the caching engine that sits on that currently.

 

@OxyFuse: Yes, you can. You'll just set up a new File System storage method for each item.

Link to comment
Share on other sites

Will the cache support tags/events?

 

 

Is there only 1 Cache instance? (e.g. it wouldn't be the best choise to put EVERYTHING into memcached,right? ;)

ATM we're using 2 different cache mechanisms for our page => memcached and filecache(filecache for big, not often changing data, which we need to fetch daily from other pages

Will this be easy to implement?

 

As the blog entry explains - there are 2 instances. One has the option of filesystem or database (for the large, infrequently changing stuff) and the other has the option of memcached or shared memory caches.

 

 

 

If you have multiple server , i can agree that Memcached is the best but if you are on single server on most of our benchmark tests Memcached is outperformed by APC & xCache 

 

couple questions :

Will we be able to define different cache for apps , like store core on Memcached , store board cache on APC

 

And will there be an option to define custom cache prefix ?

 

So we agree then ;)

 

All apps will use the same cache system. The keys sent to the cache system are hashed with data unique to an install to avoid conflicts, so a prefix is unnecessary.

 

 

Can we define different location for each file storage?

 

For example (on a local server ):

Attachments: /uploads/attachments

Emoticons: /uploads/emoticons

Gallery: uploads/gallery

 

Yep.

Link to comment
Share on other sites

will this help with CDN and running HTTPS? i thought there was an issue currently in that configuration.

 

once a setting is established could there be any guestimated analytical comparisons such as "you could improve performance by switching to X"  or if you have it set one way then switch compare the different real word results as to which setup is faster?

Link to comment
Share on other sites

@GreenLinks: You can determine different storage locations for each application (or more specifically for each item group that needs storing) but you cannot specify the caching engine that sits on that currently.

 

 

What about Gallery ?

 

Wıll we be able to define different storage locations for gallery like images and video's on separate location ?

This is kind of very critical because on CDN configuration we need to define different zones for Video storage.

Link to comment
Share on other sites

  • Management

Interesting feature! Are there plans to support other storage services than Amazon S3?

We will start with Amazon S3 as that is, by far, the most requested service. We will see from there.

Link to comment
Share on other sites

My question is, with all these features IPB4, I'll be able to run it on a shared host (just as my forum is still starting)?

 

Of course.

 

 

What about Gallery ?

 

Wıll we be able to define different storage locations for gallery like images and video's on separate location ?

This is kind of very critical because on CDN configuration we need to define different zones for Video storage.

 

Yeah - so Gallery stores files (rather than cache data) so you can set that to use it's own storage engine separate from say, attachments uploaded to posts.

 

We haven't got that far into IP.Gallery to be able to confirm if images and videos will have separately configurable storage locations, but yes, that's a good idea and the system certainly provides the possibility for us to do that.

Link to comment
Share on other sites

Would be great in the caching area to see what is recognized by the script as installed. vbseo does that and its an extra step to show you that its there available and working.

 

EDIT: Just realized the check is there...lol. Nice job!

Link to comment
Share on other sites




Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...