IPB4 RC6 - Loss of CDN feature from IPB3 and new Amazon S3 storage missing CDN option

[AutoItScript]

tl;dr

- The simple Origin Pull / image/JS/URLs option from IPB3 has been lost and it's very "hacky" to emulate in IPB4
- The new Amazon S3 storage type is only being used for simple storage rather than CDN - it needs a URL option to work as 
CDN. 

Apologies if I've misunderstood the new IPB4 options, I've only just installed a test board for it and am in the middle of 
culture shock    The feature request tracker seems to have gone too so this seems the best place to post.

Origin Pull Feature Loss

My usage of IPB3 and a CDN is as follows:

- I want to serve my site from my own server, with the exception of images (themes, emoticons, photos) which I serve from 
Amazon Cloudfront CDN. This results in very fast page loads at a tiny cost ($15 a month on average for a site with 1+ 
million posts)
- Cloudfront is simply set up in Origin Pull mode (similar to many other CDNs). This means when you use a url like 
xxx.cloudfront.net/forum/uploads/image.jpg the CDN makes a request to my site if it doesn't already have a cache of the 
file. The resulting cached file can be distributed from the CDN edge locations. I believe CloudFlare/MaxCDN/Others can also 
be setup in the same way. They are essentially a dumb mirror. 
- In IPB3 this setup was super easy to configure - you just change use the "images" URL. Done.

In IPB4 it looks like the simple URL option is gone. You need to configure storage types to move content around. There is no 
simple "Origin Pull" option where you can enter a different URL. I imagine this would be easy to add because the file 
content stays in place - all we need is a storage type that rewrites the URL.

You can emulate the old IPB3 config by doing this hacky method. But I worry about abusing features like this because it 
tends to come back to bite during upgrades:

Let say my website file system uploads folder and URL are:
/var/www/mysite/forum/uploads
http://www.mysite.com/forum/uploads

You can create a new File System storage option which still points to the local IPB installation (a different path) but then use the CDN URL:
/var/www/mysite/forum/uploads-cdn-originpull
http://xxxxx.cloudfront.net/forum/uploads-cdn-originpull

This storage setting can then be assigned to themes/photos/etc. In the background these files are then moved from /uploads to /uploads-cdn-originpull and the CDN url is used when they are displayed. This seems to work but of course there are now files being moved between folders which is not really required and adds a bit of risk.

Amazon S3 Issues

The IPS blurb in the feature map says:
 

IPS4 has the built in ability to store files uploaded to your Suite in alternative locations. You can still store them on your local file system but you can also store them on remote FTP servers or even on Amazon Web Services S3 system. The AWS S3 is a remote file storage and delivery system that allows for very cheap storage along with an included CDN capability. A CDN is a content delivery network that allows your files to "live" all around the world for faster delivery.

 

This implies that the current IPB4 usage of Amazon S3 can be configured as a CDN. The current configuration of IPS4 does not actually allow you do use the CDN feature because it hardcodes the xxxx.s3.amazonaws.com URLs for your bucket. This does not use Amazon CDN and geographic edge servers. Files are served directly from your bucket in whatever region you setup the bucket. For example, my bucket was set up in Ireland. In the current IPS4 configuration a user in Hong Kong would still request the file from servers in Ireland. So all you gain in this configuration is not storing the file on your server and a tiny speed boost from storing the images on a different URL to your main site.

In the Amazon world, to get the CDN functionality you setup a CloudFront distribution that then targets your bucket in S3 as a Origin Pull. You then use this CloudFront URL to access your content rather than using the s3.amazonaws.com URL. The ability to specify this CDN URL is missing from from the IPB4 config.

The simple fix is to add the option of specifying an optional URL in the Amazon S3 config (like the other storage methods). Then you will have the ability to use S3 as simple storage or as a CDN as you see fit. This doesn't rely on CloudFront either you can use other CDN providers to make use of the Amazon S3 files. As long as IPB can accept a custom URL it doesn't need to know anything about the CDN inner workings (for Origin Pull).

[sobrenome]

No feedback from the IPS team? 

[Stuart Silvester]

No feedback from the IPS team? 

​It's on here: http://community.invisionpower.com/4featureplan/

[Charles]

We will be adding S3 CDN support: http://community.invisionpower.com/4featureplan/

You can still do CDN for files on local file storage. It works just like it did in 3.x. Just change the URL in config:

So instead of //community.invsionpower.com/uploads you would put in //cdn.com/uploads

[AutoItScript]

It's not quite the same as 3.x and a lot more risky - it's physically moving files around and triggering all sorts of post content rebuilds. I just moved emoticons earlier to test on 4.0 and now I'm 3 hours into a post rebuild cycle for 1 million posts, and the moved emoticons have had their filenames mangled (bug report posted) so I'll likely restore a backup (test board thankfully).

I guess that's a consequence the new v4 design having of URLs hardcoded into post content rather than parsed on display. Any change to the URL means all posts must be rebuilt. A design decision, so there's probably nothing that can be done now.    

[Charles]

​If all you do is change the URL it doesn't physically move files around. It does to a post rebuild as it must change the locations of images inside existing posts.

[kysil]

Charles, Please note in the first upgrade, which should immediately include support for secure connection content delivery (HTTPS) with Amazon S3. Thank you ~

[AutoItScript]

​If all you do is change the URL it doesn't physically move files around. It does to a post rebuild as it must change the locations of images inside existing posts.

​How do you safely change just the URL?.  My emoticons were using a unique storage type and URL. If you go to edit the URL you get a warning:

This storage setting is in use. Any edits made to this storage setting will be saved as a new storage setting and a move files process will be initiated. 

Then when I clicked save the storage setting was duplicated and the new one was assigned to emoticons (this was difficult to see as both storage settings have the same file path - and that is the only part shown in the interface). The background tasks listed moving and deleting as occurring - although the files were still in place at the end.

Am I misunderstanding your process?