S3 Compatible Downloads

[All Astronauts]

Non-Amazon S3 storage services do work with IPS 4.1x. Add your endpoint, key, secret, and sub-directory and custom url if needed, assign to some file category such as attachments or Gallery images and off you go!

However, these Non-Amazon S3 services are NOT officially supported by IPS - that's because some of the internal methods have not been altered to handle them. These methods are hard-coded for Amazon S3 exclusively.

If you are merely displaying images and the like on the front end, none of this is a problem. It becomes a problem if your end-users are going to download these files - that is when you hit the exclusively Amazon S3 code and the downloads will fail. For example, if you assign non-Amazon S3 storage to save IPS Downloads files, those files will not be downloadable. When the system attempts to make the download URL, it crafts that URL as if you were using Amazon S3, not some other S3-Compatible storage service.

The two major areas that this will impact are IPS Downloads files and Attachments (this is what the file storage is called in the Files list in the ACP - it encompasses all attachments such as those you would have anywhere the full editor is used and there is an attachment dropzone.)

This plugin intercepts the URL creation method called on Amazon S3 file storage objects, checks to see if it is an actual Amazon S3 request, and then just passes it on through as usual. If it is NOT an Amazon S3 request, it must be an S3 compatible method, in which case I provide a rudimentary URL to patch this up. It isn't anything fancy: just enough headers to push out the actual file name and trigger your browser to download it. If the attachment or file in question is an image your browser might just pop it up in a new tab, in which case just right click and download it there. This has been tested with the usual modern browsers (minus Safari), and various file-types. All testing was successful.

BTW, as an aside, external FTP storage for files is NOT COMPATIBLE with IPS Community in the Cloud. Might want to note that...

Free, support through the IPS Marketplace topic for now. Free also means lay off the instantaneous demands for support. I use this too so it's in my interest to keep this going. Support, and this plugin's existence, continues until IPS gets around to actually officially supporting Non-Amazon S3 storage.

WARNING!

The only item in the entire IPS Suite that marks files as private is IPS Downloads. This plugin will flag them as public. This means a few things:

1) If you are monetizing or otherwise require to fully control access to IPS Downloads files this is probably not the plugin for you. You will need to wait for complete 3rd Party S3 support, or at least specific mods to support obj.space, Digital Oceans Spaces, and so on. It really is just some hump work on getting the signatures correct (and it might be a V2 vs. V4 dance with varying providers as well - IPS just has a hard-coded-to-Amazon V4 method). Give it time, it will happen.

2) If you are using this plugin on an already existing bucket with Downloads files, those files will need to be flagged as public-read for them to download correctly. Again, this plugin is just getting rid of the signatures to allow public files to download correctly from 3rd-party S3 providers. 

3) IPS Downloads users on IPS Community in the Cloud may need to wait forever and ever before the plugin will work correctly (forever and ever being defined as more than handful of minutes) as CIC caching takes some time to get things working. You *will* need to be infinitely patient. Like, install and then go drink something for awhile, read a book, etc. Then test. Don't yell at me saying this is broken., It isn't. Tested and verified on both own-hosting and CIC.

Enjoy, good luck, etc...

 

 

Edited October 14, 2017 by All Astronauts

[W4L]

this is exactley what im looking for but it does not seem to work

i use s3for.me and its still messing up the urls 

 

*url its providing*
http://rest.s3for.me/xxxxx.com/monthly_2017_10/59ef1ceb256b2_S3CompatibleDownloads1.xml.6eb52243c7570e0ed901a2d7adb20c20.xml?X-Amz-Expires=1200&response-content-disposition=attachment;%20filename*=UTF-8%27%27S3%2520Compatible%2520Downloads%25201.xml

*url needed*

http://rest.s3for.me/xxxxx.com/monthly_2017_10/59ef1ceb256b2_S3CompatibleDownloads1.xml.6eb52243c7570e0ed901a2d7adb20c20.xml?AWSAccessKeyId=F38875120BB8864941CEC54647DC1A87&Expires=1508929708&Signature=X9jnTXp7Em0wShFnavr1IFJEjDQ%3D

not sure if it needs an update or if im in the wrong somehow .

thanks

[All Astronauts]

This only works if your file doesn't require a signature to grab it. It's just a hack too get around the hard coded Amazon v4 signature automatically used by IPS. If you can spin me up a bucket with this service and pm me that, key, and secret I can test here locally.

Also, what isn't downloading? Attachments? Downloads? Something else?

[W4L]

attachments,downloads and musicbox files 

none will download i moved to dreamhost until i can get it sorted.

thanks for the help

[All Astronauts]

I also just a moment ago set up an account there, and they give you $1 to start, so I can test both with my account and yours.

[All Astronauts]

Your service appears to *require* a signature for downloads, even if the file is public-read. If the attachment is an image it displays fine of course, and you can open the image in a new tab and download it directly like that, but if something is being downloaded via URL, it seems to want a signature no matter what.

I guess I can get about the business of writing out a real S3 compatibility mod but it will be a paid app.

For now I'll just keep a running list of compatible services on the Marketplace page and in this support topic.

Thanks!

[All Astronauts]

CURRENT KNOWN COMPATIBLE SERVICES

Delimiter obj.space
Digital Oceans Spaces

 

IN-COMPATIBLE SERVICES
If a service REQUIRES a signature for download URLs, even for public-read files, this plugin will not help you. The following services are known to require signatures for all download URLs.

S3for.me

[Joy Rex]

Would this work with Wasabi?

[All Astronauts]

Timing. Blame Joel who tagged me on Slack a few hours back.

Actually, this was working with Wasabi months ago, I just never updated the description. Took some time to get Joel squared away today and made a little tweak to the file. And updated the file description.

Caveats as stated in the file description still apply.

[beats23]

Any solution for downloads paid files to work?

[All Astronauts]

On 6/14/2019 at 4:52 AM, beats23 said:

Any solution for downloads paid files to work?

Yes, with caveats as of right now.

1) Will be a seperate paid mod.

2) I've a got line out to Wasabi support (I've been an account holder for nearly a year now) on this one singular hitch and I want to wait until I hear back from them first. I could launch now but it's juuuusssst a slightly messy small thing which may put off some people. Rather wait and see if I can get this cleared first.

Edited June 17, 2019 by All Astronauts

[beats23]

8 minutes ago, All Astronauts said:

Yes, with caveats as of right now.

1) Will be a seperate paid mod.

2) I've a got line out to Wasabi support (I've been an account holder for nearly a year now) on this one singular hitch and I want to wait until I hear back from them first. I could launch now but it's juuuusssst a slightly messy small thing which may put off some people. Rather wait and see if I can get this cleared first.

Great

[Steph40]

On 6/17/2019 at 4:46 PM, All Astronauts said:

Yes, with caveats as of right now.

1) Will be a seperate paid mod.

2) I've a got line out to Wasabi support (I've been an account holder for nearly a year now) on this one singular hitch and I want to wait until I hear back from them first. I could launch now but it's juuuusssst a slightly messy small thing which may put off some people. Rather wait and see if I can get this cleared first.

Any news on this? Did Wasabi get back to you?

[JustinHawk]

Does this works with OVH Object Storage ? https://www.ovhcloud.com/en/public-cloud/object-storage/

They are S3 compatible. I am planning to move back to IPS but i need to make somethings sure before i move. If someone can help with this that would be great.

[All Astronauts]

If it is S3 compatible it should be. The plugin is free so whip up a test instance somewhere and see what happens.

[JustinHawk]

8 hours ago, All Astronauts said:

If it is S3 compatible it should be. The plugin is free so whip up a test instance somewhere and see what happens.

Hey, i tried but it failed for some reasons. Like filename etc didn't worked with it addon installed.

Anyhow i spent some time to find out what was messing up thing and fixed the default amazon file for OVH. 

Default issue was that it was encoding headers twice and some other headers. Once IPS encoded filename in headers and other time http_query_builder did it again. Messing up canonical URl, i dont know how its working with amazon and much. So i just edited the headers and removed extra encode code. That fixed issue with OVH and file downloads with signature is working now without need of a plugin. i think you can also update your addon to following changes and it will improve compatibility.

[ahc]

@All Astronauts  I've upgraded to 4.5 and this plugin seems to be working without issue so far the past few days (at least I haven't noticed any errors or issues with our storage), but Invision throws an error about it being incompatible with 4.5 when you try to view it in the marketplace.

Any chance for a quick look and possible update?

[All Astronauts]

For it to show in the Marketplace, an app/plugin has to be re-uploaded and evaluated by IPS staff before they will allow it through and be visible there.

I can give it a once over and throw it into the Marketplace heap but if it's clear sailing so far you've got no worries I'd guess - this would just be for Marketplace visibility purposes. I'll try to clear this tonight.

I've only got one (two?) things into the new 4.5 Marketplace so far - there's a sliiiiight backlog (my problem actually).

I'll even give @TarunD 's thing a look - double encoding the url would explain a lot of errors away (there still may be Amz specific urls hidden within but thats no big deal to get around) If you have those changes and are willing to share feel free to throw them my way Tarun - save me another code crawl.

Edited August 13, 2020 by All Astronauts

[ahc]

21 hours ago, All Astronauts said:

For it to show in the Marketplace, an app/plugin has to be re-uploaded and evaluated by IPS staff before they will allow it through and be visible there.

I can give it a once over and throw it into the Marketplace heap but if it's clear sailing so far you've got no worries I'd guess - this would just be for Marketplace visibility purposes. I'll try to clear this tonight.

I may have spoken too soon?  I woke up and noticed we received a few errors in the system logs with info about our S3 storage when someone attempts to download a file they purchased in Downloads.  I'm not sure what information should be censored within the error log, so I'll send it to you via PM for a peek.  It may not be related and something I need to fix on my end, but I want to make sure.

 

[All Astronauts]

Just wanted to throw an update out to y'all about this plugin.

Is it 4.5 compatible?
Yes. Nothing has changed massively to break things - it still does the rudimentary things it does. It is still free.

Will I put this in the 4.5+ ACP Marketplace?
Maybe... 

Maybe...?
There are two hitches and they have nothing to do with this plugin. 

1) I've had a second person hit me up with certificate (SSL) problems and CDN stuff and Downloads (that's what the above post was about). Nothing has changed on the plugin side. Also, I've tested a bunch locally without a CDN and everything is fine and since all the errors people are throwing at me are cert problems and CDNs that's going to be me out sorry to say. Make sure you have the bucket name and S3 service url cleared in your certs and possibly your DNS and CDN settings. Don't know what to tell you other than that the web and security and browsers and standards of what will be allowed and not throw errors change.

BTW, if you do get a cert error or whatever downloading something from Downloads while using this plugin and you use Chrome - hit the advanced button on that error page - it will have a still want to go to this link link - click that and you'll probably get your file - again showing the hitch is not with the plugin but certs and dns and cdn permissions and etc....

2) Did you know that most Amazon S3 compatible services are, lol, NOT fully compatible? There is a reason why IPS has not added additional providers beyond Amazon. There is ALWAYS, (I'll say it a second time because it is hilariously, depressingly, true) ALWAYS something. For Wasabi, as an example, they do support chunking of file uploads, which you will notice IPS added to the S3 storage service with 4.5. You'll be pleased to know that Wasabi does it differently (chunks upload and then you 'compose' the final file). I refer you back to the first sentence of point #2.

Will I support Wasabi, or other service, chunking with this free, ancient, derpy thing?
No... It's a free, ancient, derpy thing that let me patch up Wasabi use on my end for my use case (files not that large, no CDN) that I figured others might find useful. I made one test for chunking locally with Wasabi and it errored out trying to save the first chunk or trying to deal with the second chunk (this using the internal IPS S3 methods of course) and that's all I tested, and will test for now. Beyond this I would write a formal S3 handler-per-service application but Marketplace release for these things is probably touch and go - it's a core, fundamental, aspect of the software (files) and letting 3rd party file-things loose on the community might be frowned on now - I do not know for certain (or at all) actually but there is a risk with endusers disabling or uninstalling things with files still existing in places, or the support tool letting you disable all 3rd party apps and plugins and then stuff gets wrecked and so on. The ability for a site to get itself wrecked probably trends up with these.

If it is an app there is a flag you can set on the app itself that on install will mark it as "forever on and uninstallable" that would mitigate some of this but does IPS want any part of that in the Marketplace? These things would get so much side-eye from the support team.

Did you know there once upon a time was a Azure file handler in the Marketplace? Long time ago... What happens when the 3rd party disappears? I suspect IPS might be content to just let things be and the more needful, larger, corporate, etc., sites can get what they need privately from 3rd parties and the support can stay way out of IPS's hands. And yes, there are 3rd party private S3 compatible service apps out there - shh, don't tell anyone - no I won't tell you who has them. It's different when you have a devops team and a plan and $$$ resources and did I mention having a plan and knowing what you are doing vs. one guy throwing this stuff into the marketplace and then someone building up a site gets this and yes, it's great you are building your awesome site up and want to save money but I'm sorry you disabled this on upgrading and Gallery is trying to make new copies of files on upgrade and can't read the files because you disabled the file handler and now all the urls to your files are NULL and....

So can I submit this to the 4.5 ACP marketplace - sure. Do I want to.... eh... I'm more inclined to attach it here and be done with it. 

If you want to support large file uploads and still use this, you will need to, as you've had to all these years, PHP tweaks to upload_max_filesize, post_max_size, memory limit, execution time...

[All Astronauts]

I'm going to amend the above by saying I'll post an updated file here. There was a single change in the way IPS does a url bit with 4.5 that does make a difference and can/will mess things up slightly with CDNs and SSL Certs and so on. Unless you diffed the files you'd probably miss the change.

[All Astronauts]

You know that thing where you casually mention something that's relevant to the problem but you are totally oblivious to the fact that you actually didn't account for that thing you just talked about? And then someone comes along and says "why didn't you deal with that thing you just talked about here..."?

That was yesterday.

@Martin A., with the oblivious assist from myself, got a handle on this. This has been cleared by at least two live sites experiencing this problem. A new version will be submitted to the Marketplace asap.

Now, just as a heads up to everyone: we are nudging, but not rushing, towards the end of the usefulness of this derpy thing. 

Amazon S3 is forcing everyone on over to the new styled urls where the bucket name is IN FRONT of the endpoint url. So, my.bucket.s3.awesomes3service.com/.... (this is virtual hosted-style) vs. ye'olde path-style urls, e.g. awesomes3service.com/my.bucket/.... 

The thing with the newer style is you gotta have wildcard support set up for your domain with your DNS settings. You gotta deal with your CDN settings. And naturally, your SSL certificate. Got dots in your bucket names? Shouldn't do that... Dashes are better but now you have some other hitches and so on. In a perfect world you would just have a single non-dotted, non-dashed alphabetic "word" for your bucket names and things work much more smoothly.

The patch here, as it were, is to account for dots in a bucket name and if present, flip things over to http instead of the likely secure SSL https. That stops all the yelling.

The point is, as Amazon S3 standards change, and web/browser security changes, and CDN requirements and changes, and so on, we're going to hit a point where this isn't going to be a viable hack anymore.

We are not there now but in a few years? Who can tell...

[All Astronauts]

Naturally this is taking time, nothing for it, and no blame or anything, they have to spin up alt-S3 service and test and so forth. 

If you gotta have it due to DL problems shoot a PM.

EDIT: If you saw another post by me after this one, apparently, you, and I, didn't... 

Edited September 25, 2020 by All Astronauts

[All Astronauts]

I honestly did not think this was getting approved (at all). But there it is.

The statement provided before still stands. Enjoy this while it remains valid. Not "approved-Marketplace" valid, more like "functional-valid".

And also remember the use-case this was generated for. To allow for users to download attachments (and Downloads for non-monetized Downloads instances) without the need for signatures. If your needs are greater, features and security-wise, look elsewhere... 

[All Astronauts]

Okie dokie. 4.1 submitted. Catch it when it shows for real in the MP (something something approval time)

What this one does is guarantees that if you have set a custom url for your bucket then that gets used. Period.

It also adds settings that will allow you to FORCE HTTP either on all your compatible S3 links or you can only force them when the bucket has dots in its name and there is no custom url set.

This plugin is still kicking out PATH-STYLE urls, so that's url/bucketname/filepath/filename. Eventually those settings might matter but right now they are mainly a hedge. If you have problems, give them a try, but they will mostly matter when this plugin supports VIRTUAL-HOST STYLE urls, the ones with bucketname/domainname/filepath/filename - also known as the style that will cause you endless grief with bucket names that have dots in them. You either write out custom SSL certificate logic, serve HTTP unsecured links, or do what IPS does which is let it be HTTPS secure but nuke the security checks themselves. That's a little upstream for this plugin (at least on first look) and since this plugin is still making PATH-STYE urls, nothing to worry about just yet. Still, you may have a use-case for these settings now, so, in they go.

Amazon is now forcing all newly created buckets over to VIRTUAL-HOSTING STYLED linking so it won't be too surprising if others follow suit. 

Note that using HTTP links work (of course), but on download, where the downloaded item appears on the bottom of your browser window (Chrome, others) you will get a little warning about this being an unsecure download and do you want to keep this file. Well. That's the deal. If your users can handle that, all is well. If they complain? :shrug: