Private bucket URLs expire for S3 after 7 days

[KT Walrus]

I'm developing a new Storage Method (call it Amazon2) to replace the current Amazon method. I've got public buckets working (I think), but I want to support private buckets in Amazon2. I'm actually using Minio as my S3 backend. Minio supports most of the S3 API, but Minio does not support ACLs at the Object level. So, I need to support private buckets to keep some/all files from being publicly available with just a "public URL".

Private buckets require pre-signed URLs to download, but pre-signed URLs have an expiration time of up to 7 days and also have a QUERY_STRING (of course).

How do I add private bucket URL support into my Amazon2 storage method?

The current Amazon storage method does implement marking files as private (by setting the X-Amz-Acl to 'bucket-owner-read' on save() to keep some files in a public bucket private, but this won't work if I use Minio as my S3 backend. Permissions are only enforced at the bucket level and not on individual files in a bucket.

I think I could "fix up" $this->url before returning from $this->load(), but I'm worried that having a QUERY_STRING in an URL isn't going to work in all cases since the code expects the $this->url to end with the $this->container/$this->filename in many places. Also, you can't modify pre-signed URLs in any way or they won't work. You must use the pre-signed URL as it was when you signed it.

Or, maybe I could leave $this->url a "public URL" in all instances and "fix up" to change to a pre-signed URL with some hook into some Output class when the URL is used in html/curl/AJAX?

I'm kind of lost how to implement this properly so I can use Private Buckets in my storage method. Any help or advice will be appreciated.

[KT Walrus]

Just thought of one other possible approach to implementing private buckets... Instead of the URL being an S3 URL to download the file, implement an interface file to download the file from S3 to my local server for serving it to the user like is done in the Database storage method where the baseUrl() is 'applications/core/interface/file/index.php'.

I don't like this since it requires that the files in a bucket are downloaded to my local server and then downloaded over HTTPS to the page. I much prefer files to be directly downloaded from S3 like the Amazon storage method does.

Maybe I could have an 'applications/core/interface/file/index.php' proxy generate the pre-signed URL and do a "temporary redirect" to the pre-signed URL?

Not sure if this would work in all cases, but maybe it would? Still there would be 2 hops for each download of a file stored in a private bucket... so maybe not a good idea...

 

[bfarber]

Hmm, well firstly, I'm not sure I have all of the answers you're after and I think it's worth keeping in mind that just because you are modeling after the built in Amazon class does not necessarily mean you have to do everything exactly as we do. Where ever changes are needed, make those changes.

I would not worry about "2 hops" for downloading files, unless this also impacts things like emoticons and so forth that are displayed quite frequently and automatically. If you are talking about things like downloading files from Downloads or downloading attachments, these are relatively rare events and the first hop is just going to 302 redirect anyways (so there's not much processing to be concerned with).

Beyond that though I'm having trouble really wrapping my head around what you are after without diving in to the code and trying to simulate or fully understand the setup, so I don't have a lot of comments as to the "best" way to do it all. Part of that depends upon whether you will release the thing you are working on in the marketplace or not when you are done (I get far hackier if the thing I'm working on won't be released to the public at times), and part of that depends on upon what limitations are present, and which limitations you are happy to live with at the end of the day.

[KT Walrus]

@bfarber, I think I have decided what I'll do here until IPS updates the \IPS\File classes to support "temporary pre-signed URLs". I see you did use "temporary URLs" for the Downloads app, but  it is hardcoded to the \IPS\File\Amazon class only and doesn't really support the general use case.

Anyway, I am now pursuing configuring my Amazon2 storage method with whether the bucket is marked private or public. If the setting is private, the baseURL will use applications/myapp/interface/file/index.php. If the setting is public, the baseURL will use the S3 public URL.

Hopefully, applications/myapp/interface/file/index.php can download small files to my server for the HTTP request and use 302 redirect for large files using the pre-signed URLs for the request.

Even if the bucket is later changed to public from private in S3, applications/myapp/interface/file/index.php will still work to download or redirect (depending on size of the file). That is, it isn't dependent on whether the bucket is public or private when serving the files in the bucket.