S3 Compatible Downloads

[Circo]

Does this work for Backblaze?

[HDiddy]

On 5/25/2021 at 5:17 AM, Circo said:

Does this work for Backblaze?

I don't think Backblaze is supported...fully at least as it seems to need a secured connection to do Content Dispostion which seems to be the issue for most of these.  For me I have been looking at multiple solutions as My Downloads egress on Wasabi was way too high like 2.5TB compared to the 150GB I have stored.  The CDN Option with cloudflare simply didn't do much at 25% cached on average and in addition to that the whole thing went up in smoke when I got a 522 I could not recover from. I tried to see if Wasabi will match some additional bandwidth cost like their competitors.  No go.  They gave me till Monday to get my bandwidth down. 3 Days.

Fast forward I decided I didn't mind paying a little extra (0.01/GiB) for additional bandwidth but I need to find a provider that works.

First I tried Digital Ocean.  Nice provider but it seems they have switched over to the dreaded must have a signed account for public read.  So this one was a no go, but I might keep it as a storage backup

Next I tried Backblaze.  And like Digital Ocean same thing, trying to download with the headers gives you an error basically saying you are not allowed to do this anonymously.  It works great for pictures if that is what you want to use it for but not so much for downloads.

Next up Linode. Nice little service but I could not get the thing to work with S3Browser, which for me was just a sign this not going to be good.  S3 Browser allows you to do things like update bucket policy and other things many providers do not have a front end for so if it didn't work, it was a no go.

Finally I tried Vultr.  They are a new Object Storage provider so I figured..."Hey Why not".  So I went off transferring my June folder over to do initial testing.  I test by basically grabbing the link that IPS uses in downloads and seeing if it will work or error out.  To my delight and surprise, It WORKED!!  This tells me vultr is pretty much setup exactly like Wasabi, and with S3 Browser any bucket policies I had I can add them directly through an API call.  Also in addition Vultr is fast....like stupid fast compared to the rest.  Backblaze I found to be the slowest of all.

So for pricing Vultr is $5/Month giving you 250GB  of storage (additional 0.02/GiB) and 1TB of Bandwidth (Additional being 0.01/GiB).  For me this turned out to be roughly $25 Per Month for the month Bandwidth I am using.  That price is not bad for the peace of mind.

Anyway wanted to put this up here as I have not really seen much convo on S3 Compatible Storage that actually works outside of Wasabi.

[All Astronauts]

Its a special hell when services say they are S3 compatible and they are...mostly S3 compatible. Its the mostly that ends up being a hitch "somewhere".

Reminder to all about this derpy thing. It does NOTHING at all, anywhere, other than when a user tries to download an attachment - then it wipes the signature in the temporary download url the user hits so they can get it when your non-Amazon S3 service chokes on the signature IPS generates (as IPS is locked in to Amazon exclusively).

That's it.

If you have problems using alt-S3 services (using the service at all, moving files, etc.), it's not on this plugin. 

When you try a not-Amazon S3 service, run through the basics. Can you upload a file via the service through Invision Community? If uploads work, can you delete? If so, can you move the files this S3 service and local storage (File System)? If so, you are in probably in business. Then, for completeness, if your users might be able to download the files that are stored on this service (Downloads, Attachments, etc.) give that a try. If it fails, throw this plugin in and see if clearing the signature lets them through (something something public buckets etc.). If it doesn't work you are sol or you'll need to find someone to write a full new filestorage method. FYI, if you go this route, there be dragons. You should have a firm control of your dev ops at a minimum...

If IPS support gives you grief if you submit a ticket regarding S3 problems because you have this plugin installed, then just uninstall this plugin and try them again (as stated above this plugin ain't doing much...). That said, I would not expect much from that request anyways as I'm pretty sure they are not going to help you with the S3 problems you have with your not-Amazon S3 service. IPS officially supports Amazon S3. Anything else is going to be outside IPS support scope.

[All Astronauts]

On 6/12/2021 at 8:58 PM, HDiddy said:

Content disposition which seems to be the issue for most of these

Yes...

On 6/12/2021 at 8:58 PM, HDiddy said:

basically saying you are not allowed to do this anonymously

And that's the thing that this plugin effectively "does", which won't work with, as you found out, a number of providers.

Someone up-thread (or privately) mentioned that when a user tries to dl a file and hits the temporaryDownloadUrl method there is "something" and the signature is doubled or generated twice or ??? (it's been awhile...) and that might be part of the problem with not-Amazon services as well. 

I just needed Wasabi to fully work, ran into the attachments dl problem, and threw this down to route around the problem. Everything else?

[HDiddy]

On 7/1/2021 at 8:38 PM, All Astronauts said:

Yes...

And that's the thing that this plugin effectively "does", which won't work with, as you found out, a number of providers.

Someone up-thread (or privately) mentioned that when a user tries to dl a file and hits the temporaryDownloadUrl method there is "something" and the signature is doubled or generated twice or ??? (it's been awhile...) and that might be part of the problem with not-Amazon services as well. 

I just needed Wasabi to fully work, ran into the attachments dl problem, and threw this down to route around the problem. Everything else?

Thanks...

I have been to hell and back with Wasabi and ultimately decided I cannot deal with heir draconian egress policy.  The fact that they will not at least let you pay for more egress per GB is one thing, but to only allow egress up to the amount you have uploaded is just insanity.

Vultr seems to be working well for me. It is fast, I can setup multiple different object storage instances (paid separately) and the fact I can simply pay for more egress use is perfect.  I also find it has the most compatibility with Amazon S3 commands like Wasabi, and unlike Backblaze. Vultr Object Storage is still new, so hopefully they don't go the route of some of the others, but an option if people have need one.

[All Astronauts]

Wasabi is fantastic for what they are - a storage service - but that's really all they are and that is specifically their niche.

For those not in the know, for Wasabi your monthly egress should match more or less what you have stored. Store 100gb? Monthly out should be 100gb, though I doubt they will care much if it is 150 or a touch more. If you are pushing that egress out DAILY you are going to get cancelled.

There are two ways around this:

1) Store dummy data. People create a separate bucket, and just manually through an S3 tool dump whatever in there. 100gb real files, 900gb dummy files, = 1tb monthly egress. You get the idea. Need more egress? Add another TB to your storage bill (that would be 2TB for 12/month) and fill that up and now your 100gb of real files have 2tb of monthly egress.

2) The common sense solution? If you are egressing at a massive level daily I have no idea why you do not have a CDN in front of your server. Cloudflare, BunnyCDN, others... This is exactly what they are there for.

You end up with a build out as follows:

Server: For IPS install, serving the site, some files.

Wasabi/S3: Off-load the STORAGE of tons of user generated file content, or Downloads files, etc. This saves you paying for large amounts of storage on your webserver where storage costs are typically not viable (only so much room on the server itself). Not to mention you can blow your egress bandwidth out on the web server.

CDN: Takes the massive load off your server and off-site storage by caching and serving all those files.

Small hobby sites will never have to worry about most of this. You can get by with just a web server, or a web server and S3-type storage somewhere. If you actually are moving gigabytes to terabytes of data DAILY between your web server and off-site storage you either have the deepest of pockets or you "know a guy" (I know guy who back in the day ran a few Counter Strike servers out of his nationally-known airline regional office for awhile. Think they were in a closet. And then an off-site higher-up opened the closet and then he was out of a job. lol) - or, you've set up your site with a CDN. 

FYI all, this has been known for awhile but Backblaze is one of those providers that really, really, wants everything signed. I can hack this plugin to be compatible-ish with it but it ends up with attachments and Downloads files to be downloaded as filename.ext.abcd1234efgh5678 as unsigned requests may not contain content-disposition or content-type which makes your browser pull the file "as-is" and that means the IPS appended (and usually necessary) alphanumeric comes along for the ride. Not an ideal situation for your end-users.

[All Astronauts]

Well, see, the thing is, you step away from something long enough, and come back to it with fresh eyes and forgotten knowledge and stuff jumps out at ya.

Cryptic? I think I solved Wasabi finally. DigitalOcean Spaces probably as well. Both with the signed temporary urls for attachments and Downloads.

I might charge a touch as I'm probably throwing this into an app and adding in ACP widgets for the service status stuff.

Gonna simultaneously poke things and mull things over, if it is possible to do both at once.

It was such a benign, little, utterly ignorable thing too...

[imJexs]

On 7/6/2021 at 8:25 PM, All Astronauts said:

Well, see, the thing is, you step away from something long enough, and come back to it with fresh eyes and forgotten knowledge and stuff jumps out at ya.

Cryptic? I think I solved Wasabi finally. DigitalOcean Spaces probably as well. Both with the signed temporary urls for attachments and Downloads.

I might charge a touch as I'm probably throwing this into an app and adding in ACP widgets for the service status stuff.

Gonna simultaneously poke things and mull things over, if it is possible to do both at once.

It was such a benign, little, utterly ignorable thing too...

Hey, was wondering if you had any updates on getting private files working with DO Spaces? I'm in the middle of moving from S3 to DO Spaces and would love to be able to move our Downloads files (which must stay private) to Spaces as well.

From my very brief investigation it seems like DO is mostly compatible with S3. They accept v4 signature, pre-signed URLs, etc. Seems like the only issue is IPS uses "bucket-owner-read" for "X-Amz-Acl" header whereas DO only accepts "private". I'm curious if it could be as simple as intercepting those requests and swapping the header value out if its a request to DO. I'll probably do some more testing later this week but was curious if you had any more insight.

[Circo]

I really hope that this hasn't been discontinued. This has been working great on my site since getting everything setup correctly.

I had something come up recently... Would this plugin possibly cause duplicates to show in the "Who downloaded this?" I'm seeing where it shows members duplicated on that list. Some members are showing on that list 4-8 times. I've been asked by support to disable everything in regards to downloads, but I can't disable this mod since all my downloads are through my Wasabi.

[All Astronauts]

It literally does nothing other than bypass a signature check when downloading files that are attachments or IPS Downloads files. That's it. So it's nothing to do with that.

Plugin disabled (not that it would matter...), testing locally with Downloads, files stored locally, this behavior you describe appears intended. I downloaded the file, then downloaded it again. And then again later. Three entries as I would expect actually - if someone is downloading the file repetitively why wouldn't that be logged? Person could just be annoying, or maybe they after downloading it were addle-brained and deleted the file accidentally and needed another copy?

[Circo]

42 minutes ago, All Astronauts said:

It literally does nothing other than bypass a signature check when downloading files that are attachments or IPS Downloads files. That's it. So it's nothing to do with that.

Plugin disabled (not that it would matter...), testing locally with Downloads, files stored locally, this behavior you describe appears intended. I downloaded the file, then downloaded it again. And then again later. Three entries as I would expect actually - if someone is downloading the file repetitively why wouldn't that be logged? Person could just be annoying, or maybe they after downloading it were addle-brained and deleted the file accidentally and needed another copy?

Ok, thanks for verifying. I didn't think it would have anything to do with this plugin, but just wanted to check.  I don't think users are doing it on purpose. It's happening with almost every user. I think there is something else causing it to log a duplicate, but I can't figure it out.

42 minutes ago, All Astronauts said:

It literally does nothing other than bypass a signature check when downloading files that are attachments or IPS Downloads files. That's it. So it's nothing to do with that.

Plugin disabled (not that it would matter...), testing locally with Downloads, files stored locally, this behavior you describe appears intended. I downloaded the file, then downloaded it again. And then again later. Three entries as I would expect actually - if someone is downloading the file repetitively why wouldn't that be logged? Person could just be annoying, or maybe they after downloading it were addle-brained and deleted the file accidentally and needed another copy?

Ok, thanks for verifying. I didn't think it would have anything to do with this plugin, but just wanted to check.  I don't think users are doing it on purpose. It's happening with almost every user. I think there is something else causing it to log a duplicate, but I can't figure it out.

[All Astronauts]

Went ahead and set a new Wasabi file storage type. Set Downloads files to that. Enabled the S3 plugin. Uploaded a new file (zip), downloaded the file - successfully. Checked log. One entry. Downloaded again. Two entries.

Be sure to check with your users to see if there is just weird behavior going on.

You can also just test this yourself. Make a new Downloads file entry and then download it. Wait a minute. Download it again. If you see just your two downloads logged (minus anyone who might just jump in there in the moments after you do all this) everything is fine and your users are just weird or something. If you see yourself logged more than those two times, then something something something ???

[catbreadbat]

On 7/6/2021 at 8:25 PM, All Astronauts said:

I might charge a touch as I'm probably throwing this into an app and adding in ACP widgets for the service status stuff.

 

Was this ever released?

[HDiddy]

So something interesting happened to me today.  Suddenly I was receiving InvalidRequest errors for downloads on the site.  Not sure what triggered it but whe contacting Vultr support they were no help.  I decided to troubleshoot so I disable the S3 Compatible Downloads Applicaion and Cleared the Cache...and welp, everything went back to working again.  Not sure why all this happened but some food for thought if anyone is having issues.

[daffyy]

Is it possible to add temporary urls for the backblaze?

[Clover13]

@All Astronauts are you still supporting this or have plans to add 4.6+ support?

[rnorth6920]

Any chances of 4.7 support?  It works fine on 4.6 but with the php 8 requirements upcoming, didn't know if it would work out of the box with 4.7.

[All Astronauts]

There will be something, this is still hacky btw. Also what I release will prob be a paid thing (albeit cheap) that includes ACP widgets monitoring some of the external service statuses like so:

[rnorth6920]

I, along with others would gladly pay for your hard work. 

[SJ77]

Will this be updated for ips 4.7?

[All Astronauts]

@SJ77 This plugin will NOT be updated. It's old hacky whatevers. I mean, it worked - right? But time marches on. People are still using it just fine though so if you have it carry on and so forth.

However...

New thing submitted tonight if I feel it. IPS approval time runs about a week if there are not any problems (and Daniel and/or Stuart are not crushed with other work). Then you can give me some money.

Backblaze, Wasabi, DigitalOcean, iDrive. Vultr too (if you count tests ran six months ago - I don't wanna spin up another $5 to check again). All tested out. 

Things are still derpy but more professionally so. Everything still piggybacks on the internal IPS S3 code. Oh, and Backblaze is solid (I bypass the S3 stuff and use the BB API directly for attachment downloads). And since those guys finally pushed out a status page (albeit a not-too-great one) you get this too:

Obviously a new support topic and so on but that will come post-release.

[All Astronauts]

In case anyone was wondering: