Jump to content

S3 Compatible Downloads


All Astronauts
 Share

Recommended Posts

  • 3 weeks later...
On 5/25/2021 at 5:17 AM, Circo said:

Does this work for Backblaze?

I don't think Backblaze is supported...fully at least as it seems to need a secured connection to do Content Dispostion which seems to be the issue for most of these.  For me I have been looking at multiple solutions as My Downloads egress on Wasabi was way too high like 2.5TB compared to the 150GB I have stored.  The CDN Option with cloudflare simply didn't do much at 25% cached on average and in addition to that the whole thing went up in smoke when I got a 522 I could not recover from. I tried to see if Wasabi will match some additional bandwidth cost like their competitors.  No go.  They gave me till Monday to get my bandwidth down. 3 Days.

Fast forward I decided I didn't mind paying a little extra (0.01/GiB) for additional bandwidth but I need to find a provider that works.

First I tried Digital Ocean.  Nice provider but it seems they have switched over to the dreaded must have a signed account for public read.  So this one was a no go, but I might keep it as a storage backup

Next I tried Backblaze.  And like Digital Ocean same thing, trying to download with the headers gives you an error basically saying you are not allowed to do this anonymously.  It works great for pictures if that is what you want to use it for but not so much for downloads.

Next up Linode. Nice little service but I could not get the thing to work with S3Browser, which for me was just a sign this not going to be good.  S3 Browser allows you to do things like update bucket policy and other things many providers do not have a front end for so if it didn't work, it was a no go.

Finally I tried Vultr.  They are a new Object Storage provider so I figured..."Hey Why not".  So I went off transferring my June folder over to do initial testing.  I test by basically grabbing the link that IPS uses in downloads and seeing if it will work or error out.  To my delight and surprise, It WORKED!!  This tells me vultr is pretty much setup exactly like Wasabi, and with S3 Browser any bucket policies I had I can add them directly through an API call.  Also in addition Vultr is fast....like stupid fast compared to the rest.  Backblaze I found to be the slowest of all.

So for pricing Vultr is $5/Month giving you 250GB  of storage (additional 0.02/GiB) and 1TB of Bandwidth (Additional being 0.01/GiB).  For me this turned out to be roughly $25 Per Month for the month Bandwidth I am using.  That price is not bad for the peace of mind.

Anyway wanted to put this up here as I have not really seen much convo on S3 Compatible Storage that actually works outside of Wasabi.

Link to comment
Share on other sites

  • 3 weeks later...

Its a special hell when services say they are S3 compatible and they are...mostly S3 compatible. Its the mostly that ends up being a hitch "somewhere".

Reminder to all about this derpy thing. It does NOTHING at all, anywhere, other than when a user tries to download an attachment - then it wipes the signature in the temporary download url the user hits so they can get it when your non-Amazon S3 service chokes on the signature IPS generates (as IPS is locked in to Amazon exclusively).

That's it.

If you have problems using alt-S3 services (using the service at all, moving files, etc.), it's not on this plugin. 

When you try a not-Amazon S3 service, run through the basics. Can you upload a file via the service through Invision Community? If uploads work, can you delete? If so, can you move the files this S3 service and local storage (File System)? If so, you are in probably in business. Then, for completeness, if your users might be able to download the files that are stored on this service (Downloads, Attachments, etc.) give that a try. If it fails, throw this plugin in and see if clearing the signature lets them through (something something public buckets etc.). If it doesn't work you are sol or you'll need to find someone to write a full new filestorage method. FYI, if you go this route, there be dragons. You should have a firm control of your dev ops at a minimum...

If IPS support gives you grief if you submit a ticket regarding S3 problems because you have this plugin installed, then just uninstall this plugin and try them again (as stated above this plugin ain't doing much...). That said, I would not expect much from that request anyways as I'm pretty sure they are not going to help you with the S3 problems you have with your not-Amazon S3 service. IPS officially supports Amazon S3. Anything else is going to be outside IPS support scope.

Link to comment
Share on other sites

On 6/12/2021 at 8:58 PM, HDiddy said:

Content disposition which seems to be the issue for most of these

Yes...

On 6/12/2021 at 8:58 PM, HDiddy said:

basically saying you are not allowed to do this anonymously

And that's the thing that this plugin effectively "does", which won't work with, as you found out, a number of providers.

Someone up-thread (or privately) mentioned that when a user tries to dl a file and hits the temporaryDownloadUrl method there is "something" and the signature is doubled or generated twice or ??? (it's been awhile...) and that might be part of the problem with not-Amazon services as well. 

I just needed Wasabi to fully work, ran into the attachments dl problem, and threw this down to route around the problem. Everything else?

No Idea Idk GIF by SWR3

Link to comment
Share on other sites

On 7/1/2021 at 8:38 PM, All Astronauts said:

Yes...

And that's the thing that this plugin effectively "does", which won't work with, as you found out, a number of providers.

Someone up-thread (or privately) mentioned that when a user tries to dl a file and hits the temporaryDownloadUrl method there is "something" and the signature is doubled or generated twice or ??? (it's been awhile...) and that might be part of the problem with not-Amazon services as well. 

I just needed Wasabi to fully work, ran into the attachments dl problem, and threw this down to route around the problem. Everything else?

No Idea Idk GIF by SWR3

Thanks...

I have been to hell and back with Wasabi and ultimately decided I cannot deal with heir draconian egress policy.  The fact that they will not at least let you pay for more egress per GB is one thing, but to only allow egress up to the amount you have uploaded is just insanity.

Vultr seems to be working well for me. It is fast, I can setup multiple different object storage instances (paid separately) and the fact I can simply pay for more egress use is perfect.  I also find it has the most compatibility with Amazon S3 commands like Wasabi, and unlike Backblaze. Vultr Object Storage is still new, so hopefully they don't go the route of some of the others, but an option if people have need one.

Link to comment
Share on other sites

Wasabi is fantastic for what they are - a storage service - but that's really all they are and that is specifically their niche.

For those not in the know, for Wasabi your monthly egress should match more or less what you have stored. Store 100gb? Monthly out should be 100gb, though I doubt they will care much if it is 150 or a touch more. If you are pushing that egress out DAILY you are going to get cancelled.

There are two ways around this:

1) Store dummy data. People create a separate bucket, and just manually through an S3 tool dump whatever in there. 100gb real files, 900gb dummy files, = 1tb monthly egress. You get the idea. Need more egress? Add another TB to your storage bill (that would be 2TB for 12/month) and fill that up and now your 100gb of real files have 2tb of monthly egress.

2) The common sense solution? If you are egressing at a massive level daily I have no idea why you do not have a CDN in front of your server. Cloudflare, BunnyCDN, others... This is exactly what they are there for.

You end up with a build out as follows:

Server: For IPS install, serving the site, some files.

Wasabi/S3: Off-load the STORAGE of tons of user generated file content, or Downloads files, etc. This saves you paying for large amounts of storage on your webserver where storage costs are typically not viable (only so much room on the server itself). Not to mention you can blow your egress bandwidth out on the web server.

CDN: Takes the massive load off your server and off-site storage by caching and serving all those files.

Small hobby sites will never have to worry about most of this. You can get by with just a web server, or a web server and S3-type storage somewhere. If you actually are moving gigabytes to terabytes of data DAILY between your web server and off-site storage you either have the deepest of pockets or you "know a guy" (I know guy who back in the day ran a few Counter Strike servers out of his nationally-known airline regional office for awhile. Think they were in a closet. And then an off-site higher-up opened the closet and then he was out of a job. lol) - or, you've set up your site with a CDN. 

FYI all, this has been known for awhile but Backblaze is one of those providers that really, really, wants everything signed. I can hack this plugin to be compatible-ish with it but it ends up with attachments and Downloads files to be downloaded as filename.ext.abcd1234efgh5678 as unsigned requests may not contain content-disposition or content-type which makes your browser pull the file "as-is" and that means the IPS appended (and usually necessary) alphanumeric comes along for the ride. Not an ideal situation for your end-users.

Link to comment
Share on other sites

Well, see, the thing is, you step away from something long enough, and come back to it with fresh eyes and forgotten knowledge and stuff jumps out at ya.

Cryptic? I think I solved Wasabi finally. DigitalOcean Spaces probably as well. Both with the signed temporary urls for attachments and Downloads.

I might charge a touch as I'm probably throwing this into an app and adding in ACP widgets for the service status stuff.

Gonna simultaneously poke things and mull things over, if it is possible to do both at once.

It was such a benign, little, utterly ignorable thing too...

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...