[Suggestion] Storage method: FTP

[Hisashi]

@Matt there's a question I always ask myself. "Why didn't IPS add other storage methods until today?"

Amazon S3 can be a very expensive option for some people. FTP can be a much more interesting option, allowing administrators to distribute their files on different servers.

This also helps to reduce the processing load of files such as downloads and gallery on main server where the site is installed.

Edited April 16, 2023 by Hisashi

[Randy Calvert]

When you upload a file to AWS, it's using HTTP(S) to post the image via an API.  That is a one-time transfer from your server to AWS servers.  Once it is there, anytime the file is called, it's delivered directly from AWS via the bucket URL.  Meaning AWS is the one delivering the image instead of your server.

If you were to implement FTP, while it would not be difficult to upload the file...  downloading it would be much more difficult.  Modern browsers no longer allow the FTP protocol in the browser.  (For example with Chrome, it was removed in version 59.) 

This means IPS would have to each time it was requested, login to the FTP server and download it...  then deliver it to the user.  That would dramatically increase the work on the server, slow down the delivery of the site (because it has to be downloaded and then sent to the user), and increase bandwidth on both the web and FTP servers.  

[Hisashi]

Downloaded?? No.

The file would be available like any other image on the site, it would just be pointing to another server.

[Randy Calvert]

OK. If it’s on an FTP server…. That FTP server is not automatically accessible to WWW.  That means there is no way for a browser potentially to access that file.  

Browsers no longer support things like ftp://www.domain.com/somefile.jpg

In addition to the above, FTP is not secure and can be be read by 3rd parties in transmission. If you really want to do something like this, instead of using FTP, mount the other server’s storage space via NFS and treat it like a local folder in IPB storage methods.  

That NFS mount might be something like /mount/share/somefile.jpg

Edited April 16, 2023 by Randy Calvert

[Hisashi]

Maybe I didn't use the correct term for this suggestion. I think the proper method is SSH

An option would be created where you would inform the data of the other server that will be used to store the files.

Then you would create a subdomain to point to the IP of this server.

Edited April 16, 2023 by Hisashi

[Hisashi]

Additional to topic

There are other service options.

• Google Cloud
• Backblaze
• Wasabi
• Azure
• DigitalOcean

[Martin A.]

13 hours ago, Hisashi said:

Additional to topic

There are other service options.

• Google Cloud
• Backblaze
• Wasabi
• Azure
• DigitalOcean

Those highlighted are S3 compatible and can be used by Invision Community.

 

14 hours ago, Hisashi said:

Maybe I didn't use the correct term for this suggestion. I think the proper method is SSH

An option would be created where you would inform the data of the other server that will be used to store the files.

Then you would create a subdomain to point to the IP of this server.

Mount it as a volume on your server and access it as if it was part of your file system, and use the "Custom URL" in the file storage setting to point it to the other server.

Edited April 16, 2023 by Martin A.

[Hisashi]

30 minutes ago, Martin A. said:

Mount it as a volume on your server and access it as if it was part of your file system, and use the "Custom URL" in the file storage setting to point it to the other server.

I kind of do this for specific files, but if there was a way to automate it with the inivision community that would be perfect.

Edited April 16, 2023 by Hisashi

[UncrownedGuard]

9 hours ago, Hisashi said:

I kind of do this for specific files, but if there was a way to automate it with the inivision community that would be perfect.

Once it is mounted, just add the folder into the Admin CP in System -> Files -> Storage Settings (Top Right) -> Configuration

Then you can go back to System -> Files -> Storage Settings (Top Right) and change all the resources you want offloading onto the mounted drive.

I have never tried this with a mounted drive myself, but no reason it should not work.

Edited April 17, 2023 by UncrownedGuard

[Hisashi]

This is impossible when a user is hosted. It will always be stored on the localhost of the installation.

[Daniel F]

We had FTP but it was really very bad

See 

 

 

[Hisashi]

9 hours ago, Daniel F said:

We had FTP but it was really very bad

See 

Does IPS intend to add other storage methods, so that customers can have more options?

[Marc]

Im not aware of any intentions to at present. If you have any specific suggestions, of course bring them up

[KT Walrus]

39 minutes ago, Marc Stridgen said:

Im not aware of any intentions to at present. If you have any specific suggestions, of course bring them up

Please consider adding a separate Backblaze B2 storage method. The existing S3 method doesn’t completely work.

B2 is very cheap at $5/TB and when combined with free Cloudflare CDN, B2 has NO egress fees  So, all you pay is $0.005/GB stored  Using Cloudflare Workers, you can even make the whole bucket private.

Making a separate storage method for B2 will allow you to fix the small issues with using Amazon S3 method so B2 works completely.

 

Edited April 20, 2023 by KT Walrus

[Hisashi]

I support, Backblaze B2 is an excellent choice and widely known in the storage market.

https://www.backblaze.com/

[KT Walrus]

2 hours ago, Hisashi said:

I support, Backblaze B2 is an excellent choice and widely known in the storage market.

https://www.backblaze.com/

Yes they are. And in full disclosure, I have a large amount invested in this company as I see them eventually becoming very large and taking significant share of the cloud storage market away from Amazon S3. Backblaze B2 is significantly cheaper to use especially since AWS charges such high egress fees to deliver content to end users.

[Hisashi]

Even finding this suggestion difficult to pass, due to the fact that IPS uses the Amazon S3 service in the cloud plans.

I really hope for news in the future in this regard, it's bad to have only one option when there are so many other services of the same quality or better. Worldwide known.

[KT Walrus]

22 minutes ago, Hisashi said:

 due to the fact that IPS uses the Amazon S3 service in the cloud plans..

It would be simple for IPS to use B2 in their cloud plans and they would reap the savings in storage and bandwidth costs. Note B2 use would necessitate using some third-party CDN, like Cloudflare, BunnyCDN, or others.

Note Backblaze is part of the Cloudflare's Bandwith Alliance:

BunnyCDN is also an excellent choice for delivering MP4 videos stored in B2 since their CDN is optimized for video/large file delivery and charges as low as $10/TB for bandwidth. Transfer from B2 to BunnyCDN is free, of course.

Edited April 20, 2023 by KT Walrus

[Hisashi]

That was not my point. I'm talking about the fact that Amazon S3's service is probably perfect in their eyes, and they don't see the point in adding another one.

 

12 minutes ago, KT Walrus said:

BunnyCDN is also an excellent choice for delivering MP4 videos stored in B2 since their CDN is optimized for video/large file delivery and charges as low as $10/TB for bandwidth. Transfer from B2 to BunnyCDN is free, of course.

I use their video(Stream) service, their CDN is ridiculously GOOD. With an excellent player and functionalities.

Edited April 20, 2023 by Hisashi

[KT Walrus]

5 minutes ago, Hisashi said:

That was not my point. I'm talking about the fact that Amazon S3's service is probably perfect in their eyes, and they don't see the point in adding another one.

My point was that IPS could save quite a lot if they are currently paying a significant amount of S3 storage and delivery. The "hidden costs" with using AWS is bandwidth fees which can really add up. Many B2 customers have saved 5x on their cloud storage/bandwidth fees by migrating from AWS to B2. Backblaze will do the migration for free and you will have minimal downtime if you use their service.

So, whatever IPS currently pays to AWS for cloud storage and delivery (including the bandwidth fees) could be reduced by up to 80% allowing IPS to make more profit on their cloud service.

17 minutes ago, Hisashi said:

I use their video(Stream) service, their CDN is ridiculously good. With an excellent player and functionalities.

If you don't want streaming, but only MP4 delivery, BunnyCDN can be configured for cacheing large MP4 files in 5MB chunks from B2 to the CDN PoP. This allows significantly better performance for playing the MP4 in the user's browser.

[Randy Calvert]

But then they increase management cost, support costs, more separate vendors to deal with… 

Not to mention now needing two separate WAFs. 

There is a lot to be said for a single throat to choke. 🙂 

Also at the enterprise level, they don’t pay what is listed on the website. I know for a fact that lower prices are available based on overall spend. 

[KT Walrus]

21 minutes ago, Randy Calvert said:

Not to mention now needing two separate WAFs. 

Why would you need to maintain a WAF between the user's browser/BunnyCDN/B2 bucket for serving files? Nothing is traveling to AWS and IPS's cloud infrastructure. BunnyCDN and Backblaze are very simple to use and set up.

Now, if the files need to somehow make there way through AWS, it can add a big cost. Amazon's goal is customer lock-in and they try to do everything to prevent customer's to easily use services that AWS also provides.

It really is the "devil's bargain". But, the web is evolving to easy multi-cloud integration and cloud storage is easily separated from AWS, if the customer wants to benefit from significant cost savings.

Edited April 20, 2023 by KT Walrus

[Randy Calvert]

I want a WAF in front of EVERYTHING I have.  This includes being able to apply rate controls/DDoS protection, etc.  

Even if rates for bandwidth are "cheap"...  If an attacker sends a trillion requests for the objects, I don't want stuck with a huge bill.  

If you're going to talk about "lock-in", solve it by using multiple providers that offer complete stack solutions such as Azure or Google where you have the database, application, security, caching, micro services, etc all brought into multiple providers.  Simply moving CDN out somewhere else is not a good way of designing an enterprise level application.  

[KT Walrus]

@Randy Calvert, you misunderstand what I am saying.

I am not considering cloud storage/file serving to be part of your "enterprise level application". 

You simply store files in Backblaze B2 and serve those files with Bunny CDN. That's it. It's "plug and play".

Here is a blog on Backblaze about Bunny vs Cloudfront that is a quick read you might be interested in:

https://www.backblaze.com/blog/aws-cloudfront-vs-bunny-net-how-do-the-cdns-compare/

Invision can continue to use Cloudfront for Invision Community since that would be cheaper for AWS servers. But attachment serving can easily be separated from AWS where AWS doesn't get a nickel for any files stored/served to end users.

Setting up both B2 and Bunny CDN is incredibly easy and, if that saves you thousands of dollars a year you were paying to AWS for attachment hosting/serving, then this is a no-brainer. All that cost savings can either be passed along to the clients paying for Invision Community in the Cloud or can go to the bottom line as additional profits for the company.

In my opinion, Invision would be more profitable if they could halve the current pricing of their cloud plans. But, that is just an opinion. But businesses being more efficient is never a bad idea, again IMO.

[KT Walrus]

Note: I have put in a request to Backblaze to add tus uploading to a B2 bucket. If this is implemented, Invision Community could directly upload all files directly from the user's device using tus or an uploader like Uppy. This would mean files wouldn't need to be sent to AWS at all so AWS wouldn't be able to charge for egress to the B2 bucket saving even more money.

These egress fees aren't a problem for me since I use OVH Cloud for my backend servers and bandwidth is included/unlimited for egress to B2 (or anywhere else). Also, I use Cloudflare Tunnel to protect my backend servers. No WAF needed for my backend servers since they reach out to establish the tunnel to the Cloudflare CDN.

Edited April 20, 2023 by KT Walrus