deprecated multiple storage methods?

In Invision Community 5 there is a deprecation hint.

Multiple file storage locations will be deprecated in a future release. It is recommended that you start consolidating files to a central location. 

Currently, I have all in uploads but my files from Downloads. This directory is outside webroot. What does this hint exactly mean? Only, that the multiple options Advertisements, Attachments, Badges etc. are moved to one directory or does it mean that the files from Downloads have to be moved as well?

This is currently not in your deprecation tracker.

Anything that isn't pointed to the uploads folder needs to get consolidated there.

Thank you. So I can't store my files from Downloads outside webroot anymore?

Edited November 27, 2024Nov 27 by Hatsu

also I have the same question, 

files in downloads are pointed to the storage similar to AWS S3 and use domain example storage.domain.com, also every uploaded image on the community is using private CDN in a domain: cdn.domain.com,

how will it be possible to do when you deprecated multiple storage methods?

I will not be. As mentioned in the message there, these must be consolidated into a single storage location

This is very bad news for me. 

  On 11/28/2024 at 11:40 AM, Marc said:

these must be consolidated into a single storage location

I do not understand,

What is the technical reason for deprecating such a useful feature?

Why I should abandon optimization using CDN in favor of storing files in an uploads folder?

many communities with downloads feature have a situation that makes it impossible to keep files in the uploads folder on the host because they have large files or too many files

I never used an external storage and if it is really what it looks like (local storage only {uploads folder}), I’m curious: will this be discontinued for Cloud users too or only for self-hosted? I don’t see IPS hosting files from so many clients (+85%).

That’s a genuine question.

Edited November 28, 2024Nov 28 by Adriano Faria

How will this be handled in the upgrade from 4 to 5? 

  On 11/28/2024 at 3:36 PM, Joel R said:

How will this be handled in the upgrade from 4 to 5? 

Not for 5.0.0. Future.

An IPS staff person can correct me if I'm wrong, but my read of this was not that everything had to be ONLY under uploads, but that it had to be in a single location.  

Today, you could have a different storage location for each of the following:

Advertisements, Attachments, Badges, Club Custom Fields, Clubs Images, Custom Emoji, Icons & Logos, Image Proxy Cache, Login Method Icons, Profile Photos, Custom Profile Fields, Social Media Promotion Images, Ranks, Reactions, Referral Banners, Theme Resources, Forum Cards, Forum Icons, Customer-Uploaded Advertisements, Support Request Custom Fields, Purchase Fields, Customer Fields, Payment Gateways, Product Groups, Product Images, CMS Media, CMS Pages, CMS Records, Calendar, Event Cover Photos, Gallery Images

Do we really need a custom storage location for login method icons or custom profile fields, etc?

It SOUNDS like they're saying you'll define a single location for storage (be it the "uploads" folder or an S3 bucket or whatever).  Everything would be stored in that location, whatever/wherever it might be.  So you could continue to use a CDN, however you would not be able to have multiple subdomains for each of them.  It would simply be cdn.domain.com/club_images and cdn.domain.com/attachments or whatever the taxonomy is going to be.  

 

  On 11/28/2024 at 9:32 PM, Randy Calvert said:

An IPS staff person can correct me if I'm wrong, but my read of this was not that everything had to be ONLY under uploads, but that it had to be in a single location.  

Today, you could have a different storage location for each of the following:

Advertisements, Attachments, Badges, Club Custom Fields, Clubs Images, Custom Emoji, Icons & Logos, Image Proxy Cache, Login Method Icons, Profile Photos, Custom Profile Fields, Social Media Promotion Images, Ranks, Reactions, Referral Banners, Theme Resources, Forum Cards, Forum Icons, Customer-Uploaded Advertisements, Support Request Custom Fields, Purchase Fields, Customer Fields, Payment Gateways, Product Groups, Product Images, CMS Media, CMS Pages, CMS Records, Calendar, Event Cover Photos, Gallery Images

Do we really need a custom storage location for login method icons or custom profile fields, etc?

It SOUNDS like they're saying you'll define a single location for storage (be it the "uploads" folder or an S3 bucket or whatever).  Everything would be stored in that location, whatever/wherever it might be.  So you could continue to use a CDN, however you would not be able to have multiple subdomains for each of them.  It would simply be cdn.domain.com/club_images and cdn.domain.com/attachments or whatever the taxonomy is going to be.  

 

Interesting POV. Hope you’re right, even not having used it or needed it so far.

Edited November 28, 2024Nov 28 by Adriano Faria

I wish Invision would use the flysystem composer package in v5. 

I’ve been building apps in Laravel for the past six months and it is fantastic at letting you place your storage items anywhere you want using the league/flysystem package.  

There are so many great PHP composer packages these days, many of which can be used in any PHP, not just in the major frameworks like Laravel or Symfony. 

I plan on try to use Laravel’s Illuminate/Database package for Eloquent and Query Builder in my next Invision Community v5 app. This package uses PHP’s PDO driver so hopefully I can use SQLite and Postgres for per user SQLite databases and Postgres for my “source of truth” database.
 

 

  On 11/28/2024 at 9:32 PM, Randy Calvert said:

An IPS staff person can correct me if I'm wrong, but my read of this was not that everything had to be ONLY under uploads, but that it had to be in a single location.  

Today, you could have a different storage location for each of the following:

Advertisements, Attachments, Badges, Club Custom Fields, Clubs Images, Custom Emoji, Icons & Logos, Image Proxy Cache, Login Method Icons, Profile Photos, Custom Profile Fields, Social Media Promotion Images, Ranks, Reactions, Referral Banners, Theme Resources, Forum Cards, Forum Icons, Customer-Uploaded Advertisements, Support Request Custom Fields, Purchase Fields, Customer Fields, Payment Gateways, Product Groups, Product Images, CMS Media, CMS Pages, CMS Records, Calendar, Event Cover Photos, Gallery Images

Do we really need a custom storage location for login method icons or custom profile fields, etc?

It SOUNDS like they're saying you'll define a single location for storage (be it the "uploads" folder or an S3 bucket or whatever).  Everything would be stored in that location, whatever/wherever it might be.  So you could continue to use a CDN, however you would not be able to have multiple subdomains for each of them.  It would simply be cdn.domain.com/club_images and cdn.domain.com/attachments or whatever the taxonomy is going to be.  

 

 storing your theme/css/javascript files in the same single storage location as say your gallery images (s3 bucket) may not work that well in some cases

  On 11/27/2024 at 6:51 PM, Jim M said:

Anything that isn't pointed to the uploads folder needs to get consolidated there.

is there a time scale yet for this depreciation?

 

Edited November 29, 2024Nov 29 by sound

  On 11/28/2024 at 9:32 PM, Randy Calvert said:

An IPS staff person can correct me if I'm wrong, but my read of this was not that everything had to be ONLY under uploads, but that it had to be in a single location.  

Today, you could have a different storage location for each of the following:

Advertisements, Attachments, Badges, Club Custom Fields, Clubs Images, Custom Emoji, Icons & Logos, Image Proxy Cache, Login Method Icons, Profile Photos, Custom Profile Fields, Social Media Promotion Images, Ranks, Reactions, Referral Banners, Theme Resources, Forum Cards, Forum Icons, Customer-Uploaded Advertisements, Support Request Custom Fields, Purchase Fields, Customer Fields, Payment Gateways, Product Groups, Product Images, CMS Media, CMS Pages, CMS Records, Calendar, Event Cover Photos, Gallery Images

Do we really need a custom storage location for login method icons or custom profile fields, etc?

It SOUNDS like they're saying you'll define a single location for storage (be it the "uploads" folder or an S3 bucket or whatever).  Everything would be stored in that location, whatever/wherever it might be.  So you could continue to use a CDN, however you would not be able to have multiple subdomains for each of them.  It would simply be cdn.domain.com/club_images and cdn.domain.com/attachments or whatever the taxonomy is going to be.  

 

This is correct. 

  On 11/29/2024 at 10:14 AM, sound said:

storing your theme/css/javascript files in the same single storage location as say your gallery images (s3 bucket) may not work that well in some cases

Why not? 

  On 11/29/2024 at 12:11 PM, Esther E. said:

Why not? 

Last time I tried to store everything in an S3 bucket, it slowed the site down incredibly since uploading to S3 was incredibly slow. Downloading is fine though in most cases. 

How slow will it be to upload all those theme/css/javascript files?

Is this only an issue IN_DEV or will it affect  prod?

Or, is most of the uploading done in a cronjob?

Maybe I don’t understand how these non-user uploaded files are handled by the framework  

I suppose Invision has thoroughly tested the performance aspects of storing everything in an S3 bucket before making the huge decision to deprecate multiple storage methods so I should trust that this isn’t going to slow down my site from a user’s perspective. 

I’m worried that this is a bad decision. I wish Invision would support Flysystem for flexibly storing anything anywhere on a collection basis. Flysystem is fantastic in Laravel. 

Edited November 29, 2024Nov 29 by KT Walrus

  On 11/29/2024 at 12:59 PM, KT Walrus said:

Last time I tried to store everything in an S3 bucket, it slowed the site down incredibly since uploading to S3 was incredibly slow. Downloading is fine though in most cases. 

How slow will it be to upload all those theme/css/javascript files?

Our Cloud stores everything in S3 and so do countless self-hosted users.  You also are not forced to stored everything in S3 if you don't want too though.

  On 11/29/2024 at 12:59 PM, KT Walrus said:

Last time I tried to store everything in an S3 bucket, it slowed the site down incredibly since uploading to S3 was incredibly slow.

I can't say that's ever been our experience. Maybe you were using a location a long way from your server? It's always best to choose the nearest one possible. 

  On 11/29/2024 at 12:59 PM, KT Walrus said:

Last time I tried to store everything in an S3 bucket, it slowed the site down incredibly since uploading to S3 was incredibly slow. Downloading is fine though in most cases. 

How slow will it be to upload all those theme/css/javascript files?

Is this only an issue IN_DEV or will it affect  prod?

Or, is most of the uploading done in a cronjob?

Maybe I don’t understand how these non-user uploaded files are handled by the framework  

I suppose Invision has thoroughly tested the performance aspects of storing everything in an S3 bucket before making the huge decision to deprecate multiple storage methods so I should trust that this isn’t going to slow down my site from a user’s perspective. 

I’m worried that this is a bad decision. I wish Invision would support Flysystem for flexibly storing anything anywhere on a collection basis. Flysystem is fantastic in Laravel. 

 

 

  On 11/29/2024 at 12:11 PM, Esther E. said:

 

Why not? 

I'd say cost and performance, as was talking self hosted sites there

if we move to one single location then using a separate s3 bucket set up for large file directories such as gallery etc isn't possible

while such a move isn't end of the world stuff, I just thought that it needed mentioning as it is removing options

off topic but while here, I did send a pm ref support issue on your old account a few days ago

 

  Quote

is there a time scale yet for this depreciation?

guess we are talking months here, yep?

 

  On 11/29/2024 at 1:20 PM, Jim M said:

Our Cloud stores everything in S3 and so do countless self-hosted users.  You also are not forced to stored everything in S3 if you don't want too though.

Don’t you use only AWS?

That would explain why you don’t see a performance penalty for uploading to S3. 

I won’t use AWS or any big cloud service. I stay away due to cost and vendor lock in. 

I use Backblaze B2 for user uploaded content and local Minio S3 for everything else (and temporarily storing some user content). 
 

 

  On 11/29/2024 at 2:10 PM, KT Walrus said:

I use Backblaze B2 for user uploaded content and local Minio S3 for everything else (and temporarily storing some user content). 

If you're seeing performance impacts, you should explore your hosting and your connection to the vendors. The performance impact should be negligible. We run tests outside of our own AWS environment and we also are exploring other vendors where the cache reset on the theme files in v4 aren't an issue. Keep in mind, with v5, this won't be an issue because there are theme changes between v4 and v5 that rebuilds won't happen so frequently.

  On 11/29/2024 at 10:14 AM, sound said:

storing your theme/css/javascript files in the same single storage location as say your gallery images (s3 bucket) may not work that well in some cases

It’s the default behavior for any vanilla self hosted install and literally all cloud install. 

  On 11/29/2024 at 3:23 PM, Randy Calvert said:

It’s the default behavior for any vanilla self hosted install and literally all cloud install. 

you read it wrong, I was pointing out that splitting storage locations  via 2 different servers (self/S3) will no longer be possible

no big deal, just another option removed

onwards...

  On 11/29/2024 at 2:10 PM, KT Walrus said:

Don’t you use only AWS?

That would explain why you don’t see a performance penalty for uploading to S3. 

I won’t use AWS or any big cloud service. I stay away due to cost and vendor lock in. 

I use Backblaze B2 for user uploaded content and local Minio S3 for everything else (and temporarily storing some user content). 
 

It looks like IPS is using Backblaze as well...  I just looked at my cloud install and all of the storage is now on a non-S3 bucket.

back to this

I have just checked a beta test installation and it looks like that this new storage location format will be in place for the initial release of v5 

Which means that as part of the upgrade all files will be moved to the new locations during the upgrade?

Is that correct?