Gallery - delete uploaded image after resizing?

[NordWulf]

When you upload an image to the Gallery, it resizes the original to a large, medium, small and thumbnail size. The sizes for these 4 formats can be set in the admin settings.

The original image is not deleted from the file system. Can these uploaded images automatically be deleted? They can be quite big in size and can fill up disk space quickly when you have many images in the Gallery.

[Stuart Silvester]

Hi Frank,

Gallery does require that the original image is preserved (it's used for various things like rebuilding thumbnails and applying watermarks -- or when watermark settings are changed). Gallery in 4.3 however does reduce the number of images stored to 3, reducing both disk space and bandwidth usage:

 

[NordWulf]

Thanks for the information Stuart, I figured this was why the original image is not deleted. When you have a large number of gallery images, even only 1000 images of 1MB will take up 1GB of space that is never used. When you have the largest image size set to say 800x800 which is under 100 Kb per image, those 1000 images only take up 100 Mb.

If you have 100 members with 100 gallery images each, that is already 10GB of used space without the other resized images. 

It would be nice if there was an option to upload the image, rezise it to the largest size in your gallery settings, discard the original and use the large resized image as the stored image for rebuilding thumbnails, resizing and watermarks applying and changing.

[Joel R]

I'm absolutely against this idea even though I absolutely share your concerns.  You should never ever delete the original. Its called the original for a reason.  I do, however, commiserate with your concerns about storage and bandwidth and there are a couple of different ways to approach this:

- in 4.3 IPS only saved two variants instead of four on top of the original.  That will be a huge boost to image-centeic communities by just upgrading.  

- you should periodically check your attachments in ACP.  You can filter by largest file sizes.  I was blown away that I had some images (normal JPEG file size) that were 20 MB for that single file! 

- if you're a big uploader then make sure you prep your own files in advance.  There are a variety of free batch tools to resize in advance.  

- there are tools in the Marketplace like Kraken by @stoo2000 that offers additional image optimization.  I love this app because it pays for itself in the long run.  

[NordWulf]

Thanks for the tips Joel.

I did a test with an image in my iPhone library and checked the file sizes in cPanel File Manager

Uploaded to a forum post with max 800x800 size:

91 Kb

Uploaded to the gallery with 1024 x 768 setting for large image size:

Large: 85 Kb
Medium: 41 Kb
Small: 14 Kb
Thumbnail: 5 Kb
Original: 2.75 Mb

Yes, that's 2.75 Mb for just one 3000x4000 smartphone image that is being saved as the original that will probably never be used again. Users really have no idea about the size of images they are taking and uploading with any mobile device.

Is it ok to manually delete these files?  Or perhaps create some kind of batch job that deletes these original files. For example, delete any .jpg file with a name that does not start with 'large', 'medium', 'small' or 'thumbnail'.

[jair101]

While I do appreciate the concern that originals are important in case you want to process the files again hypothetically in the future, I think there should be an option to delete them, admin takes full responsibility. I don't like decisions wasting many GB taken for me by default. 

Reducing the number of processed images from 4 to 2, cool, save 50kb, but keep the monstrous original of 5MB intact. Nothing "huge" about this "boost", it is more like "meh" in the grand scheme of things. Same with kraken, which can cost quite a bit on top and even charges you just for stripping metadata if they decide that your image is optimized enough to begin with. Prepping files in advance is nice, but I am one of the thousands members in my community and they don't do that and upload images like crazy. Joel, most of your suggestions are not solutions to the problem, they are barely helping.

At the very least, gallery originals should have their separate storage settings, allowing us to send them to S3 or Glacier and not keep them on the live storage. My live storage is fast and expensive, cluttering it with files that will never be accessed by the users is waste of space. 

 

[bfarber]

Unfortunately, we cannot *not* store the original for the reasons mentioned. Taking watermarks for instance, if you apply a watermark to the original, then don't keep the original, but then adjust your watermark down the road and rebuild your images from the AdminCP, the images in your gallery would be double watermarked. This isn't an issue we can just let users "deal with".

That said, I'm not opposed to giving a secondary storage option for the original image if that would be helpful (so you could store them in Glacier or something similar, as you mentioned). That seems like a good compromise to me?

[NordWulf]

25 minutes ago, bfarber said:

Unfortunately, we cannot *not* store the original for the reasons mentioned. Taking watermarks for instance, if you apply a watermark to the original, then don't keep the original, but then adjust your watermark down the road and rebuild your images from the AdminCP, the images in your gallery would be double watermarked. This isn't an issue we can just let users "deal with".

The idea is to resize the original uploaded image to the size settings for the large image settings. So you will have duplicate images for the original and 'large' image but you can still rebuild all sizes and re-apply watermarks.

In my example, having the original 2.75 Mb 3000x4000 image resized to 100 Kb 1024x768 first and then create the large/medium/small/thumbnail from that resized image would be perfect.

[bfarber]

2 minutes ago, Frank Wulfers said:

The idea is to resize the original uploaded image to the size settings for the large image settings. So you will have duplicate images for the original and 'large' image but you can still rebuild all sizes and re-apply watermarks.

In my example, having the original 2.75 Mb 3000x4000 image resized to 100 Kb 1024x768 first and then create the large/medium/small/thumbnail from that resized image would be perfect.

(1) You lose all EXIF data associated with the image when it is resized with GD

(2) If you later change your mind in 2 years because super-retina displays and larger viewports are available and want your large images 2048x1536, you wouldn't be able to do that without distorting the images.

[jair101]

40 minutes ago, bfarber said:

That said, I'm not opposed to giving a secondary storage option for the original image if that would be helpful (so you could store them in Glacier or something similar, as you mentioned). That seems like a good compromise to me?

Yeap, this will be great. Right now I am paying about 50c per GB additional space per month. Compare that to $0.0245 for regular S3 and $0.0045 per GB for Glacier. I suppose Glacier will be harder to implement in case rebuild of images is requested, but I will be more then happy with regular S3 as well. 

If you don't do it, maybe some third party developer can see a market. Thats 20x the difference for regular S3 and 100x for Glacier, I don't mind passing some of the savings to a developer. 

[NordWulf]

An option for site admins to keep the original in its full uploaded size or resize to a new setting would be best. The size of a 2048x1536 file would still be much better than 3000x4000.

Being able to upsize in the future is a great idea for sites that have a gallery with good quality photography. But with galleries where regular users just want to share snapshots with their average quality smartphone, a smaller original image will be just fine. When you upload an image in a forum post, the original isn't saved either.

[Stuart Silvester]

13 minutes ago, jair101 said:

but I will be more then happy with regular S3 as well. 

You can already put all of your Gallery images on S3, take a look at: 

 

[jair101]

29 minutes ago, Stuart Silvester said:

You can already put all of your Gallery images on S3, take a look at: 

Yeah, but this does not work for me as S3 has pretty hefty traffic charges, adding cloudfront somehow helps, but their pops are far away from me. On top of that my traffic is very localized, so overall it is not a good solution for me. I made some brief tests, S3 was slower than my local VPS. 

But storing the originals will be awesome, noone access them anyway, so no traffic charges and the storage is really cheap. 

[NordWulf]

23 minutes ago, jair101 said:

But storing the originals will be awesome, noone access them anyway, so no traffic charges and the storage is really cheap. 

If the originals are stored in a seperate folder, you can probably download them, delete online and store offline? And upload again in case you want to to resize or make watermark changes.