Amazon S3, Cloudfront and IPS - Not so Simple Storage

[The Old Man]

Since I read the recommendations here to try the combination of Amazon S3, Cloudfront with IPS, I've tried several times over the last 3 months to get it working properly with little lasting success. Just when you think you've got it working, the monthly cost is almost non existent, you find you haven't got it working for long and you begin the process of recovering your files and studying the AWS documentation before trying again.

The main issues I've found each time I've tried this are:

1. repeated issues with objects (e.g. Gallery images, emoticons etc) randomly disappearing from S3 buckets after a few days.
2. large numbers of files being abandoned and orphaned in the S3 bucket after transferring them back to my IPS web hosting storage.
3. the repeated attempts to transfer to S3 and then reverse to retrieve them eats into the free tier allowance and end up costing but only a few dollars.
4. Files end up with very long double file names because the have references appended, so you end up with image file names like:

myphoto.jpg36544785)5)(4(4ggygytytdythuhkuh.myphoto.jpg3534uuyvghgyvtyv54564574675877yukyukt.jpeg

I've tried again and failed again, only half of the Gallery images were accessible via S3 and Cloudfront, so again I'm now left with loads of Gallery images broken on my site because IPS 4.2.7 failed to safely retrieve them all back again, there are no log entries in IPS to confirm why it failed to retrieve them all (presumably because according to the task progress bar it, believes it finished the task), and there is no option to retry.

Why does IPS not provide the option to instead leave the original files where they are, and instead upload copies to S3 storage, so that you can later remove the local originals if if there are no problems?

Why no functionality to retry and fetch orphaned files left on the S3?

[Joel R]

Yeah you have to be extremely careful about disabling / enabling a file storage method, because that triggers the background move process of all images.  In the future, only enable S3 for one file storage while you test it out.  I made the mistake of enabling everything and I also encountered the same issues as you many years ago. 

You should submit a support ticket to: 

1) For IPS to update their Amazon S3 documentation in case there have been any changes. Make sure you're linking to the correct URLs like https://  

2)  Are you cloud or self?  If you're cloud, then keep asking support tickets addressed to Rhett.  

[The Old Man]

Good advice, Joel. I actually started this attempt with just Calendar cover images and all seemed okay. So I got complacent and did  Gallery images. Big mistake once again, especially since a folder of them is now orphaned. 

Yes self-hosted, I have put a ticket in before and Lindy spotted an issue with the CDN path, I thought that had fixed it but the files started disappearing randomly again.

Only my AWS non-root admin account has permissions to modify the S3bucket, along with Cloudfront Origin.

I have a CNAME set up pointing at the Cloudfront distribution, but have now found some conflicting advice on naming the bucket. Like the outdated video in the guidance section here and also as per the AWS documentation, I called my buckets by simple domain friendly website names, but I've since read how you can, possibly should, name them with your exact domain name like my website.co.uk or with a sub domain like cdn.mywebsite.co.uk. 

I noticed here that they don't use a CNAME to mask the Cloudfront url any more, perhaps they had similar issues.

Next time I try it will be with the different bucket naming convention and some logging enabled.