Moving files to amazon S3 broke them

[zyx]

Followed the steps to move 'gallery image' files over to amazon s3 storage. 

Many worked as expected, but around 20% the files no longer load/work. Tried clearing cache. 

All of them are in the same folder with same permissions.

Any ideas? 😕 

Thanks in advance.

Edited April 24, 2020 by theipsguy

[zyx]

I believe it may be files that have special characters in its title that are affected (could be wrong about that, havent checked them all yet as many thousands). 

Just seems like that would be a crazy bug to have not already been picked up... so I'm hoping I'm just missing something. 

Edited April 24, 2020 by theipsguy

[Eudemon]

how do you sync with the app, using s3 fuse?

[zyx]

9 hours ago, Eudemon said:

how do you sync with the app, using s3 fuse?

As of right now I just followed this guide exactly:

 

Just using standard S3 storage. Majority of files worked as expected after transfer, but some (seemingly those with special characters in the title) no longer load via the gallery. 

I believe it's a (pretty major) bug related to the file names. 

 

Edited April 25, 2020 by theipsguy

[Eudemon]

you have link to an example gallery page that doesn't work? maybe i can take a look

 

the other method I mentioned is using aws sdk, and a package called s3 fuse or something that mounts a buckets to your server like a regular folder

Edited April 25, 2020 by Eudemon

[GTServices]

TIP 1: If you are paying over $100 per month for S3 services you may want to consider a dedicated server @ $120 + Stackpath CDN @ $25.

As your site traffic increases so will your S3 costs.

You can sync your local static folders to the new server so that all images are copied to the new server. The CDN will then fetch the static files from the new server.

TIP 2: If you stick with S3 make sure you use a CUSTOM DOMAIN so that all your static files are using a domain that you control - that you can move over to another service in case costs keep climbing. If you don't do this you can lose SEO from images, pdfs, etc that are currently driving traffic to your site when and if you decide to move to another service.

[zyx]

Thank you both very much for the tips.

However, surely invision's S3 integration should work without buying anything extra? 

@Matt I believe this is a critical fault with changing the storage method. By switching to amazon s3, any file with special characters in the title no longer works. (all other files did work, so it as not an issue with the transfer, but with the integration between IPS and S3 - I followed the official Invision guide exactly). 

This is not just a minor glitch, it has wiped out around 20% of our files which is an incredible amount of data.

Edited April 26, 2020 by theipsguy

[Eudemon]

https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

if this is what you talking about, you prob want to do some pre processing to alter the file name both in your db and s3 key

[zyx]

15 minutes ago, Eudemon said:

https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

if this is what you talking about, you prob want to do some pre processing to alter the file name both in your db and s3 key

Thank you. That seems to be it exactly. The files affected are the ones that had 'characters to avoid' from the page you sent.

Unfortunately I do not know how to do the kind of preprocessing to either the s3 key or my db in order to fix that. I very much assumed that the s3 tutorial provided by the Invision was enough, and I cannot find any kind of guidance on the pre processing you mention. 

So I think you are exactly right on what the problem and solution is, I just have no idea how to perform the solution. 😅

Thanks for your continued help, it really is very appreciated. 

[Eudemon]

have a look at /applications/downloads/modules/front/downloads/submit.php

you can prob have a hook to filter out special character in filename before upload, not tested

[zyx]

2 minutes ago, Eudemon said:

have a look at /applications/downloads/modules/front/downloads/submit.php

you can prob have a hook to filter out special character in filename before upload, not tested

Thank you, although it is all of the existing files I'm most concerned about.

It just seems odd Invision wouldn't account for that in the transfer procedure, no?  I can't be the first person to have gallery files with special characters in the title 😂

I am not expecting you to come up with a solution by the way, I already appreicate all your efforts.

I am just posting in the hope someone out there perhaps has been through this and found a problem, or ideally someone from Invision can work on a solution to this problem, and I'll then re-do the transfer. 

[Eudemon]

don't worry I am just bored of quarantine

for existing files, you can run a query to update db, keep in mind you also need to query update s3 keys as well, prob not hard to do thru aws command line

[zyx]

4 hours ago, Eudemon said:

don't worry I am just bored of quarantine

for existing files, you can run a query to update db, keep in mind you also need to query update s3 keys as well, prob not hard to do thru aws command line

I think you are right that running a query to db and to aws command line is the answer, I'm just not sure what that would be to rename titles with special characters?

Edited April 27, 2020 by theipsguy

[Eudemon]

UPDATE files
SET filename = replace(filename,'{','-') 

 

for s3, might be better to write a simple script, get an aws credential,

fetch filename, replace special char then update old key with new key,

aws sdk is widely available for many laguages

Edited April 27, 2020 by Eudemon

[Donkerrood]

On 4/26/2020 at 6:20 PM, GTServices said:

TIP 2: If you stick with S3 make sure you use a CUSTOM DOMAIN so that all your static files are using a domain that you control - that you can move over to another service in case costs keep climbing. If you don't do this you can lose SEO from images, pdfs, etc that are currently driving traffic to your site when and if you decide to move to another service.

Currently looking into this - any guidance on how to do that? Should I just add a CNAME record like cdn.domainname.com and point that to Amazon?