Broken Images Storage Settings

[Josiah Wallingford]

I changed my storage settings from Amazon S3 to File System and now all of my images are broken. Is there something I need to run to let the system know to use the local file system images instead? Do the images not also get stored on the local system? If not, I will have to reactivate my S3 and will change it back to S3 (hopefully that will restore them).

[AlexWright]

It will break until they are all moved back from s3 to the local storage.

[opentype]

What Alex said.

Also: Don’t change the storage location unless you have a full backup. The good thing about the way the files are stored is that the location and the file names are independent. If something goes wrong while moving, you can just manually upload your backup to the new location and it will work. 

[Josiah Wallingford]

Ah okay, I thought it worked a little different. I thought all images were stored on local drive but for bandwidth saving reasons you could choose to have the images fed through the S3 server. 

[Joel R]

Amazon S3 = storage.  It's a replacement for local storage.

Amazon Cloudfront = CDN.  Caches your items on edge servers to make it faster for users.  

[Josiah Wallingford]

Ah okay, I will switch it back to S3... hopefully that doesn't mess something up and get the CDN setup.

[Joel R]

You don't have to have a CDN, and you can use any CDN provider you want.  Keep in mind that a CDN service will incur extra costs. 

[Josiah Wallingford]

On 4/24/2018 at 5:25 PM, Joel R said:

You don't have to have a CDN, and you can use any CDN provider you want.  Keep in mind that a CDN service will incur extra costs. 

Well, I want to host my images on s3 because the bandwidth costs are much higher on my host. I am hoping that the amazon cdn bandwidth costs will be ever lower than s3's are and then I can just store everything on s3 as a kind of running backup system (because storage for backups is also expensive on my host).

[Josiah Wallingford]

Well I messed something up (while not understanding how exactly the system works). While S3 was down I chose to use local storage. This broke all (most) of the images on the site (like users profile photos). When S3 came back up I switched it back to S3 and my images are still broken. What do I do now?

[opentype]

Check all locations you used previously (and backups if available) to see if you have the files somewhere. If you do, upload them manually to the active storage location. 

You can also establish an “FTP-like” connection to S3 and browse the file system there. 

[Josiah Wallingford]

The files are still there in the active storage location. They just show as broken on IPB. Should I download them all, delete them from the S3 and then re-upload them (upload and replace)? The images look fine when I download them from the s3.

[Josiah Wallingford]

I spoke too soon. It appears the files are damaged when I try to download them from s3...

Maybe it is just an odd coincidence in timing? Somehow amazon damaged all these files right when I switched it up?

[opentype]

Yeah, make sure to find a backup where the files work and are not 0 Byte. If you can’t find a working copy of your files, there is not much you can do. You could only delete the file references from the MySQL member database so the broken images are at least gone. 

[Josiah Wallingford]

Unless s3 keeps a backup I am sol. They are all showing as 357 byte. This was way more than just user profile images. This was files users uploaded as well. What do you think happened? It seems strange that s3 would just fail like that.

[opentype]

Well, lesson learned. Don’t touch the storage settings unless you have a backup. 

I once messed it up by just hitting save without making any changes. It then tried to move the files to the same location and afterwards deleted the old copy, which was also the new copy. ?

[Josiah Wallingford]

Interesting. When I try to preview one of the files inside of s3 or on my server I get a message like this:

InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.very-complex-access-key-here

 

[Josiah Wallingford]

So, if I have backups and everything is pointed to s3... I re-upload them to the s3 or to the uploads directory?

[opentype]

To the current upload location. You can also check by looking at the URL of the broken images. 

If you upload the original files over those 357 byte version ones, it should work again instantly. 

[Josiah Wallingford]

Okay. What happens to all the thumbnails that are produced by IPB that are also showing as 354 bytes? Will IPB recognize that the image has been replaced and rebuild those thumbnails? 

Also, s3 renames files which makes this a big pain. It would be nice if I could just replace the images through the IPB attachments area of the CP. 

[opentype]

3 minutes ago, Josiah Wallingford said:

Okay. What happens to all the thumbnails that are produced by IPB that are also showing as 354 bytes? Will IPB recognize that the image has been replaced and rebuild those thumbnails? 

No, but you can trigger rebuilding thumbnails. A function usually meant for when you change the image size settings. 

 

3 minutes ago, Josiah Wallingford said:

Also, s3 renames files which makes this a big pain. 

No. Not that I am aware of. I just checked your site and picked a random broken avatar:

https://s3.us-east-2.amazonaws.com/ttcommunityoffload/monthly_2017_09/W.png.bfd90e6f141219397d60c9655b42ec5c.png

The first part is the storage location: https://s3.us-east-2.amazonaws.com/ttcommunityoffload/
The second part is the image path: monthly_2017_09/W.png.bfd90e6f141219397d60c9655b42ec5c.png

Nothing about the file path depends on Amazon storage or any other type of storage location. That’s what I said before. You set the storage location and the you just need to have that file there. It doesn’t matter how it got there.

[Josiah Wallingford]

40 minutes ago, opentype said:

The second part is the image path: monthly_2017_09/W.png.bfd90e6f141219397d60c9655b42ec5c.png

The actual image name is W.png. All that is after that was added by S3.

[opentype]

8 hours ago, Josiah Wallingford said:

The actual image name is W.png. All that is after that was added by S3.

No. This is the regular IPS file name. 

[Joel R]

I believe IPS adds a file obfuscation so if you upload W.png, it becomes W.png.bfd90e6f141219397d60c9655b42ec5c.png

I'm sure they have a good reason.  Or maybe they just enjoy jackin' with our filenames.  

[Josiah Wallingford]

Either way, the backups I have the regular file name so I have to manually go into each content item and upload the images again by editing on the front end... yuck.

[opentype]

44 minutes ago, Joel R said:

I'm sure they have a good reason.  

The point of the obfuscation is to make file names unique. People could upload images with the same name. What should happen? Overwrite the previous files? It needs to be unique every time. The long file names makes sure of that. And this is added by the IPS software. it never changes and it doesn’t depend on the storage location.