Ported to external storage, but some images still show old link

[Clover13]

I ported my /uploads to external storage (Backblaze, which I know isn't supported directly but does work and I followed this guideline), however some of the images didn't update to the external source URL.  The majority are just fine, but there are exceptions I'm finding that are resulting in broken images pointing to my host server's original URL rather than the external URL.

Are there additional tasks I can or need to run?

I tried clearing the IPS cache, but that won't resolve the image URLs being incorrect.

[Clover13]

Some additional info from the forum_posts.post table...

Old/Bad host stored image references (20,747 of them):

<img class="ipsImage ipsImage_thumbnailed" data-fileid="145857" data-ratio="133.21" width="563" alt="85296E61-B8EF-4C55-8640-BE98CD471290.jpeg.ed178aa445a008763273ec91835a0721.jpeg" data-src="<___base_url___>/uploads/monthly_2022_05/85296E61-B8EF-4C55-8640-BE98CD471290.jpeg.ed178aa445a008763273ec91835a0721.jpeg" src="<___base_url___>/applications/core/interface/js/spacer.png" />

 

New/Good external image references (72,055 of them):

<a href="<fileStore.core_Attachment>/monthly_2022_05/FDE37ACC-83DC-462E-BB82-7D5EF141747C.jpeg.b8cbb5ee97faaa5cf7f188beae43e774.jpeg" class="ipsAttachLink ipsAttachLink_image"><img data-fileid="145846" src="//www.mysite.com/applications/core/interface/js/spacer.png" data-src="<fileStore.core_Attachment>/monthly_2022_05/FDE37ACC-83DC-462E-BB82-7D5EF141747C.jpeg.b8cbb5ee97faaa5cf7f188beae43e774.jpeg" data-ratio="133.21" width="563" class="ipsImage ipsImage_thumbnailed" alt="FDE37ACC-83DC-462E-BB82-7D5EF141747C.jpeg"></a></p>
<p><a href="<fileStore.core_Attachment>/monthly_2022_05/82AF7ED7-AD6B-46EE-BEEF-134197D01463.jpeg.8c4b4ec85b591f988aa5c00d38b52522.jpeg" class="ipsAttachLink ipsAttachLink_image"><img data-fileid="145847" src="//www.mysite.com/applications/core/interface/js/spacer.png" data-src="<fileStore.core_Attachment>/monthly_2022_05/82AF7ED7-AD6B-46EE-BEEF-134197D01463.jpeg.8c4b4ec85b591f988aa5c00d38b52522.jpeg" data-ratio="133.21" width="563" class="ipsImage ipsImage_thumbnailed" alt="82AF7ED7-AD6B-46EE-BEEF-134197D01463.jpeg"></a>

 

Edited May 12, 2022 by Clover13

[Clover13]

@ASTRAPI@Joel R @CoffeeCake

@All Astronauts

Re:

 

Edited May 12, 2022 by Clover13

[Clover13]

Additional update: 

Found this seems to relate to existing attachments in a post that were NOT explicitly Inserted but rather just attached and implicitly displayed. 

However once the external source is setup, and you create a new topic and attach a file without explicitly Inserting it, it does embed it with the correct external link.

So now the challenge is how to update 20,000 posts where users didn't explicitly Insert their images into a post but rather just attached them.

I wonder if this same problem exists on the S3 integration as it seems to be coupled to attachments versus explicitly inserting the attachment after uploading them.

Edited May 12, 2022 by Clover13

[Marc]

 

Unfortunately, we cannot provide support with 3rd party storage methods. Our software has only been internally tested with the s3 storage method it was designed for. Anything outside of that would need to be fully tested, and if problems occur, reverted to previous state.

[Clover13]

@Marc Stridgen

OK let me ask, regarding migrating /uploads from server storage to S3 by changing the configuration within IPS, is there an IPS process that updates the forum_posts.post table rows as part of that changing the references from

 

<___base_url___>

to

<fileStore.core_Attachment>

If not, what changes those references?  Is it even part of this process/change to File Storage or is this something that should have been updated by an upgrade earlier?  Seems odd that there is a mix of <___base_url___> references mixed in with the <fileStore.core_Attachment> references.

[Marc]

Changing items in the database is not something you should be doing in any case. There will be items such as the above that are placemarkers for items that would be replaced with URL locations in code.

[Clover13]

1 minute ago, Marc Stridgen said:

Changing items in the database is not something you should be doing in any case. There will be items such as the above that are placemarkers for items that would be replaced with URL locations in code.

OK, I'm simply trying to find where these are changed/parsed and why the parsing is inconsistent.

I am not sure how any arbitrary attachment URL sometimes gets parsed one way (as <___base_url__>) and other times a different way (as <fileStore.core_Attachment>), when they are in the same location.  I would think everything would consistently be <fileStore.core_Attachment> which points to whatever your fileStore is (whether it's external or it's your base_url).

I would think I'd be running into this on S3 as well, unless there was something that either updated the table data or at runtime parses and replaces <__base_url__> with the proper <fileStore.core_Attachment> reference.  Are you saying that is what happens at runtime?

I believe this is data sided as there are pre-external migration forum_posts.post references that are correct <fileStore.core_Attachment> and others that are not <__base_url__>.  How or why would that be even if I moved to S3?

 

[Clover13]

FWIW, I did just proof out updating a forum_posts.post record from <__base_url__>/uploads to <fileStore.core_Attachment> did work as desired and the previously problematic post displayed the image correctly.

Edited May 12, 2022 by Clover13

[Marc]

Glad to see you have that resolved on your instance. 

[Clover13]

12 minutes ago, Marc Stridgen said:

Glad to see you have that resolved on your instance. 

This looks like the same issue being experienced recently after porting to S3:

 

[Clover13]

@Marc Stridgen is there a way to re-trigger the push to an external storage location via a task?  There are some files that weren't moved over and are still stored locally (different problem than the one reported here initially).  I don't want to switch the IPS AdminCP configuration from my external bucket storage back to the origin server-sided storage and push back again given the time it takes to do all files.

Edited May 12, 2022 by Clover13

[Jim M]

30 minutes ago, Clover13 said:

@Marc Stridgen is there a way to re-trigger the push to an external storage location via a task?  There are some files that weren't moved over and are still stored locally (different problem than the one reported here initially).  I don't want to switch the IPS AdminCP configuration from my external bucket storage back to the origin server-sided storage and push back again given the time it takes to do all files.

There are no means to do this via the software again, I'm afraid. The only means would be to manually move them.

[Clover13]

1 minute ago, Jim M said:

There are no means to do this via the software again, I'm afraid. The only means would be to manually move them.

Thanks @Jim M!  Is there anything that needs to run after I manually move them to update any other aspect of the IPS table data or files to account for the move of that subset of files?

[Jim M]

15 minutes ago, Clover13 said:

Thanks @Jim M!  Is there anything that needs to run after I manually move them to update any other aspect of the IPS table data or files to account for the move of that subset of files?

There should not be as this should already have been updated by the process that ran.

[Clover13]

6 minutes ago, Jim M said:

There should not be as this should already have been updated by the process that ran.

OK, thank you.  For reference, is there anywhere I can check to see what that process is or what it runs to verify everything it was supposed to do has been done and if not correct it myself (as I'll be doing in moving these files over to external storage manually)?

[Jim M]

5 minutes ago, Clover13 said:

OK, thank you.  For reference, is there anywhere I can check to see what that process is or what it runs to verify everything it was supposed to do has been done and if not correct it myself (as I'll be doing in moving these files over to external storage manually)?

I'm unsure what you mean here? You would simply move the files manually to your external storage. The software part would have already been converted. If you see any missing images or files, we would need to look at this but again, we only directly support Amazon S3.

[Clover13]

14 minutes ago, Jim M said:

I'm unsure what you mean here? You would simply move the files manually to your external storage. The software part would have already been converted. If you see any missing images or files, we would need to look at this but again, we only directly support Amazon S3.

Right I understand regarding support, I'm trying to understand the scope of change.  Was hoping there was a master script or something to that effect I could review and peel through the spot check.

The problem with discovery is I'm reliant on someone reporting it, which members (and certainly guests) won't do but could have an influence on content and visits long term. 

Are any logs generated during the external storage move that I could refer to and review for errors (i.e. would have identified these files that weren't successfully moved and I'm now manually moving)?

[Jim M]

I'm afraid, there isn't a log or any means really to perform what you're wanting to do here. 

[Clover13]

2 minutes ago, Jim M said:

I'm afraid, there isn't a log or any means really to perform what you're wanting to do here. 

OK brute force it is! 🙂

Maybe a good enhancement to add logging and support re-triggering/re-sync in the future given the potential scope of such a move.

Thanks for your prompt feedback! 👍

[Jim M]

14 hours ago, Clover13 said:

Maybe a good enhancement to add logging and support re-triggering/re-sync in the future given the potential scope of such a move.

Thanks for your prompt feedback! 👍

I will pass this on internally but with any feedback/feature requests, please post in our Feature Suggestions forum so this way we can keep track of this as a community with other input and not lose it in a support request 🙂 .

You're very welcome!

[Clover13]

Hey @Jim M, one follow-up question.

When IPS moves files from a given server-sided folder to external (S3), it seems like the server-sided folder remains (with an index.html) file in it.  Does this file need to be kept for any reason or can all of that be cleaned up as well (i.e. deleting all the /uploads/monthly_* folders once all media under them has been moved)?

[Jim M]

5 minutes ago, Clover13 said:

When IPS moves files from a given server-sided folder to external (S3), it seems like the server-sided folder remains (with an index.html) file in it.  Does this file need to be kept for any reason or can all of that be cleaned up as well (i.e. deleting all the /uploads/monthly_* folders once all media under them has been moved)?

If you've moved over all media, the /uploads/monthly_* folders are not needed any more and it can be deleted.

[Clover13]

@Jim M

So one strange observation for Gallery is that it seems to use both /uploads/gallery/album* and /uploads/monthly* for images.  At least going up to 06/2021 in my file sets.  How I discovered this is moving Gallery to external storage (along with other /uploads/monthly* folders/files) and then bringing Gallery back to server-sided storage and suddenly a bunch of monthly folders were recreated with various filenames ({someHexValue}.jpg, small.{someHexValue}.jpg, large.{someHexValue}.jpg).  Was this something Gallery used to do before but later shifted to /uploads/gallery/album/*?

[Clover13]

Appears the following happens:

• when uploaded to a gallery category, the image is saved to /uploads/monthly-{mm}-{dd}
• when uploaded to a gallery category's album, the image is saved to /uploads/gallery/album-{album-id}