Jump to content

Moving theme to S3 and back to local leaves files still on S3


Clover13

Recommended Posts

37 minutes ago, Jim M said:

Are the background tasks complete? Did you clear your cache in ACP -> Support?

Yes, all complete and cache cleared a number of times since the move back was made.  I was working on something else when I noticed the residual files today.  So it's been quite a few days of them being there.

Link to comment
Share on other sites

Do you have an example? Looking at your community, largely, these items are on your local server. I do see that your Icons for Web Apps are on a separate sub-domain. Please keep in mind that these have their own storage location of "Icons & Logos".

Link to comment
Share on other sites

There appear to be files under the following directories specifically:

  • css_built_0
  • css_built_2
  • javascript_calendar
  • javascript_cms
  • javascript_core
  • javascript_forums
  • javascript_gallery
  • javascript_global
  • javascript_ignoretopics
  • javascript_rsvpevents
  • javascript_tthumb
  • set_resources_0
  • set_resources_2
  • set_resources_8

Some of these are third party, but even the core ones were not removed once I reconfigured to local.

Examples:

Could contain: Page, Text, File

Could contain: Page, Text, File

Link to comment
Share on other sites

6 minutes ago, Clover13 said:
  • javascript_ignoretopics
  • javascript_rsvpevents
  • javascript_tthumb

These are third party so those may be managed by the respective author's storage.

The rest though is odd that they did not get moved over. Was the original move done via the software as well?

Link to comment
Share on other sites

Please check permissions on your s3 bucket. From what you have said there, am I correct in saying that files were copied to local, but just not actually removed? That would lead me to believe it may be delete permissions that are an issue

Link to comment
Share on other sites

6 hours ago, Marc Stridgen said:

Please check permissions on your s3 bucket. From what you have said there, am I correct in saying that files were copied to local, but just not actually removed? That would lead me to believe it may be delete permissions that are an issue

Yes, the files were recopied back down locally via IPS automatically when I switched the storage option back but the S3 files were not removed.

The IAM group and user under it have AmazonS3FullAccess as part of the Configuring Amazon S3 guide here.

At the bucket level, I see only the following ACL:

Grantee:  Bucket owner
Objects:  List, Write
Bucket ACL: Read, Write

There is no mention of an explicit delete option in the guide.

The API uses the user's access key, which should have delete via AmazonS3FullAccess right?

Link to comment
Share on other sites

  • 2 weeks later...

@Marc Stridgen sorry for the delay in getting back to this.  I was able to test it again and when I moved from local to S3, there were files remaining in local storage.   When I moved from S3 back to local again, files still remained on S3.  I see no indication of errors in the System Logs or Error Logs.  Is there anywhere else there might be logging to review?

Link to comment
Share on other sites

On 8/29/2023 at 4:25 AM, Marc Stridgen said:

Your server error logs maybe

Nothing in the server logs.

I set up another S3 File Storage configuration to point to a bucket path of "themes".  Then I moved to the themes/JS/CSS to that new (empty) S3 config.  Then I moved them back from S3 to local again.

The move to S3 broke my theme and the page wouldn't render properly (missing artifacts), but bringing it back to local restored it.

There are no errors in the Logs for Amazon for that configuration, only Log type messages.

Not sure where to go from here, at this point I want my files back on local and off Amazon but not sure if I can simply copy them back.  The fact that the files moved to a new S3 config and broke the theme makes me believe some parts of the DB update still point to local despite the files being moved to S3 (CDN URL path).  Then when I move back to local, they seem to be referenced correctly.

Could contain: Page, Text, White Board

Edited by Clover13
Link to comment
Share on other sites

Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it).  It's hard to tell what is active without querying all the IPS tables to try and find defined links.

What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage?

Link to comment
Share on other sites

1 hour ago, Clover13 said:

Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it).  It's hard to tell what is active without querying all the IPS tables to try and find defined links.

What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage?

Im not sure what you mean by duplicates there. Files themselves would not be duplicated

Link to comment
Share on other sites

1 hour ago, Marc Stridgen said:

Im not sure what you mean by duplicates there. Files themselves would not be duplicated

I do see many of the same files in both locations (S3 and local).  S3 also has a .gz version of the file that is not on local.

Example from S3

Could contain: Page, Text, File, Webpage

 

And local:

Could contain: Page, Text

Link to comment
Share on other sites

On 9/1/2023 at 7:18 PM, Clover13 said:

@Marc Stridgen any other ideas on how to clean things up?  Since I see no failure/errors, I can't easily tell what is being referenced unless I remove things from S3 and see if anything breaks (as issues arise and get reported).  Anything I can run (SQL, command line, etc) to identify and update any references?

There is no other way in which to achieve this, no

Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...