Jump to content

Recommended Posts

Posted

I was doing some testing of Theme Resources locally vs on S3.  I changed back to local but see the theme related files still out on S3.  Should they not have been removed/cleaned up on the move back?

Namely:

  • css_built_*
  • javascript_*
  • set_resources_*
Posted
37 minutes ago, Jim M said:

Are the background tasks complete? Did you clear your cache in ACP -> Support?

Yes, all complete and cache cleared a number of times since the move back was made.  I was working on something else when I noticed the residual files today.  So it's been quite a few days of them being there.

Posted

Do you have an example? Looking at your community, largely, these items are on your local server. I do see that your Icons for Web Apps are on a separate sub-domain. Please keep in mind that these have their own storage location of "Icons & Logos".

Posted

There appear to be files under the following directories specifically:

  • css_built_0
  • css_built_2
  • javascript_calendar
  • javascript_cms
  • javascript_core
  • javascript_forums
  • javascript_gallery
  • javascript_global
  • javascript_ignoretopics
  • javascript_rsvpevents
  • javascript_tthumb
  • set_resources_0
  • set_resources_2
  • set_resources_8

Some of these are third party, but even the core ones were not removed once I reconfigured to local.

Examples:

Could contain: Page, Text, File

Could contain: Page, Text, File

Posted
6 minutes ago, Clover13 said:
  • javascript_ignoretopics
  • javascript_rsvpevents
  • javascript_tthumb

These are third party so those may be managed by the respective author's storage.

The rest though is odd that they did not get moved over. Was the original move done via the software as well?

Posted
2 minutes ago, Jim M said:

These are third party so those may be managed by the respective author's storage.

The rest though is odd that they did not get moved over. Was the original move done via the software as well?

Yes, both moves done via IPS AdminCP.

Posted

Please check permissions on your s3 bucket. From what you have said there, am I correct in saying that files were copied to local, but just not actually removed? That would lead me to believe it may be delete permissions that are an issue

Posted
6 hours ago, Marc Stridgen said:

Please check permissions on your s3 bucket. From what you have said there, am I correct in saying that files were copied to local, but just not actually removed? That would lead me to believe it may be delete permissions that are an issue

Yes, the files were recopied back down locally via IPS automatically when I switched the storage option back but the S3 files were not removed.

The IAM group and user under it have AmazonS3FullAccess as part of the Configuring Amazon S3 guide here.

At the bucket level, I see only the following ACL:

Grantee:  Bucket owner
Objects:  List, Write
Bucket ACL: Read, Write

There is no mention of an explicit delete option in the guide.

The API uses the user's access key, which should have delete via AmazonS3FullAccess right?

Posted
29 minutes ago, Marc Stridgen said:

Are you seeing anything in your system logs for the times you did this?

Nothing that I can see.  I can try to push the themes back to S3 and then back to local again if you'd like to re-test?

  • 2 weeks later...
Posted

@Marc Stridgen sorry for the delay in getting back to this.  I was able to test it again and when I moved from local to S3, there were files remaining in local storage.   When I moved from S3 back to local again, files still remained on S3.  I see no indication of errors in the System Logs or Error Logs.  Is there anywhere else there might be logging to review?

Posted (edited)
On 8/29/2023 at 4:25 AM, Marc Stridgen said:

Your server error logs maybe

Nothing in the server logs.

I set up another S3 File Storage configuration to point to a bucket path of "themes".  Then I moved to the themes/JS/CSS to that new (empty) S3 config.  Then I moved them back from S3 to local again.

The move to S3 broke my theme and the page wouldn't render properly (missing artifacts), but bringing it back to local restored it.

There are no errors in the Logs for Amazon for that configuration, only Log type messages.

Not sure where to go from here, at this point I want my files back on local and off Amazon but not sure if I can simply copy them back.  The fact that the files moved to a new S3 config and broke the theme makes me believe some parts of the DB update still point to local despite the files being moved to S3 (CDN URL path).  Then when I move back to local, they seem to be referenced correctly.

Could contain: Page, Text, White Board

Edited by Clover13
Posted

It may break during the move or if its finished and your site cant access them. If they are already there, you would be best to manually copy them over if you are having permissions issues (which is very much the most likely issue)

Posted

Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it).  It's hard to tell what is active without querying all the IPS tables to try and find defined links.

What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage?

Posted
1 hour ago, Clover13 said:

Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it).  It's hard to tell what is active without querying all the IPS tables to try and find defined links.

What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage?

Im not sure what you mean by duplicates there. Files themselves would not be duplicated

Posted
1 hour ago, Marc Stridgen said:

Im not sure what you mean by duplicates there. Files themselves would not be duplicated

I do see many of the same files in both locations (S3 and local).  S3 also has a .gz version of the file that is not on local.

Example from S3

Could contain: Page, Text, File, Webpage

 

And local:

Could contain: Page, Text

Posted

@Marc Stridgen any other ideas on how to clean things up?  Since I see no failure/errors, I can't easily tell what is being referenced unless I remove things from S3 and see if anything breaks (as issues arise and get reported).  Anything I can run (SQL, command line, etc) to identify and update any references?

Posted
On 9/1/2023 at 7:18 PM, Clover13 said:

@Marc Stridgen any other ideas on how to clean things up?  Since I see no failure/errors, I can't easily tell what is being referenced unless I remove things from S3 and see if anything breaks (as issues arise and get reported).  Anything I can run (SQL, command line, etc) to identify and update any references?

There is no other way in which to achieve this, no

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...