Clover13 Posted August 16, 2023 Share Posted August 16, 2023 I was doing some testing of Theme Resources locally vs on S3. I changed back to local but see the theme related files still out on S3. Should they not have been removed/cleaned up on the move back? Namely: css_built_* javascript_* set_resources_* Link to comment Share on other sites More sharing options...
Jim M Posted August 16, 2023 Share Posted August 16, 2023 Are the background tasks complete? Did you clear your cache in ACP -> Support? Link to comment Share on other sites More sharing options...
Clover13 Posted August 16, 2023 Author Share Posted August 16, 2023 37 minutes ago, Jim M said: Are the background tasks complete? Did you clear your cache in ACP -> Support? Yes, all complete and cache cleared a number of times since the move back was made. I was working on something else when I noticed the residual files today. So it's been quite a few days of them being there. Link to comment Share on other sites More sharing options...
Jim M Posted August 16, 2023 Share Posted August 16, 2023 Do you have an example? Looking at your community, largely, these items are on your local server. I do see that your Icons for Web Apps are on a separate sub-domain. Please keep in mind that these have their own storage location of "Icons & Logos". Link to comment Share on other sites More sharing options...
Clover13 Posted August 16, 2023 Author Share Posted August 16, 2023 Hi @Jim M, I added Notes to the Client Area providing the locations and details. Link to comment Share on other sites More sharing options...
Jim M Posted August 16, 2023 Share Posted August 16, 2023 As I do not have access to your S3 bucket, I would need an example of a file you're looking for which has not been returned. Link to comment Share on other sites More sharing options...
Clover13 Posted August 16, 2023 Author Share Posted August 16, 2023 There appear to be files under the following directories specifically: css_built_0 css_built_2 javascript_calendar javascript_cms javascript_core javascript_forums javascript_gallery javascript_global javascript_ignoretopics javascript_rsvpevents javascript_tthumb set_resources_0 set_resources_2 set_resources_8 Some of these are third party, but even the core ones were not removed once I reconfigured to local. Examples: Link to comment Share on other sites More sharing options...
Jim M Posted August 16, 2023 Share Posted August 16, 2023 6 minutes ago, Clover13 said: javascript_ignoretopics javascript_rsvpevents javascript_tthumb These are third party so those may be managed by the respective author's storage. The rest though is odd that they did not get moved over. Was the original move done via the software as well? Link to comment Share on other sites More sharing options...
Clover13 Posted August 16, 2023 Author Share Posted August 16, 2023 2 minutes ago, Jim M said: These are third party so those may be managed by the respective author's storage. The rest though is odd that they did not get moved over. Was the original move done via the software as well? Yes, both moves done via IPS AdminCP. Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 17, 2023 Share Posted August 17, 2023 Please check permissions on your s3 bucket. From what you have said there, am I correct in saying that files were copied to local, but just not actually removed? That would lead me to believe it may be delete permissions that are an issue Link to comment Share on other sites More sharing options...
Clover13 Posted August 17, 2023 Author Share Posted August 17, 2023 6 hours ago, Marc Stridgen said: Please check permissions on your s3 bucket. From what you have said there, am I correct in saying that files were copied to local, but just not actually removed? That would lead me to believe it may be delete permissions that are an issue Yes, the files were recopied back down locally via IPS automatically when I switched the storage option back but the S3 files were not removed. The IAM group and user under it have AmazonS3FullAccess as part of the Configuring Amazon S3 guide here. At the bucket level, I see only the following ACL: Grantee: Bucket ownerObjects: List, Write Bucket ACL: Read, Write There is no mention of an explicit delete option in the guide. The API uses the user's access key, which should have delete via AmazonS3FullAccess right? Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 17, 2023 Share Posted August 17, 2023 Are you seeing anything in your system logs for the times you did this? Link to comment Share on other sites More sharing options...
Clover13 Posted August 17, 2023 Author Share Posted August 17, 2023 29 minutes ago, Marc Stridgen said: Are you seeing anything in your system logs for the times you did this? Nothing that I can see. I can try to push the themes back to S3 and then back to local again if you'd like to re-test? Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 18, 2023 Share Posted August 18, 2023 If you can do that, it woul;d be helpful, yes. Also please ensure that you backup everything first, before we test this Link to comment Share on other sites More sharing options...
Clover13 Posted August 28, 2023 Author Share Posted August 28, 2023 @Marc Stridgen sorry for the delay in getting back to this. I was able to test it again and when I moved from local to S3, there were files remaining in local storage. When I moved from S3 back to local again, files still remained on S3. I see no indication of errors in the System Logs or Error Logs. Is there anywhere else there might be logging to review? Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 29, 2023 Share Posted August 29, 2023 Your server error logs maybe Link to comment Share on other sites More sharing options...
Clover13 Posted August 30, 2023 Author Share Posted August 30, 2023 (edited) On 8/29/2023 at 4:25 AM, Marc Stridgen said: Your server error logs maybe Nothing in the server logs. I set up another S3 File Storage configuration to point to a bucket path of "themes". Then I moved to the themes/JS/CSS to that new (empty) S3 config. Then I moved them back from S3 to local again. The move to S3 broke my theme and the page wouldn't render properly (missing artifacts), but bringing it back to local restored it. There are no errors in the Logs for Amazon for that configuration, only Log type messages. Not sure where to go from here, at this point I want my files back on local and off Amazon but not sure if I can simply copy them back. The fact that the files moved to a new S3 config and broke the theme makes me believe some parts of the DB update still point to local despite the files being moved to S3 (CDN URL path). Then when I move back to local, they seem to be referenced correctly. Edited August 30, 2023 by Clover13 Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 30, 2023 Share Posted August 30, 2023 It may break during the move or if its finished and your site cant access them. If they are already there, you would be best to manually copy them over if you are having permissions issues (which is very much the most likely issue) Link to comment Share on other sites More sharing options...
Clover13 Posted August 30, 2023 Author Share Posted August 30, 2023 Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it). It's hard to tell what is active without querying all the IPS tables to try and find defined links. What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage? Link to comment Share on other sites More sharing options...
Marc Stridgen Posted August 30, 2023 Share Posted August 30, 2023 1 hour ago, Clover13 said: Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it). It's hard to tell what is active without querying all the IPS tables to try and find defined links. What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage? Im not sure what you mean by duplicates there. Files themselves would not be duplicated Link to comment Share on other sites More sharing options...
Clover13 Posted August 30, 2023 Author Share Posted August 30, 2023 1 hour ago, Marc Stridgen said: Im not sure what you mean by duplicates there. Files themselves would not be duplicated I do see many of the same files in both locations (S3 and local). S3 also has a .gz version of the file that is not on local. Example from S3 And local: Link to comment Share on other sites More sharing options...
Jim M Posted August 30, 2023 Share Posted August 30, 2023 The gz would only exist on S3. Your server should gzip the files as a best practice so we do not do that for the server file system. Clover13 1 Link to comment Share on other sites More sharing options...
Clover13 Posted September 1, 2023 Author Share Posted September 1, 2023 @Marc Stridgen any other ideas on how to clean things up? Since I see no failure/errors, I can't easily tell what is being referenced unless I remove things from S3 and see if anything breaks (as issues arise and get reported). Anything I can run (SQL, command line, etc) to identify and update any references? Link to comment Share on other sites More sharing options...
Marc Stridgen Posted September 4, 2023 Share Posted September 4, 2023 On 9/1/2023 at 7:18 PM, Clover13 said: @Marc Stridgen any other ideas on how to clean things up? Since I see no failure/errors, I can't easily tell what is being referenced unless I remove things from S3 and see if anything breaks (as issues arise and get reported). Anything I can run (SQL, command line, etc) to identify and update any references? There is no other way in which to achieve this, no Clover13 1 Link to comment Share on other sites More sharing options...
Recommended Posts