Clover13

I do see many of the same files in both locations (S3 and local). S3 also has a .gz version of the file that is not on local. Example from S3 And local:

Overall it seems like there are quite a few duplicates because the deletions did not succeed (despite the log messages indicating it). It's hard to tell what is active without querying all the IPS tables to try and find defined links. What's the best way to determine what is used versus unused to verify proper link references (local vs S3) when it comes to "Theme Resources" in File Storage?

Nothing in the server logs. I set up another S3 File Storage configuration to point to a bucket path of "themes". Then I moved to the themes/JS/CSS to that new (empty) S3 config. Then I moved them back from S3 to local again. The move to S3 broke my theme and the page wouldn't render properly (missing artifacts), but bringing it back to local restored it. There are no errors in the Logs for Amazon for that configuration, only Log type messages. Not sure where to go from here, at this point I want my files back on local and off Amazon but not sure if I can simply copy them back. The fact that the files moved to a new S3 config and broke the theme makes me believe some parts of the DB update still point to local despite the files being moved to S3 (CDN URL path). Then when I move back to local, they seem to be referenced correctly.

Sorry my fault, I had no idea that X would do anything more than simply close out the window in the active browser window/tab. Hard to tell there that the X is a "clear" and not just a "close" popup.

Turns out this was even easier 🤦‍♂️ Just click the X to the right of the popup at the bottom right of the screen that has the number of Quoted Posts selected. That clears them all out. Note you can multi Quote posts across topics and that list gets carried around. Seems a bit odd, as you can selected quoted posts from one topic and use them in another topic where they wouldn't have any context. Not sure if that's by design or not.

@Marc Stridgen sorry for the delay in getting back to this. I was able to test it again and when I moved from local to S3, there were files remaining in local storage. When I moved from S3 back to local again, files still remained on S3. I see no indication of errors in the System Logs or Error Logs. Is there anywhere else there might be logging to review?

Is there a non-technical "easy" mode for doing this? I have to explain to a non-technical member, once I get into Dev Tools, they're going to be lost in the sauce.

A member must have been accidentally multi quoting posts and hit the limit. Now they don't know where the quoted anything. Is it possible to clear their multi quote selection? Seems to span multiple topics.

To be clear, as I was one who brought up the security concerns in this topic, I absolutely was not referring to any known instances or developer here, but merely the potential for it to occur and increasingly so when IPS becomes 100% hands off with third party applications. Apologies if it came across that way. I've seen enough security exploits in my own career (not anything IPS or IPS third party related) to warrant the concern. Again not a reflection of any developer here or the quality of their code, I'm simply proactively thinking about the possibility and considerations regarding preventative measures. If anyone feels such a security concern is completely unnecessary or overkill, I would appreciate your particular insight as to why. I certainly don't know the underpinnings of IPS code, so perhaps there is a reason a client doesn't need to have an elevated concern over it. I'd still like to know what IPS corporate customers do, if it's anything like the corporations I've worked with (unrelated to my IPS projects), there is full fledged InfoSec and AppSec scanning of all application code before any deployment with Production (real user/member) data. Generally for a hobby site, I'm not very concerned with data loss (with regular backups available to restore as needed), but I am concerned about data breaches involving PII.

There are automated tools that can be run against code bases to scan for vulnerable code and libraries. Reputation and trust alone are not enough when it comes to software. Developers with the best intentions can still unknowingly use something they are unaware of has an exploit (in deep dependencies, it can be something they aren't even using directly, but is included in their code base). It's simply good practice to run scans (code, infrastructure, app testing, etc) regularly to ensure security. Doesn't mean there's no risk, but it does greatly reduce it.

Chargebacks and tickets are the least of their concern. Legal issues jump right to the front of the line. Again, this could be an added reason for them to effectively exit all responsibility by closing their Marketplace. They reduce their own cost of maintaining it and eliminate their own risk (as noted in their own notes in the Provider directory).

I think that's quite a bad gap to have. With PII on the line, it's important to check for any potential vulnerabilities that could put it at risk. IPS clearly has a security and PII focus in their own code (right?), meanwhile you're saying they blindly host apps/plugins in their own Marketplace with the only requirement being not exhibiting any IP theft, while at the same time facilitating the sale of unchecked code? Why is that a good thing to not be their responsibility? Clearly, IPS is moving entirely away from any level of responsibility (IMO a case could be made they do indeed have responsibility in their current Marketplace). Now, how much of an issue this WILL be is debatable. How much of an issue it CAN be is not. Third party code is a risk and always will be. We can rely on trusted developers, which is great for those who exist already and have built up such trust. It's quite bad for new developers trying to enter the game and raise the level of competition and quality in applications. We are still taking risks using unchecked third party code. I am trusting the IPS Marketplace, reviews, etc on my own sites currently, but it certainly doesn't make me feel warm and fuzzy inside knowing it hasn't been vetted at a code/exploit/security level. It's more of a "I kinda need this functionality and sure hope nothing bad happens" and that honestly isn't a good mode of operation. I'd be curious to know what IPS has observed with their corporate clients. Yes they are built in house or by private consultants, but that doesn't mean it's immune from human error, outdated libraries, exploits, etc. Are they doing any level of suite wide testing to guarantee their own security?

Nothing that I can see. I can try to push the themes back to S3 and then back to local again if you'd like to re-test?

Too late! 😬 Images just weren't presenting well on desktop even though they fit, it wasn't a very fluid viewing experience because they were too big. Many also had a max quality setting, which was impacting page load times too much. So downsizing them to a better size and quality was needed. I'll just have to address any issues that may come up. Worst case I guess will be scripting something to query posts, parse images, read image properties, calc width/height and ratios, then update the forums_posts.post data. Not something I want to do if I don't have to though 🙂 Moving forward things should be all good on new uploads. CLS seems OK after the resizing and custom CSS, testing against a Topic page that had many resized images. Desktop CLS: 0.045 Mobile CLS: 0.022 FWIW, I used ImageMagick's mogrify to do the resizing and quality changes, which is the same as I have configured in IPS now.

Yes, the files were recopied back down locally via IPS automatically when I switched the storage option back but the S3 files were not removed. The IAM group and user under it have AmazonS3FullAccess as part of the Configuring Amazon S3 guide here. At the bucket level, I see only the following ACL: Grantee: Bucket owner Objects: List, Write Bucket ACL: Read, Write There is no mention of an explicit delete option in the guide. The API uses the user's access key, which should have delete via AmazonS3FullAccess right?

Not possible via IPS, but an external process can be used. I'm just trying to figure out what else has to change to get things working as they should. I'll test out the lazy loading and page jumping impact to CLS with my current solution.

FWIW, my current solution to this is to drive intrinsic sizing via custom CSS, while leaving the DB data alone. So the original width of a resized thumbnail may be stored in the DB with a larger value as part of the IMG element in the HTML, but once I resize (downsize) the actual image, I can force a max-content width and it will display correctly. One would probably never upsize an existing image due to pixelation. Original image before resizing is 900x1200. It is stored in the DB with a width="900" (as seen in the code example below). Resized image is 720x960 which is done via an external process, but the DB has not been updated (and may not need to be). Custom CSS: /* Override the original thumbnail image sizing to use the intrinsic sizing due to the static width defined and stored in the DB as HTML during the thumbnailing process */ .ipsType_richText img.ipsImage.ipsImage_thumbnailed { width: max-content !important; } Original IMG element generated from thumbnailing process and stored as HTML in the DB: <img class="ipsImage ipsImage_thumbnailed" data-fileid="11639" data-ratio="133.33" width="900" alt="IMG_7522.thumb.jpeg.2519c5354c6b4b35301e60eadcf52944.jpeg" data-src="//cdn.domain.com/monthly_2023_07/IMG_7522.thumb.jpeg.2519c5354c6b4b35301e60eadcf52944.jpeg" src="//cdn.domain.com/monthly_2023_07/IMG_7522.thumb.jpeg.2519c5354c6b4b35301e60eadcf52944.jpeg" style="height: auto;" data-loaded="true"> Without the CSS, the image will have an intrinsic size of 720x960 but display with a rendered size of 900x1200. With the CSS, the image displays as 720x960.

What is the purpose of these and what is the impact of removing them from the forums_posts.post data? They are contained within the forums_posts.post data, and then used for the HTML rendered in the image display of an attachment in a forum Topic. These data-ratio and width values appear to be created during thumbnail creation process and correlate to the AdminCP Posting settings for image sizing at the time of the thumbnail creation, however I am not sure what their purpose is. These reflect what the original image was resized into as a thumbnail, but the image itself already has those preferred size settings intrinsically. Why does it also need to be statically embedded within the HTML itself (and also statically stored in the DB)? The reason I ask is because in the event you want to revise your image sizing and rebuild your thumbnails, all of those forums_posts.post rows will need to also be recalculated and updated. If not and you resize (downsize) the thumbnails but leave the DB as-is, per what I've observed, then the rendered size of the image will be larger than the intrinsic size of the image and forced to display that rendered sizing (i.e. enlarged) based on the IMG width property. Without the IMG width property (being in the forums_posts.post HTML), it would display the image's intrinsic value which matches the configured value in the AdminCP settings.

Yes, both moves done via IPS AdminCP.

There appear to be files under the following directories specifically: css_built_0 css_built_2 javascript_calendar javascript_cms javascript_core javascript_forums javascript_gallery javascript_global javascript_ignoretopics javascript_rsvpevents javascript_tthumb set_resources_0 set_resources_2 set_resources_8 Some of these are third party, but even the core ones were not removed once I reconfigured to local. Examples:

Hi @Jim M, I added Notes to the Client Area providing the locations and details.

Yes, all complete and cache cleared a number of times since the move back was made. I was working on something else when I noticed the residual files today. So it's been quite a few days of them being there.

I was doing some testing of Theme Resources locally vs on S3. I changed back to local but see the theme related files still out on S3. Should they not have been removed/cleaned up on the move back? Namely: css_built_* javascript_* set_resources_*

The problem is, without the current IPS scan/approval process of apps/plugins, any new development is a risk. We also don't know how many iterations of scan/approval a given version of a given app had to go through to get final IPS approval, nor what those rulesets/barriers were for good practice per IPS standards. We just know the end product from the Marketplace dev. Now clients are subject to the intermediate iterations and any issues they expose. This is particularly concerning when we get into PII and any level of security risk to our sites (which we had a confidence level IPS was protecting us from with their scan/approval process).

Right, so this bodes well for well known and established devs as they have already created a foundational trust model with clients. For new devs, that's a barrier they'd have to create over time. Meanwhile, clients either have no way to validate new devs work like IPS previously did to guarantee the safety of the app/plugin. I think this greatly elevates the risk for clients and subsequently harms the potential for developers to grow the product. Hobbyist sites will suffer the most as they simply don't have the resources to invest in robust security evaluation. Perhaps another opportunity for a dev to provide some level of AppSec and InfoSec scanning of applications to lower the risk.