Abandoned Attachments

[Joel R]

I'm doing a deep dive into Attachments this morning, and I'm discovering quite a few attachments that are taking up unnecessary space that are no longer available because the primary content item (eg. PM) is deleted. 

I'd like to request the ability to identify attachments that have no location.

I can't imagine how many abandoned attachments are floating around in my community.  

[Joel R]

As a follow-up to this, I've identified quite a few files that I'm pretty sure are associated with a deleted PM.  If the primary content item is deleted, then the attachments should be deleted as well.  

[bfarber]

Yes, they should be. You shouldn't need to manually look for and remove orphaned attachments. You may wish to submit a ticket so we can investigate.

[SJ77]

THIS is something I have been worried about for a long time. I brought it up here a few times and everyone insisted that orphaned attachments are cleaned up automatically.

I have found they are not.

I have orphan attachments from downloads being uploaded part way and failing. I have orphans from posts that are removed. ETC ETC. I have been having to manually clean it up for a long time now.

[Joel R]

1 hour ago, bfarber said:

Yes, they should be. You shouldn't need to manually look for and remove orphaned attachments. You may wish to submit a ticket so we can investigate.

Well, I already manually deleted the attachments.  I'll see if I can reproduce.  

1 hour ago, SJ77 said:

THIS is something I have been worried about for a long time. I brought it up here a few times and everyone insisted that orphaned attachments are cleaned up automatically.

I have found they are not.

I have orphan attachments from downloads being uploaded part way and failing. I have orphans from posts that are removed. ETC ETC. I have been having to manually clean it up for a long time now.

Go to your Attachments list like I did and see if you can find a pattern (eg two or more files that were not properly deleted from the same type of content items.), so you can send in a ticket.  I need to do the same tonight.

[sound]

16 hours ago, SJ77 said:

THIS is something I have been worried about for a long time. I brought it up here a few times and everyone insisted that orphaned attachments are cleaned up automatically.

I have found they are not.

I have orphan attachments from downloads being uploaded part way and failing. I have orphans from posts that are removed. ETC ETC. I have been having to manually clean it up for a long time now.

same here

could be a legacy issue due to various changes/issues over the years?

a clean up script could be useful

[SJ77]

Here is how it really fails hard: It does not look in the folder for files that have no reference in the database.

Go into downloads app and upload a 1GB file but don't click save. Notice that in a few days no matter what you try and do in the IPB suite, if you check the folder using your SSH access or FTP, the 1GB is still there. This is a big problem for me.

On 8/13/2018 at 8:27 AM, bfarber said:

Yes, they should be. You shouldn't need to manually look for and remove orphaned attachments. You may wish to submit a ticket so we can investigate.

Doesn't work. Tested extensively on my side. certain this isn't working.

[bfarber]

The downloads manager stores the files in the database (even before they're saved) and then will delete any files not saved after 1 day via the task named tempFileCleanup. As I said, if this isn't working on your site, please submit a ticket and a support agent will be happy to take a look. :)

 

[Joel R]

14 minutes ago, SJ77 said:

Here is how it really fails hard: It does not look in the folder for files that have no reference in the database.

Go into downloads app and upload a 1GB file but don't click save. Notice that in a few days no matter what you try and do in the IPB suite, if you check the folder using your SSH access or FTP, the 1GB is still there. This is a big problem for me.

Doesn't work. Tested extensively on my side. certain this isn't working.

I think there are two potential gaps  of files.

1. Files that were uploaded (and captured by IPS) with no location.  

Do you have access to phpMyAdmin?

You can look into your table for attachments, gallery images, and download files to see if you have files not associated with locations.  

2. Files that were uploaded and never captured by IPS.  

What's even scarier is that we might have files that were never captured by the IPS software for whatever reason, and they're now forever floating in Amazon's cloud, haunting my monthly bill for the rest of time.  

 

I sent in a ticket on Monday night for my attachments, but I didn't think to look for gallery images or download files.  I also literally have no way of tracking down files not used or tracked by the software.  

[marklcfc]

Would be good to have a way to delete attachments no longer attached to a post as its been deleted

[Joel R]

58 minutes ago, marklcfc said:

Would be good to have a way to delete attachments no longer attached to a post as its been deleted

To clarify, that should already be happening.  If you notice attachments that aren't attached to a post, then send in a ticket.  

[sound]

On 8/16/2018 at 7:25 PM, Joel R said:

I think there are two potential gaps  of files.

1. Files that were uploaded (and captured by IPS) with no location.  

Do you have access to phpMyAdmin?

You can look into your table for attachments, gallery images, and download files to see if you have files not associated with locations.  

2. Files that were uploaded and never captured by IPS.  

What's even scarier is that we might have files that were never captured by the IPS software for whatever reason, and they're now forever floating in Amazon's cloud, haunting my monthly bill for the rest of time.  

 

I sent in a ticket on Monday night for my attachments, but I didn't think to look for gallery images or download files.  I also literally have no way of tracking down files not used or tracked by the software.  

1

in theory if you use the 'storage settings' in the acp to change the files storage location of the app in question  - after the files have been  'moved' to the new location all 'unrecorded' files should be left in the 'old' directory...

[Circo]

On 8/17/2018 at 12:00 PM, sound said:

in theory if you use the 'storage settings' in the acp to change the files storage location of the app in question  - after the files have been  'moved' to the new location all 'unrecorded' files should be left in the 'old' directory...

Has anyone tried this, I have a ton of bloat in my download section which has tens of thousands of files in it.  Support is convinced that no tool is needed.  If anyone has a solid solution to this bloat problem I would gladly pay if needed.

[AndyF]

The issue with 'cleanup' tools is (as experienced in previous major versions) there's the potential to accidentally delete things that you wanted to keep or very rarely for there to be a problem which causes unwanted effects. For a boring minor history lesson those who have been using previous major versions may / may not of noted the legacy 'rebuild posts' tool was not present in the last few versions of that series, for a good reason. It was safe used as intended however careless use could potentially cause issues.

I can see both sides of this, in that an 'attach clean' tool would work in some cases however there is as I say the potential for misuse and that could cause Support issues if someone wonders why their data went awol, think along the lines of how the previous version used to have a mass prune feature on the front end, the potential for accidents or a moderator not being 100% sure what they were doing, well...

Having said that previous versions did have a few cleanup tools, I am just going to include a bit of a screenshot of the attach part of it, however if such a tool was needed in the IPS4 Suite no doubt one would of been provided anyway as the software should 'take care of this' for you hence 'no tool provided or needed'

[Circo]

I just got done trying to talk to Marc via a support ticket.  He did say the moving directory idea would work.  So I guess that’s our official solution.  

[AndyF]

5 minutes ago, Circo said:

I just got done trying to talk to Marc via a support ticket.  He did say the moving directory idea would work.  So I guess that’s our official solution.  

Well I guess 'move' is a lot safer than 'delete' 🙂

[Joel R]

2 hours ago, Circo said:

I just got done trying to talk to Marc via a support ticket.  He did say the moving directory idea would work.  So I guess that’s our official solution.  

I still think you should identify the abandoned attachments and provide eaxmples to IPS to see how they weren't cleaned up as part of their routine.  Even though IPS says they remove abandoned attachments, they miss items.  

For example, I have a ticket where I identified a group of attachments from deleted PM's.  

[jair101]

3 hours ago, Circo said:

I just got done trying to talk to Marc via a support ticket.  He did say the moving directory idea would work.  So I guess that’s our official solution.  

Keep in mind that switching the storage settings does not move the files, but does copy+delete. Hence, make sure you have enough disk space for 2x the current upload folder. Otherwise it will get VERY unpleasant. 

That was my experience when I tried the same move on my dev board. 

[Circo]

IPS support was little to no help, 9 years in and first level help is strictly there to frustrate you enough to go away.  I’ve only had results when I get 2nd tier techs or better.  But anyway I’m just glad there is any kind of solution. I would still love a tool to audit.  Sure automation is great but if anything ever goes wrong or gets missed your stuck with bloat.  I just wish my downloads folder wasn’t 500gb.  Come to think about it I may not have enough space... ever wish you got started in a different ecosystem?

[Joel R]

You can set up an Amazon S3 account and shift your files there.  

[Circo]

I meant a different system than IPB, I was about to start the move until I read Jair101’s post.  So back at square 1.  I may just have to setup a local test environment and attemp the cleanup there.  I’ll watch this thread just Incase there is a better solution.

[SJ77]

On 8/16/2018 at 11:13 AM, bfarber said:

The downloads manager stores the files in the database (even before they're saved) and then will delete any files not saved after 1 day via the task named tempFileCleanup. As I said, if this isn't working on your site, please submit a ticket and a support agent will be happy to take a look. 🙂

 

Therein is the problem.

I think it should scrub the folder for ALL files in the folder regardless of what is in the database, and then cross reference and remove orphan stuff.

[SJ77]

What is confusing to me is why support insists that the current solution is working when MANY folks who check can see that there is loads of stuff that should NOT be in the attachments folder.  Can't we just align on the obvious, and recognize that the current solution is insufficient? Only then will a proper solution be developed. It's not meant to be in insult or personal attack to IPS. We just want clean folders. 

[Joel R]

5 minutes ago, SJ77 said:

Therein is the problem.

I think it should scrub the folder for ALL files in the folder regardless of what is in the database or not and then cross reference and remove orphan stuff.

I agree with @SJ77.  

The fear is not known attachments that are abandoned.  There's also unknown attachments that are abandoned.  

[Joel R]

One of my fears with abandoned attachments is that IPS is only looking at cleaning known system files during the upload process.  Sure, that makes sense and you clean them up one day later.  But what about other files that get added through *some* manner or that aren't marked for deletion. 

For example:

- all of the default avatars 

- third party stuff