Abandoned Attachments

[SJ77]

 I am skeptical that every file that enters the folder via the system gets tracked correctly and removed.

I can see multiple versions of same file in my folders where failed upload attempts have happened.

[Joel R]

1 minute ago, SJ77 said:

 I am skeptical that every file that enters the folder via the system gets tracked correctly and removed.

I can see multiple versions of same file in my folders where failed upload attempts have happened.

If you have those examples, you need to send them in to Support!  They need clear examples to identify a system problem.  

[Phillyman]

I am going thru this right now, I feel as though my IP Gallery is bloated. Why?

So back in this blog post.....we got this information.

Which says the following....

Quote

Firstly, we have adjusted the software to only store two copies of an image (in addition to the original), instead of four. In previous versions, we stored a thumbnail, a small copy, a medium copy and a large copy of an image, all of which arbitrarily sized and designed to best meet our layout needs without showing an image too large or too small in a given space. We have simplified this vastly by storing a slightly larger "small" image, and storing a large copy. Diskspace usage is reduced dramatically as a result, and bandwidth usage is actually lowered as well since only two copies of an image need to be delivered to the browser instead of four.

 

Ok cool, so back a few versions ago we had Thumb, Small, Medium, Large and Original Gallery images. Now we are going to Small, Large, Original. So I should not see Medium or Thumb versions stored in my datastore location that has all my IP Gallery Images......so I download that directory and do a search....

 

Over 7000+ Thumb versions still exist....Ok lets look for Medium versions.......

And that yields over 10,000+ extra files. So between these two queries I have calculated about 2GB (10% of my total website size) of files that don't seem to need to exist. Also the way everything is sorted is just odd and I can't make heads or tails of it.

Some of my Gallery Images are stored nicely in a folder like....

• monthly_2018_09

While others are stored in a folder structure like this...

• gallery/category_175/

[Joel R]

Send in a ticket that identifies these files.  

Also, are you running 4.3.X?  

[bfarber]

I wish it were as easy as it may seem on the surface.

Nothing prevents you from manually uploading files to the folders we automatically generate. What if we deleted a file that you manually uploaded via FTP, because we went the route of "just delete everything we don't recognize"? What if third party addons store files inappropriately (it happens often actually - they incorrectly reuse our editor and file extensions instead of creating new ones) and we delete them? It's not so simple.

As for the files with "medium" in the filename (as an example), it's entirely possible those are still in use, or that they belonged to images long ago deleted from the database and were orphaned at that time. I couldn't say for sure without looking at your site.

So, as has been mentioned a few times - if you believe you have identified files that were left on the filesystem but should have been deleted, we encourage you to submit a ticket so we can investigate as to why. There could be bugs causing this, or it could be legacy files from older versions or older bugs since resolved. But if you don't submit a ticket and there is a bug present, we wouldn't know or be able to resolve it. 🙂 

[Joel R]

Don't ever abandon us @bfarber

We have attachment issues.  

Get it?

🤣

[Phillyman]

1 hour ago, bfarber said:

I wish it were as easy as it may seem on the surface.

Nothing prevents you from manually uploading files to the folders we automatically generate. What if we deleted a file that you manually uploaded via FTP, because we went the route of "just delete everything we don't recognize"? What if third party addons store files inappropriately (it happens often actually - they incorrectly reuse our editor and file extensions instead of creating new ones) and we delete them? It's not so simple.

As for the files with "medium" in the filename (as an example), it's entirely possible those are still in use, or that they belonged to images long ago deleted from the database and were orphaned at that time. I couldn't say for sure without looking at your site.

So, as has been mentioned a few times - if you believe you have identified files that were left on the filesystem but should have been deleted, we encourage you to submit a ticket so we can investigate as to why. There could be bugs causing this, or it could be legacy files from older versions or older bugs since resolved. But if you don't submit a ticket and there is a bug present, we wouldn't know or be able to resolve it. 🙂 

 

I wish it were easy too! 😛

Nothing does prevent me from uploading a stray file into my IP Gallery datastore location, but if I do and the software deletes it because of house keeping, I bet I will learn my lesson and create a directory for oddball stuff

With the medium and thumb gallery images, I am trying to figure out if I can delete them. I am that IPB end user that has been with you guys for 13 years, I know just enough to be dangerous . So I went into my database to see if I could find where these calls to the ORIG, LARGE, and SMALL versions are being called. I believe I found them in this table and with these fields. I am not sure if this is the ONLY place image locations are defined.

SELECT `image_id`,`image_caption`,`image_masked_file_name`,`image_file_name`,`image_original_file_name`,`image_small_file_name` FROM `ibf_gallery_images` 

So that command gives me back this CSV, and I will just post one line .....

• 123
• Nintendo Power Issue 010 (January/February 1990)
• monthly_2018_09/large.gallery_3_81439.jpg.3036d908e052b23282e07556f969f32f.jpg
• np-9001.jpg
• 1253818241/gallery_3_81439.jpg
• monthly_2018_09/small.gallery_3_81439.jpg.20c859a49d9bf00ff7975267ab398e1f.jpg

Now here is the kicker, I downloaded my IP Gallery datastore, and I can find only the item in green, not red. So I am confused why I can't find the large.gallery or small.gallery files, even though they are defined, I am wondering if these somehow still point back to the old medium and small versions. I thought that the image_file_name was only in the database to rename the file back when it is downloaded again, but the response I just got from that ticket suggests otherwise. If I could figure out what images are called, I could write a script to tell me exactly what files are not being referenced and delete them from my FTP (after backing up).

 

[AndyF]

15 hours ago, Joel R said:

Don't ever abandon us @bfarber

We have attachment issues.  

Get it?

🤣

I see what you did there

[bfarber]

17 hours ago, Phillyman said:

 

I wish it were easy too! 😛

Nothing does prevent me from uploading a stray file into my IP Gallery datastore location, but if I do and the software deletes it because of house keeping, I bet I will learn my lesson and create a directory for oddball stuff

With the medium and thumb gallery images, I am trying to figure out if I can delete them. I am that IPB end user that has been with you guys for 13 years, I know just enough to be dangerous . So I went into my database to see if I could find where these calls to the ORIG, LARGE, and SMALL versions are being called. I believe I found them in this table and with these fields. I am not sure if this is the ONLY place image locations are defined.

SELECT `image_id`,`image_caption`,`image_masked_file_name`,`image_file_name`,`image_original_file_name`,`image_small_file_name` FROM `ibf_gallery_images` 

So that command gives me back this CSV, and I will just post one line .....

• 123
• Nintendo Power Issue 010 (January/February 1990)
• monthly_2018_09/large.gallery_3_81439.jpg.3036d908e052b23282e07556f969f32f.jpg
• np-9001.jpg
• 1253818241/gallery_3_81439.jpg
• monthly_2018_09/small.gallery_3_81439.jpg.20c859a49d9bf00ff7975267ab398e1f.jpg

Now here is the kicker, I downloaded my IP Gallery datastore, and I can find only the item in green, not red. So I am confused why I can't find the large.gallery or small.gallery files, even though they are defined, I am wondering if these somehow still point back to the old medium and small versions. I thought that the image_file_name was only in the database to rename the file back when it is downloaded again, but the response I just got from that ticket suggests otherwise. If I could figure out what images are called, I could write a script to tell me exactly what files are not being referenced and delete them from my FTP (after backing up).

 

That is where the image URLs are stored, and the image files referenced should exist on disk.

"image_file_name" was the original filename, and won't exist but the other files should. If you just recently upgraded, it's possible you have background tasks rebuilding the images, although they're referencing a monthly_2018_09 folder leading me to believe the task already ran and wrote the images to that folder.

[SJ77]

22 hours ago, bfarber said:

I wish it were as easy as it may seem on the surface.

Nothing prevents you from manually uploading files to the folders we automatically generate. What if we deleted a file that you manually uploaded via FTP, because we went the route of "just delete everything we don't recognize"? 🙂

This is exactly what I want. Anyone loading their own files manually to a system folder should expect this.  I would settle for optional features to have this behavior as I really need A LOT of junk cleaned up. I could never do it all manually.

 

[Phillyman]

2 hours ago, SJ77 said:

This is exactly what I want. Anyone loading their own files manually to a system folder should expect this.  I would settle for optional features to have this behavior as I really need A LOT of junk cleaned up. I could never do it all manually.

 

[Joel R]

On 9/24/2018 at 1:36 PM, bfarber said:

Nothing prevents you from manually uploading files to the folders we automatically generate. What if we deleted a file that you manually uploaded via FTP, because we went the route of "just delete everything we don't recognize"? What if third party addons store files inappropriately (it happens often actually - they incorrectly reuse our editor and file extensions instead of creating new ones) and we delete them? It's not so simple.

1. To be clear, this is for files specifically within the /uploads folder.  This isn't a general web folder.

2.  The only files in my /uploads folder should be ones used specifically by my Invision community, yes? Otherwise they wouldn't be in there.  Even if you manually upload media through Pages, theyre still tracked in there.  

3. If a third party Dev incorrectly stores files, then I would yell at him for being a terrible coder and write an appropriate review.

[Loren]

28 minutes ago, Joel R said:

1. To be clear, this is for files specifically within the /uploads folder.  This isn't a general web folder.

2.  The only files in my /uploads folder should be ones used specifically by my Invision community, yes? Otherwise they wouldn't be in there.  Even if you manually upload media through Pages, theyre still tracked in there.  

3. If a third party Dev incorrectly stores files, then I would yell at him for being a terrible coder and write an appropriate review.

 

To your point 1. - I have many files in the uploads folder and IPS put them there in IPS 1.0.0.
So, I do not think it is true (for aged IPS installs) to say that there should be no files there.

 

[bfarber]

Yes, unfortunately assumption 1+2 just can't be relied upon. Nothing stops the admin from opening the folder up in FTP and uploading a file (I used to do it all the time to store files I would link to but didn't want to clutter up my regular webspace for). And as Loren mentioned, when you factor in much older installations, conversions and so on, things become even trickier.

[Phillyman]

1 minute ago, bfarber said:

Yes, unfortunately assumption 1+2 just can't be relied upon. Nothing stops the admin from opening the folder up in FTP and uploading a file (I used to do it all the time to store files I would link to but didn't want to clutter up my regular webspace for). And as Loren mentioned, when you factor in much older installations, conversions and so on, things become even trickier.

What is the IPB recommended method for cleaning up our sites? I know when I look in my database, that I see IPB Arcade, IPB Casino, IPB Battle systems from back when I was running IPB 2.x. Is there a comparison tool that could point out non default tables and fields for us to evaluate? Same with IPB files, is there any tool that could run a comparison and tell me what differences there are between a default install and my file structure?

[bfarber]

There's no real one size fits all that we recommend, but there are options. Either way, we're venturing outside of "feedback" getting into discussions of how to clean up your site.

[Ryan Ashbrook]

9 hours ago, Joel R said:

2.  The only files in my /uploads folder should be ones used specifically by my Invision community, yes? Otherwise they wouldn't be in there.  Even if you manually upload media through Pages, theyre still tracked in there.

In theory, yes. In practice, no. I have come across several clients who actively use their file storage for much more than just their community. Realistically, we cannot assume that the contents of a particular container (that container being anything between a specific folder In the file system, or a bucket on Amazon S3) only contains files uploaded by the suite.

In this case, it's better to be safe than sorry.

However - if you believe the suite is not removing files as they are deleted, please submit a ticket. We will be more than happy to look and fix the problem.

[RObiN-HoOD]

I know I have extra files that I don't even remember now from where they are...

The best solution I guess is to create a script (either run from ACP or shell) that will output suggested files not-in-use in IPB and then admin do whatever sees fit.

[SJ77]

8 hours ago, Ryan Ashbrook said:

However - if you believe the suite is not removing files as they are deleted, please submit a ticket. We will be more than happy to look and fix the problem.

Hi just to be clear, the suite might be removing "as intended", the issue is the intended method leaves a lot of junk in the folders. I really need an option to clean up the attachment folders. Right there there isn't any way to do it at all. 

[Joel R]

2 hours ago, SJ77 said:

Hi just to be clear, the suite might be removing "as intended", the issue is the intended method leaves a lot of junk in the folders. I really need an option to clean up the attachment folders. Right there there isn't any way to do it at all. 

You need to identify three or more files that should be removed and to make a support ticket for IPS to confirm the pattern.  We're all in agreement that files tracked by IPS that were not removed and should be removed is an issue, and those need to be escalated via a ticket.  

Whether or not IPS feels like it's their responsibility to control the /uploads folder is debatable.  To me, it's a system folder controlled by the software, but I can also understand the reluctance to not impetuously delete non-IPS uploads.