Cleaning tools: How remove all orphan file (recursively) ?

[NoSpy]

Hello,

I noticed that there are many photos that are no longer attached to topics or blogs yet they still occupy disk space. How remove all orphan file (recursively) Disk & DB ?

[Marc Stridgen]

There is no way in which to automatically do this, unfortunately. Files will be removed once they are no longer in use. Are these older files you are referring to?

[DawPi]

3 minutes ago, Marc Stridgen said:

Files will be removed once they are no longer in use.

Removed by what exactly? Task?

What about other scenarioo - someone uploaded "few" files into uploads/ directory and it's "merged" with current files being used. How to recognize these uploaded and deleted them recursively as OP asked?

[Marc Stridgen]

If someone has manually uploaded to the directory, there is no way in which to know this.

When an item is deleted on the community, its deleted at that point if its not being used on any other resource. For example, if you delete a topic with images, those images are deleted if they have not been attached elsewhere using the "Use existing attachment" function.

The reason I ask if these are older files, is there were bugs a long long while ago where these were not being removed effectively

[DawPi]

Sure. Thanks for clarification. I need to write custom mod then. 

[NoSpy]

Hello Marc Stridgen,

We had an issue with comments associated with outrageous images submitted by dishonest users who hired a lawyer to seek financial compensation. That's why this feature is interesting:

• https://invisioncommunity.com/forums/topic/473859-feature-hash-regeneration-hash-dynamic/

Since trying to understand how they manage to display images for topics that have been deleted. Many avenues have been explored (SCAM). We are not in a world of care bears and that there are people who exploit this security flaw to get rich. This is also why on Facebook, YouTube and co, when there is a subject that would violate copyrights, they delete it without trying to find out if it's true or not.

Website is closed yet crawlers are downloading images without loading website content. By analyzing their interactions, we discovered that there are images that have not been deleted and yet they are not associated with any topic.

 

This required me to manually delete them from the CPAdmin 6 weeks of work to identify them. On other platforms, there is a tool to do this comparison. For example, Mamoto has a feature that checks integrity by looking for files that shouldn't exist or have been modified. It seems to me that in earlier versions of invision there was this feature.

How remove all orphan file (recursively) Disk & DB ?

[Randy Calvert]

There is no “orphan” file checker. 

[NoSpy]

If I click (ƒi) to identify what they are associated with, I get an red error.

• If I delete them from the CP Admin nothing happens on the storage device.
• If I delete them on the storage device I have no error message in the CPAdmin.

In other cases there was an image and again no subject was associated.

[Marc Stridgen]

The simplest method to achieve this en-masse would be to move your storage to another location. You would need to do a full backup of both files and database first, and be prepared to revert if needed. But essentially you would

• Create a new storage configuration. Say "uploads2"
• Move all items to that storage location

Whatever is left, are items not being used. So whatever is left in uploads (if thats the location you are using) should be able to be deleted. You can then move everything back to the original storage method if you wanted to do so.

[NoSpy]

This is a very good idea. 👍

[NoSpy]

I realize there's a big bug, because in the blogs the images aren't deleted even though all the blogs have been deleted. How can I permanently remove the IP.Blog module from my subscription?

[NoSpy]

Hi Marc Stridgen,

sensitive case are supported with invision ?

[Jim M]

1 hour ago, NoSpy said:

I realize there's a big bug, because in the blogs the images aren't deleted even though all the blogs have been deleted. How can I permanently remove the IP.Blog module from my subscription?

Files are deleted on a task to check if the content exists and if not, it deletes the file. It will also keep the files for items in your delete content deletion timespan. Once those two items are met and the content is deleted permanently, the files should be removed. If that is all met, do you have an example?

[NoSpy]

Hi,

go in files of blogs -> monthly_10_2014 … 1414375170 ; 1414374893 …

[Jim M]

These look to be from 3.x. Did you delete these before or after the upgrade to version 4?

[NoSpy]

We didn't even know they existed. I've started the transfer procedure in another folder to identify what's in excess.

[Jim M]

What I am assuming happened here is that the content was deleted while you're still on 3.x and something happened there to cause the disconnect. As we now handled that differently, the software is unaware of these abandoned images and won't deleted them because they aren't tied to content that was deleted on version 4. 

If that is not the case, we can investigate further. However, what you're in the middle of now with transferring these, is probably the best way to clear them.

[NoSpy]

I have hundreds of files that have not been transferred, even though they date from version 4.5. The IP.Gallery file tells me that the transfer went well, but it's empty, while I have no error in CPAdmin. In the member interface, but I don't know how to fix it. I have backups but each time I restore the problem gets worse because there's a mismatch between the database and the files.

Is it possible to communicate via e-mail so that I can send you the URLs and screenshots as the content is private ?

[Jim M]

If you want to share URLs privately, you're more than welcome to place these in the Notes field of the Access Details in the Client Area. 

[NoSpy]

I can't find note field in the customer area

[teraßyte]

33 minutes ago, NoSpy said:

I can't find note field in the customer area

It's in the same place where you enter the FTP/ACP details. See this guide: https://invisioncommunity.com/4guides/client-services/getting-support-r292/#access

 

[NoSpy]

Sorry for the late reply. Since I started moving files as recommended my whole website has become unstable and I have lost many files. This required restoring the entire website 3 times.

When I use the feature it doesn't move the files but deletes them and modifies the CHMODE on the folder making it totally unavailable. The upgrade from 4.5 to 4.7 created database errors.

when I go to CPAdmin -> Get Support

I can't solve the problem.

Edited June 2, 2023 by NoSpy

[Marc Stridgen]

We would ned more information on that. You need to click on the system logs to see what those entries are. Note however, if things are working fine, it may well have been errors you were having when you initially moved

[NoSpy]

SELECT record_id FROM `cms_custom_database_80` WHERE category_id=768

Unknown column 'record_id' in 'field list'

SELECT record_id FROM `cms_custom_database_80` WHERE category_id=773

Unknown column 'record_id' in 'field list'

UnderflowException: forum_sync_disabled

Error Non-InnoDB tables. But in my config setting I have $INFO['mysql_tbl_type']            =    'MyISAM';

Edited June 2, 2023 by NoSpy

[NoSpy]

I had to restore the whole website with a previous archive, as I had many files that were not copied to the new location and were deleted. Which justifies the need for a data recount feature, as moving files can take days/weeks/months depending on the volume of data. We've lost more than 1 month's work, since there's nothing to do but wait for the procedure to be completed.

Edited June 19, 2023 by NoSpy