Jump to content

Please clean up obsolete files


chuckf

Recommended Posts

Can you guys clean up obsolete files when you install new versions?

For example, I just installed Download Manager 2.3.1 over the top of an older version. The admin/applications_addon/ips/downloads/modules_public/ajax folder on my site has a comments.php file in it but that file is not in the 2.3.1 zip file. There are several other old DM files on my site.

I ran into the same problem with an add-on I wrote. I used the version_upgrade.php file to delete my obsolete files. Can you guys do the same or something similar?

Link to comment
Share on other sites

They'd have to keep a full record of the thousands of files in IP.Board, and keep that updated in every version. Some server setups probably won't let you delete files from just any directory either. And if it did delete all of these extra files, it might delete third party stuff too.

Link to comment
Share on other sites

How exactly can we clean up obsolete files? Do you mean like a tool in the ACP that will delete files not used by the current version of IPB and addons?

Problem #1) What about modifications? We'd end up deleting any modification files you have uploaded because we wouldn't "know" them.

Problem #2) MOST servers won't allow a script to delete an average file that was uploaded via FTP. I know there are server setups that allow this sort of thing (yes, yes, before people start hammering on me about it), however the fact is most people run Apache via ISAPI module, and the FTP user and PHP user are different, thus the PHP user cannot delete the files uploaded via FTP if they are not chmod appropriately, which you'd never want to do unless absolutely necessary.


It's just unrealistic I'm afraid. The best I could think we could do is on a major version upgrade, have a script which scans all directories and lists out any non-default files. That would require a lot of overhead and maintenance on our part, however, so I'm not sure how feasible it is, or if it would even get much use.

(An alternative is before you upload the new files for your site, delete the existing ones (except the folders you want to save, such as /cache, /uploads/ etc.) and then upload the new files).

Link to comment
Share on other sites


(An alternative is before you upload the new files for your site, delete the existing ones (except the folders you want to save, such as /cache, /uploads/ etc.) and then upload the new files).




This is what I was going to suggest :) , take a backup and delete them then upload the new ones.

Although its rare I'd go to that much trouble, I usually ignore them (exceptions being major upgrades like 2.x to 3.x). I may have a 'spring clean' when the next 'major' version appears though.

I'm not sure I like the idea of the upgrader doing unlinks incase it removes something I wanted and as has been pointed out some configs will not allow it anyway.
Link to comment
Share on other sites


Maybe a list of the correct files for each separate App could be done. That way those wanting to clean up their folders could do so be checking the list and removing extra ones.



When you download IP.Board, that's the list of what the 'correct' files are for that version. Anything else that used to be part of IP.Board can be deleted. The amount of space you'd be saving by doing this for anything but major version changes, though, is going to be negligible.
Link to comment
Share on other sites

What would help is if all files produced by IPS had version numbers. Most do but not all of them. It's then just a matter of running a version scan. If you do a backup of the files first you can then delete the old version files. The problem is the files with no version number. It's a time consuming job checking whether the latest version still uses this file.

3DKiwi

Link to comment
Share on other sites

What's with all the resistance?

I think we can all agree these obsolete files needlessly increase:
- the space used on hosting site
- the size of backups
- the time it takes to make backups
- the time it takes to transfer backups

Look, some programmer decided that some file is no longer need in the next version of a product so they deleted it from their hard drive and from whatever source code control system they use. It's only a little more effort to add the filename to a list of files to be deleted by the upgrader system.

If the upgrader can't delete files on some hosts so be it. At least they'll be deleted on the hosts that allow it, like mine :)

Bottom line is this was a request. It's your product so you'll do what you want.

Link to comment
Share on other sites


What's with all the resistance?


No one's saying it wouldn't be great to have extra files get automatically deleted, but there are very real issues with making it work, it's simply not possible for this to happen on most setups. Just because people point out these problems doesn't mean we're resisting the idea.
Link to comment
Share on other sites


The best I could think we could do is on a major version upgrade, have a script which scans all directories and lists out any non-default files. That would require a lot of overhead and maintenance on our part, however, so I'm not sure how feasible it is, or if it would even get much use.


I think the best solution here would be to provide a tool from the tools section in the acp which scans the folders and finds files that are not included in the out of the box-version - without deleting them. If people want them deleted they could then do it manually.

Not sure if I want you to use your development time on implementing such a tool however. But it seems that quite a lot of other users here want it.
Link to comment
Share on other sites

I don't know where some of you got the idea I wanted a tool or something to clean up everything since the beginning of time LOL.

I believe I was quite clear in my original post where I explained the problem by example and provide a possible solution but I'll try again:

All I'm asking is from now on when you (and this goes for modders too) release a new version of some product that no longer uses a file that was in the previous version, can you please delete that file during the upgrade process?

Let's say you start working on the next version of IP.Board (3.1.5, 3.2, whatever it will be called). You'll start the work with the currrent 3.1.4 version which consists of a specific set of files. As you work you decide that a particular file that is in 3.1.4 won't be included in the new version. This is when you would add a line to version_upgrade.php (or whatever mechanism you come up with) that would attempt to delete that file during the upgrade process. If the delete works then great. If not the upgrade process logs the failure and continues on. Users on systems where the deletes failed can manually delete those obsolete files with confidence because they now have a list of them in the log file and in the version_upgrade.php file. Note: if you decide to move a file you should still delete it from it's original locaiton.

As you can see this does not affect any other product as you are deleting your own products' files.

Admittedly doing this would require a little more effort from programmers but marketing types will love it. They can sell it as a new type of GREEN computing. One less file needlessly taking up space. One less file that anti-virus programs need to check. One less useless file!

Link to comment
Share on other sites

And what we're trying to tell you is that the technology doesn't allow that. In order for PHP to delete a file on the file system, it has to have permission to do so. It would only have this permission in directories that are set to be writable by everyone, and only a handful of directories in IP.Board are set up that way: things like /cache and /uploads. There are commands in PHP that you can execute to delete a file or directory, but they will fail to do so in any non-writable directory. That means anything in one of the application directories will never be able to be deleted this way.

No one is disagreeing that something like this wouldn't be great, it just can't be done.

Link to comment
Share on other sites


What would help is if all files produced by IPS had version numbers. Most do but not all of them. It's then just a matter of running a version scan. If you do a backup of the files first you can then delete the old version files. The problem is the files with no version number. It's a time consuming job checking whether the latest version still uses this file.



3DKiwi




They should all have a revision number.


We actually do keep an list of the filenames and the md5sum of their contents for the current major version. A tool to list extraneous files is possible, however I can already foresee numerous issues (it would flag up any files added by modifications, plus would be difficult to work out what skin/language files are actually in use, etc.) and I really don't see that many people using it.



As Michael mentioned, it is not possible for the upgrader to delete arbitrary files - the server won't allow it (think about it - doesn't that sound like a bad idea ;)).
Link to comment
Share on other sites

Michael, I understand what you are saying. All I can say is I have used the technique I've described on my hosting site and it works.

Also, your response completely ignored the part about logging failed deletes so users would have a list of files they could manually delete.

Link to comment
Share on other sites

They should all have a revision number.



Nope. Not all do.

From the 3.1.3 full zip file the following files do not have version numbers or to be more accurate return a blank when running the version checker. Now maybe some of them aren't files written by IPB. Some aren't and some are. I don't really care. IPB requires these to run yet if things change and those files are no longer used they will remain on my server. The obsolete files with old version numbers are easy to spot and to delete. What's concerning is a number of the files not returning a version number are in fact new files and written by IPS.

Files with no version numbers in 3.1.3 zip. Running a version check in the ACP has a blank for version info:


/ips_kernel/pop3class/parse_message.php 	

/ips_kernel/pop3class/test_pop3.php 	

/ips_kernel/pop3class/browse_mailbox.php 	

/ips_kernel/pop3class/pop3.php

/ips_kernel/twitter/OAuth.php 	

/ips_kernel/twitter/twitteroauth.php

/admin/sources/loginauth/convert/newauth.php

/admin/sources/loginauth/live/lib/windowslivelogin.php

/interface/email/piping.php

/admin/sources/classes/comments/bootstrap.php

/admin/sources/classes/member/status.php

/admin/sources/classes/like/composite.php



I note that the ACP version checker returns a blank result on a few files that when opened and checked have the 3.1.3 version number so something not quite right with the checker. I did not list these incorrect results.

So please have it so that all files have a version number. I can then manually check the blank results and see whether these are uploaded mods or if the version number is a previous version I can check and delete them.

3DKiwi

Link to comment
Share on other sites


What's with all the resistance?



I think we can all agree these obsolete files needlessly increase:


- the space used on hosting site


- the size of backups


- the time it takes to make backups


- the time it takes to transfer backups



Look, some programmer decided that some file is no longer need in the next version of a product so they deleted it from their hard drive and from whatever source code control system they use. It's only a little more effort to add the filename to a list of files to be deleted by the upgrader system.



If the upgrader can't delete files on some hosts so be it. At least they'll be deleted on the hosts that allow it, like mine :)



Bottom line is this was a request. It's your product so you'll do what you want.






So the maybe 1MB at max is really hurting your available hosting space eh?
1MB takes under 1 second to gzip , and a php file gets 99% compression with gzip

Your argument's are invalid, or you have a terrible host

A few extra files laying around, which you could easily not have if you managed your OWN sub-version system (build locally, than upload) would fix all of the "concerns" in this topic.

There's Code in OS X, Windows 7, Debian, CentOS, MySQL etc that is depreciated and not used.. but still there. Should everyone write in to all those Programmers and ask them to remove the 1-5MB of actual code that's not used to save you some disk space...




And KiWi they aren't going to put a version on something they didn't write.



<?php

/*

 * test_pop3.php

 *

 * @(#) $Header: /home/mlemos/cvsroot/pop3/test_pop3.php,v 1.7 2006/06/11 14:52:09 mlemos Exp $

 *

 */


?><HTML>

<HEAD>

<TITLE>Test for Manuel Lemos's PHP POP3 class</TITLE>

</HEAD>

<BODY>

<?php

Link to comment
Share on other sites

And KiWi they aren't going to put a version on something they didn't write.



Maybe, but they can put the version in a file they did write. That would have cut my list down by about half if they had done so.

Heck, even posting a list of files that are no longer required after a new version comes out would probably do the job.

3DKiwi
Link to comment
Share on other sites

1) Terabyte is going through ALL of our files and updating the headers as appropriate for 3.2, so the number of IPS-written files without a version number should be reduced to 0 by release time ideally.

2) For the record, some hosting setups *do* allow scripts to delete files, however the majority (that we run across at least) do not.


As for a log of the files that have been removed or deleted, I'll think about it a little more. It's a lot of extra overhead on our part, and I don't think there's any benefit to handling this at the IPB level, but maybe we can generate a list from SVN when we go to make a release of any files that were removed between the last release and the current one and post it for those who are interested.

Link to comment
Share on other sites


Michael, I understand what you are saying. All I can say is I have used the technique I've described on my hosting site and it works.


Just because it works on your site doesn't mean it will work for everyone. Your site is the exception, not the rule.


Also, your response completely ignored the part about logging failed deletes so users would have a list of files they could manually delete.


The fact that this tool you're suggesting simply will not work for most setups means that it would be a waste of time to consider a feature such as flagging those failed deletes. I ignored that suggestion because the tool is not feasible before you even get to making that feature work.

Look, I would love it if what you're suggesting was possible. I'm the guy who created a similar application for just the obsolete database stuff. I'm anal retentive about keeping my site cleaned up of these sorts of files too. But it's not fair to suggest IPS spend their time on a tool that most people will not be able to make use of. Except for major releases when the basic structure of the IP.Board file system changes, any space savings this tool would bring would be negligible; far less and even 1%. On most boards, it's not the file size that starts to become a concern, it's the database that really grows the bigger the site gets. Being able to delete a handful of obsolete files every couple of updates isn't going to make any real difference to the performance of the site.
Link to comment
Share on other sites


1) Terabyte is going through ALL of our files and updating the headers as appropriate for 3.2, so the number of IPS-written files without a version number should be reduced to 0 by release time ideally.



2) For the record, some hosting setups *do* allow scripts to delete files, however the majority (that we run across at least) do not.




As for a log of the files that have been removed or deleted, I'll think about it a little more. It's a lot of extra overhead on our part, and I don't think there's any benefit to handling this at the IPB level, but maybe we can generate a list from SVN when we go to make a release of any files that were removed between the last release and the current one and post it for those who are interested.




can SVN not date files individually as well

eg if a file has not been modifed then it will still have the "old" date (when last modified) not the "release" date

drupal did this and it made life easier as could tell at a glance if a file was needed to be uploaded when updating etc
Link to comment
Share on other sites


can SVN not date files individually as well



eg if a file has not been modifed then it will still have the "old" date (when last modified) not the "release" date



drupal did this and it made life easier as could tell at a glance if a file was needed to be uploaded when updating etc




SVN has a last modified date property, yes. That's not really what the OP is after though - he's wanting to know of files that are no longer required (i.e. have been deleted from subversion on our side, and thus are no longer included in the release).
Link to comment
Share on other sites


SVN has a last modified date property, yes. That's not really what the OP is after though - he's wanting to know of files that are no longer required (i.e. have been deleted from subversion on our side, and thus are no longer included in the release).




yep was just a suggestion based on the first point


on the what files are needed

one simple method I use if its a bog standard install is to upload dirs under a new name
eg
upload "galley" dir as "gallery-new" dir
once uploaded rename "gallery" to "gallery-old"
and then rename "gallery-new" to "gallery"

you can then remove galley-old (which should inc all obsolete files) when sure that everything is aok after backing it up etc etc
Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...