Lost Attachments In Move

[David_B]

I just moved my site from one host to another. Both were VPS plans with WHM, and Virtuozzo. I attempted to move the site through WHM, but unfortunately I did not have enough free disk space at my old host to allow for this method. My new host suggested I "lighten the load" by backing up and cleaning my uploads folder. (I also did this with my arcade folder which contained over 400 megs) I copied the uploads folder over to my computer and proceeded to empty it for the transfer. I have not had enough disk space to do local backups through cPanel, so I have always FTPed the entire public_html folder to my computer and backed up my db through phpMyAdmin. So I was confident that I should be able to clean out the uploads folder and restore it manually by ftp after the site was moved.

I backed up my site by downloading the public_html folder which of course contained the uploads folder, then proceeded to empty it and the arcade folder. This gave me a lot of free disk space. It seemed like a lot more than what I should have gained by cleaning out those 2 folders. They showed in my file managers (cPanel, Virtuozzo, FTP client as well as the downloaded files on my computer) to contain roughly 600 megs combined, with the uploads folder being less than 100,yet after cleaning the contents out of these folders my free disk space seemed to jump by about a gig and a half. I went forward with the transfer through WHM and it was successful. I later uploaded the contents of my "cleaned out" folders and found I was missing a ton of attached images. My arcade folder seemed to be a successful restore.

To make a long and frustrating story short, the uploads folder only contained a fraction of the accumulated attachments the board has had in the almost 4 years we have been running. It is heartbreaking to say the least.

Here is the weird part, I went through my backups, fortunately I had a good 3 years worth. In each and every instance, the upload folder contained something less than 100 megs. However, each folder contained a different mix of files. I was able to salvage quite a lot of the attachments through pulling the different files from various backups. I have nearly tripled the amount of attached images originally contained in the last backed up folder. Sadly, I think we have still lost something like 3/4 of our attached pics.

I know there is no hope of recovering anything else, but I am at a loss for why my uploads folder did not contain the entire attachments for my site. Nor has it ever. I have to wonder where the board was drawing from for all the attachments. Is there a hidden archive somewhere? I mean I would have thought that by downloading the entire public_html folder that would contain the entire contents of the site.

Paint me confused and perplexed, but at this time the best I can hope for is a reasonable explanation of what happened, why I was unable to back up all my board attachments and maybe hope that no one else has this happen to them. :huh:

[MrFixIt_merged]

oops! :lol: Sam Adams is here!


Edited because I was responding to a PM....but I was in the 'Add Reply' box......

[bfarber]

All attachments would have been in the configured uploads directory - I don't think the board has any involvement here, outside of a possible software limitation in previous versions (read on)..

Many Linux operating systems don't like more than x number of files in a directory (I think it hovers around 1000 or 2000), and prior to 2.2 or 2.3, IPB would upload all files to one directory. Many times I've logged into FTP on a site and couldn't even get a full directory listing via FTP because there were too many files in the uploads folder.

It's possible if you had a very large number of files (not size, but count) then your operating system was only pulling the most it was designed to pull (for example, 1000) both for backups, and one doing FTP listings.

[David_B]

Thanks, thats consistent with what I found out. I believe the ftp server has a default limit of 2000 files per folder. I was only able to back up the visible part of that folder. It sure isn't Invisions fault, but maybe knowing this can prevent someone else from making this mistake. It's nice to know 2.3 no longer uses a single folder.

I also found out that the reason for this limitation is to preserve resources. I now have plenty of extra disk space so I will use the cPanel backup, which I understand will back up everything, visible or not.

This was a costly mistake, my board lost a lot of history. Live and learn. ;)

[Bono]

All attachments would have been in the configured uploads directory - I don't think the board has any involvement here, outside of a possible software limitation in previous versions (read on)..

Many Linux operating systems don't like more than x number of files in a directory (I think it hovers around 1000 or 2000), and prior to 2.2 or 2.3, IPB would upload all files to one directory. Many times I've logged into FTP on a site and couldn't even get a full directory listing via FTP because there were too many files in the uploads folder.

It's possible if you had a very large number of files (not size, but count) then your operating system was only pulling the most it was designed to pull (for example, 1000) both for backups, and one doing FTP listings.

I'm atm using 2.1.17 version of IPB board, when i migrate to 2.3 will upload directory automatically be migrated to new way of storing attachments? Or i would need to do it manually?

I noticed this problem also thats why i tared my previous backup and moved it to new server, directory with a lot of files also degrades performance of server.

[Darkside_RG_merged]

:unsure: Avatars and photo's still end up in just the uploads directory does the profiles directory of uploads only come into play if you have the new profile system enabled rather than classic? as its a pain to still have to fish around in the main directory to back up more recent avatars etc

[bfarber]

As for the attachments, the upgrader does not move them or anything. You'd have to move them to a folder and then update the database if you wanted to do that.

photos should be put in a subdirectory, though I believe there's an open bug report on that. It works in some cases, but not all.

[Darkside_RG_merged]

I understood about attachments and as they are backed up it's not a problem and the monthly subdivision is a great improvement, it's just the issue with Avatars photos etc which makes things more awkward.

Thank you for your reply bfarber I'll go and look for the bug report.

[NTGforum]

As for the attachments, the upgrader does not move them or anything. You'd have to move them to a folder and then update the database if you wanted to do that.

photos should be put in a subdirectory, though I believe there's an open bug report on that. It works in some cases, but not all.

There is a tools for automatic move the attachments ?

[Bono]

There is a tools for automatic move the attachments ?

There is no automatic tool.

Can someone give example how it would it look like? Move some old posts to one dir and which table to edit.

Thanks

[bfarber]

You'd have to create directories, move the files there (maybe do it based on the file's atime or something), and then update the ibf_attachments table to add the folder name to the filename columns (look at a recently uploaded attachment as an example).