IPB Upload directory. Great I/O killer ?

[FabioPaz]

Hello,

Talking with my hosting support, they told me to look into the invision power upload folder that have 116572 files (boardurl.com/upload). That this could be a great I/O killer, decreasing server performance.

He suggested to me that I should create a date based structure with no more than 100 files per folder. But this is manually impossible :no:

What do you guys think about this ? My hosting is overreacting or this is, indeed, a bad idea and IPB should do a structure more like wordpress (date based) or there's some configuration that I didn't see ?

(Just for the sake of information, my board have 150.000 users with a lot of attached images, avatars, etc...)

Thank you!

[AndyF]

Did your board start on the 2.1.x (or lower) series ?

Those versions kept all uploads in one directory, iirc it was the first 2.2. release that started to split them to monthly_xyz directories.

Board on its own is typically 3777 files in 651 directories, according to looking at the upload section of the zip via Windows.

[FabioPaz]

Did your board start on the 2.1.x (or lower) series ?

Those versions kept all uploads in one directory, iirc it was the first 2.2. release that started to split them to monthly_xyz directories.

Board on its own is typically 3777 files in 651 directories, according to looking at the upload section of the zip via Windows.

Hello Andy,

Unfortunatelly, I don't remember what version I started using invision.

But looking in my "uploads" directory, I actually have monthly subdirectories, but only with attached files (that is the minority) and in the main "uploads" folder I still have a massive number of pictures, with av- and photo- prefixes.

Thank you

[AndyF]

I'd personally consider another host if they are suggesting you are limited to 100 files per directory.

Although hosts do not like massive, massive amounts of files in one folder (filesystem concerns iiic) 100 appears far too low I think.

[rct2·com]

Those will be the members photos and avatars typically. They will only be dragged from the server to a visitor's browser when viewing pages that include those members. I don't see it as an I/O killer unless you have views that show all 150,000 members on a single page ;)

[rct2·com]

The filename is linked to the member number. So:

av-<membernumber>.<filetype>
photo-<membernumber>.<filetype>
photo-<membernumber>_thumb.<filetype>

If you have deleted members in the past, it may be worth checking that they don't still have photos/avatars in that uploads folder?

[Grumpy]

I'd say your host is overreacting or misdiagnosing an issue.

Sure, having massive number of files in one directory is ugly and not best of performance, but just sitting there doesn't cause any issues. Unless the system is somehow constantly double-checking for files in that folder, there's little real reason for it to create IO problems besides space usage.

If you have an IO problem, I suggest you look elsewhere for the real issue.

[yacenty]

Just now I have realized it could be a small issue especialy when You have 100k users, multiplied by 3 files it gives us 300.000 files.

Then when You have 1000 users online (by GA real time) ~2000 users in last 15min by IPB, it could a problem.

Any idea how to optimize it?