Coffinated
asked on
optimal number of files per folder on Linux platform?
I need to store about 120,000 pictures on the server for a shopping cart application, obviously they are not going in to one folder, so... I am considering two scenarios, which would be more efficient in your opinion?
1. create 36 subfolders with average=3,100 max=22,000, min=106 images per folder
Top folders will have: 21100, 10612, 6741, 4919, 4496, 4268, 4267, 3756, 3736, 3489 images per folder
2. create 1,100 subfolders with average=100, max=13,000 min=1 images per folder
Top folders will have: 13615, 5280, 3834, 3567, 3195, 2262, 2231, 2127, 1726, 1031
In addition to above images/folders, additional 120,000 images will be uploaded to different folder with the same subfolder structure, those will be the thumbnails, image/thumbs locations will be stored in mysql DB, which scenario will make sense? I want to achieve max performance if there is any, In the near future another 100k-200k images will need to be stored.
Thank you
1. create 36 subfolders with average=3,100 max=22,000, min=106 images per folder
Top folders will have: 21100, 10612, 6741, 4919, 4496, 4268, 4267, 3756, 3736, 3489 images per folder
2. create 1,100 subfolders with average=100, max=13,000 min=1 images per folder
Top folders will have: 13615, 5280, 3834, 3567, 3195, 2262, 2231, 2127, 1726, 1031
In addition to above images/folders, additional 120,000 images will be uploaded to different folder with the same subfolder structure, those will be the thumbnails, image/thumbs locations will be stored in mysql DB, which scenario will make sense? I want to achieve max performance if there is any, In the near future another 100k-200k images will need to be stored.
Thank you
ASKER
Superdave:
I received this response from my web host
"...the way the linux filesystem works, directories with more files are much slower. Your best bet is to try to keep directories under 5000 entries; even then you will see significant performance degradation with some utilities. The absolute maximum of the filesystem is 31998 entries so keep that in mind also..."
Scenario 1 will have 3 folders with more than 5k files
Scenario 2 will only have 1 folder with more than 5k files.
It is ext3 file format, I don't understand why they recommended to keep it under 5k per folder.
I received this response from my web host
"...the way the linux filesystem works, directories with more files are much slower. Your best bet is to try to keep directories under 5000 entries; even then you will see significant performance degradation with some utilities. The absolute maximum of the filesystem is 31998 entries so keep that in mind also..."
Scenario 1 will have 3 folders with more than 5k files
Scenario 2 will only have 1 folder with more than 5k files.
It is ext3 file format, I don't understand why they recommended to keep it under 5k per folder.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
My guess would be that on most filesystems there won't be any difference between your two possible schemes, because they're both two-level lookups, and that putting all 120,000 files in one directory would be slightly faster. If the filesystem is something really old-fashioned like FAT32 where the directory entries aren't indexed, then I'd go for the 36 subfolders scheme.