NETAPP maxdirsize

Hi Experts,

I have a question regarding a NETAPP storage.
I have the unit FAS2552.
Our ERP system generates a lot of files each day and they are stored in an archive folder on the NETAPP.
Now we have 1.6 million files in it.
On the NETAPP there is a limit which is called MAXDIRSIZE. This is sometimes a problem and I have to increase the limit.

Is it possible to switch it off ?
Can you explain the MAXDIRSIZE ?
What does it mean this value ?
Which value I need for 3 million files in one folder ?
Eprs_AdminSystem ArchitectAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Eprs_AdminSystem ArchitectAuthor Commented:
Who can help me here ?
0
Eprs_AdminSystem ArchitectAuthor Commented:
And why it is neglected ?
0
robocatCommented:
MAXDIRSIZE is the limit for the number of files that you can store in a folder.

The limit is there for performance reasons.

Computer file systems are designed to store a moderate number of files in a single folder, thinking that you can create sub folder if you need to store more files (kind of like a tree). As a consequence, file systems are not very good for storing large numbers of files in a single folder and doing so will seriously hurt performance.

So the limit forces you to think about this. You can increase the limit, but this will hurt performance even more or even lead to file system issues. Best is to move the files into sub folders, (e.g. by date) and not increase the limit.
0
Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

Eprs_AdminSystem ArchitectAuthor Commented:
Thanks a lot.
May you answer my questions ?

Is it possible to switch it off ?
Can you explain the MAXDIRSIZE ?
What does it mean this value ?
Which value I need for 3 million files in one folder ?
0
Mark GeerlingsDatabase AdministratorCommented:
It looks to me like Robocat did answer your questions.  We also have NetApp NAS storage and a large ERP system that generates lots of files.  But trying to keep 3 million files in one folder is not a good idea.  You need to come up with a way to either reduce the number of separate files, or to sub-divide those files into multiple folders.
1
yo_beeDirector of Information TechnologyCommented:
if you want to increase you can use, but once set you cannot revert back.  Like Robocat stated this can lead to performance issues and other unwanted things.

First I run just to list all the vols and the current limit.
maxfiles

Open in new window


Volume FirmData: maximum number of files is currently 31876689 (5724589 used).
Volume LIT_IMG01: maximum number of files is currently 93535556 (88124169 used).
once you have the volume name you want to increase the size in small increments.  .  I normally do 10% of the current.
ie.  LIT_IMG01 is set to 93,535,556.   I would increase it to 102,889,111.  This will round to the nearest block value.
maxfiles LIT_IMG01 102889111

Open in new window

0
Eprs_AdminSystem ArchitectAuthor Commented:
ok thanks so far
0
Eprs_AdminSystem ArchitectAuthor Commented:
As far as I know, the maxfiles value is the total amount of files on the volume.
This is not my problem.

My problem is the maxdirsize, the files per folder.
0
Mark GeerlingsDatabase AdministratorCommented:
What options do you have to reduce the number of files, or to sub-divide these into multiple folders?  Those may be good questions to ask your application vendor if you don't know the answers.

If you absolutely must support this large number of files in a single directory, you may want to explore configuring your Net APP to support a SAN instead of NAS connection.  (Most NetApp systems these days support both.)  I've had good results in the past with large numbers of files in a single SAN volume.  That may be a good question for you to ask Net App.
0
yo_beeDirector of Information TechnologyCommented:
It looks like the Author is asking what is the max size of a file or folder not the actual number of files per volume
0
robocatCommented:
Maxdirsize is a volume option (vol options maxdirsize).

The number is not the number of files but the maximum (technical) size of a directory in KBytes.  So it's not easy to calculate how big it must be for a certain number of files.

Increasing it without a real understanding of what this means or why the limit is there can lead to severe performance issues, and can not be undone.

So do not increase this limit, but try to move files into subfolders. About 100K files is a reasonable maximum per folder.
0
Eprs_AdminSystem ArchitectAuthor Commented:
I spoke to my vendor of the EPR system but they say it is hard to change.
So I think this takes time and I do not have time.
Because we write each day 2000 files into this archiv folder.
0
Eprs_AdminSystem ArchitectAuthor Commented:
NETAPP told me to create a new volume just for this archive and disable UNC path.
And maxdirsize should not bigger than 3% of the RAM in the controller.
Can you tell how to find out, how much RAM I have on the controller ?
0
robocatCommented:
>I spoke to my vendor of the EPR system but they say it is hard to change.

The ERP system doesn't need to do this or even be aware of this. This is actually quite a common problem with many ERP systems that is easily solved, assuming these files can be archived (audit files, log files, ...).

You can start by moving the files manually. Move all files that are no longer actively used to a few subfolders (at least two). Say all files older than a month?

This automatically clears the problem and the original folder will have room again for the months to come. Each subfolder that you create can itself contain as many files as the original folder, no need to increase maxdirsize.

Next, you can automate the archiving of these files, eg. by scripting or using some kind of archival software.
0
Mohammed KhawajaManager - Infrastructure:  Information TechnologyCommented:
If your ERP system is using a relational database then the files generated are usually kept there for recovery which basically means that you restore your last backup and assuming you do not have your transaction logs backed up then your ERP has kept logs for point and time recovery.  Now realistically, will you ever restore your ERP from 6 months ago, last month or even from 2 days ago?  The answer is resounding NO.  Therefore, these are archive files and what you could do is create a policy to keep at the most one week of files and if you are really paranoid then ensure they files are backed up.
0
Eprs_AdminSystem ArchitectAuthor Commented:
Hi ROBOCAT,

ok I understand.
But when I move the files to another folder, the ERP system cannot find the articles anymore.
So this is also not an option.
Because in this archive folder they save all pictures for our articles.
0
robocatCommented:
Well because you can't undo an increase, you it is wise follow Netapp's advice and create a separate volume just for this purpose. Move the directory over to that volume.

Take note of the current size of maxdirsize for that volume and increase in 2% increments until ok.

This is only a temporary solution, you will run into the limit the controller can handle sooner or later. So it buys you some time to tackle the real problem.

Your ERP system is clearly not designed for a large number of pictures because no well designed system will store all files in a single directory. Contact the ERP people to have this changed.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Eprs_AdminSystem ArchitectAuthor Commented:
many thanks so far.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.