Windows File Server Share Size Auditing Tracking and Reporting Solution Needed

We have a Windows 2008 R2 File Server with massive amounts of user data, that's constantly growing.  Currently we are tracking some of the root level shares to get a better idea of where this growth is occurring, but we are looking for an automated solution that get's more granular.  Something that will track file changes and report on a daily basis.  I've run free tools like WinDirStat and TreeSize, but the issue is that there are tens of millions of files and several TB of data that the software either bombs out or does not complete.  If I specify smaller subdirectories then it works, but then I have to repeat this dozens of times.  If there is a software that can schedule reporting on different paths and ideally report on the deltas since the last report would be ideal.  Any thoughts and suggestions are appreciated.
LVL 1
fireguy1125Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
Your auditing events deal with created/accessed, modified?

What resources are in use in your environement?
Do you have server montoring zabbix, nagios, snmptrap server?

what types of reports are you looking for?
NinjaStyle82Systems AdministratorCommented:
Have you looked in to the FSRM role in Server 2008?

https://technet.microsoft.com/en-us/library/cc732431.aspx
Gerald NegrotaIT ProfessionalCommented:
A little scheduled script that do:

- list file down on the tree, with details ( Creation Time, Last Access Time, Last Modification Time, Size)
- make a DIFF from a previous list ( must be available)
- write a "report" file
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

fireguy1125Author Commented:
We use SolarWinds, but does not have capability that we find to monitor and report on directories.

We have implemented the FSRM role, but the reporting capabilities are limited.

I'd be interested in the scheduled script - do you have one available?
arnoldCommented:
Does your system have the solarwind trap receiver?
Presumably, your servers are SNMP enabled to be monitored via SNMP
You can configure the fileserver auditing and then use evntwin to configure the event to snmptrap mapping
All that would be left is to have the solarwind receiver record and you can then use those entries/events to report ...........
fireguy1125Author Commented:
Not sure how that would report on directory sizes, can you explain? I'm looking for more of a scheduled "tree size" type automated report and capturing changes. Our monitoring is WMI based.
arnoldCommented:
I thought you wanted reports on access/file creation/deletion/auditing not usage calculations.
For system/file usage, you would need to use powershell/vbscripts running at scheduled events.
NinjaStyle82Systems AdministratorCommented:
You want a daily report of all changes on a file server, like when any file changes, get added, or removed correct? Or do you just need sizes day to day?
Joe Winograd, Fellow&MVEDeveloperCommented:
Hi fireguy,

I'm a huge fan of NirSoft's (free!) utilities, which I've been using for many years:
http://www.nirsoft.net/

One of them is SearchMyFiles:
http://www.nirsoft.net/utils/search_my_files.html

Scroll to the bottom of the page at the above URL for the download links. Note that there are both 32-bit and 64-bit versions. It is a no-install/stand-alone executable — just unzip the file and run the EXE.

In addition to its standard GUI, it has a command line interface, which is described in detail about halfway down the page.

SearchMyFiles has never failed on me, although I've never used it on the massive number of files that you have. But I think it's worth a try, and with the command line interface, you can easily call it from a script that can be run in an automated fashion on a scheduled basis via the Task Scheduler. Regards, Joe
Gerald NegrotaIT ProfessionalCommented:
Keep in mind that any scheduled script and/or any solution based on something that is scheduled is not an real time monitoring solution. To have something "real time" must build a daemon (resident program or service) that record any "touch". This will generate a huge quantity  of data.
For a scheduled script, you don't need a WMI, you can archive all by using simple "DOS" ( cmd) script. For example DIR command with the right switch(s) is a very fast way .
Gerald NegrotaIT ProfessionalCommented:
Just to be onest : http://www.codeproject.com/Articles/42212/WMI-and-File-System-Monitoring . Keep in mind that the "sensors" using the Windows Management Instrumentation (WMI) protocol have high impact on the system performance!
fireguy1125Author Commented:
It is more of identifying the size of the directories, and where the changes occured since the previous polling period. Ideally it would also show a report of which files were created in those locations.
arnoldCommented:
The create/delete might be what you want to audit reported into solarwinds
There are directory monitoring "services" but that would add load/overhead to the .....
With the create/delete/access/modify auditing events
You could speed up the process by defining the subset of the "top" directoies you want checked for the current size. You could then use the comparative logic that should the size reported be of a certain delta size change from the prior check, perform what actions you think are  meaning further drill down, record the event window last run of this script to now. which will let you look at the audit to see which files were created/modified/deleted that could be the reason for the change in the .......

Depending on the setup, I think what you are looking for is made more difficult. i.e. if you can use FSRM to set specific size restrictions on individual shares that are part of the DFS root or you have a hierarchical dfs root as a single  share with many subfolders.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.