Compare NTFS permissions between source and destination folder

Hello experts,
I hope all is well. We have tool in C# that copies data from one folder to another, we are looking for a way to speed up the permission comparison and apply any changes made to the NTFS security permissions.

Currently it takes 2 hours to go and compare 20k folders between source and destination (destination is on a remote machine). Do any of you have some good thoughts on how to possibly improve that?
LVL 5
onlinerackAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

btanExec ConsultantCommented:
This past EE posting is good resource of the options, NTFS Permissions Reporter has been mentioned and has CLI, can take a look as it mentioned "caching" in execution which may helps.
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_28065160.html
In the past, this tool scan folder tree & produce permission report which can be saved in Excel hence having 2 reports you can do comparison in Excel, but that doesn't seems fast and automated ... it may have changed - See "Compare Reports feature shows you the differences between permissions in 2 different reports" http://blog.cjwdev.co.uk/2013/09/23/ntfs-permissions-reporter-1-5-released/

Another I saw in this tools too - Beyond Compare. Looks like it also can automate repetitive tasks using a flexible scripting language, and any script can be called from the command line
http://www.scootersoftware.com/features.php

 other forum  as of below
setacl.
This is a free tool you can obtain from helgeklein.com
To compare two directories I did this:
- made a list of all ACL's in both folders
- list in a text format
with setacl this is done like this
setacl -ot file -on folderA -actn list -lst "f:tab" > listA
setacl -ot file -on folderB -actn list -lst "f:tab" > listB
- compare the 2 lists
I do this with diff (cygwin collection)
diff listA listB
https://helgeklein.com/setacl/
0
Lionel MMSmall Business IT ConsultantCommented:
Another thought is to run icacls after the copy to make sure the proper permissions are applied afterwards -- you add it to the end of your copy process and it will run through the destination folders(s) and assign the proper permissions. Another way of solving the issue.
0
arnoldCommented:
Incorporating the attribute/ntfs security settings into the copy  process will/could eliminate the need.


What is the environment? rsync, DFS-R, robocopy with the /L can be used to get the comparison report.

Is there a possibility to modify the tool?
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

onlinerackAuthor Commented:
Thank you all for your input, the tool is custom developed in-house using C#.

The issue we run into copying them at the same time could cause an issue as the source folders may have (read only) in source, so once we copy it to target then the target folder would be read only as well which would then deny us from copying files to the location.

Ideally we want to be able to have it compare the permissions fast enough instead of taking 4 hours over 500k folders.
0
Lionel MMSmall Business IT ConsultantCommented:
not sure what you are using to do the copying but can you remove the Read only attribute when copying for the first time and then run it again for the second time and add it back or setup a parameter that you only copy files that meet certain parameters and then you only have to remove and add back the read only attribute to those files instead of all 500K folders. for instance robocopy has the following switches to add and remove attributes
/A+:[RASHCNET] : Set file Attribute(s) on destination files + add.
    /A-:[RASHCNET] : UnSet file Attribute(s) on destination files - remove.
0
arnoldCommented:
not sure what your script does that could run into a situation that it creates a file holder, sets its attributes and then tries to copy the content. Usually the reverse is done without the need to remove/readd attributes.

The issue I suspect deals with folder permissions, i.e. the folder is created, its attributes copied, and then the contents are attempted to be copied.

Unfortunately, there are way too many possible ways with which this issue can arise.
0
onlinerackAuthor Commented:
It is not the read only attribute, It is the read permission, so I am ending up writing all the data then pushing down the permissions to avoid if there is a permission with read access only.
Trying to see if anyone has a way to push permissions or compare permission in C# that does not take as long. :)
0
arnoldCommented:
One options is to catalog and recursively apply the permissions i.e. while traversing/copying files you have access to the directory and the permissions it has. Not sure how robocopy does it.
but I think the attributes on the files are applied as soon as the file is copied, my guess the permissions/attributes on a directory are set when all directory content has transferred.

It sounds as the delay in your case is occurring because your process is doing a double pass. Create the directory structure and Copy the files, then go through and make sure the permissions are identical.

Presumably you have a recursive go through the directory structure function.
directory1 if you find another directory you will cal the same function with the new directory path. at the same time capturing the current settings/attributes is the inerit from parent set, and what permissions if any are not inherited from the parent. now that I think of it, must it flow this way or whether the environment is such that certain directories are not even accessible to Administrators, how is that accomplished?

rsync and robocopy have over time possibly achieved this in less time.

In horse/animals  barn analogy, you are asking how to more quickly gather the horses/animals that exited the barn when the doors are left open.

Depending on your environment and what it is exactly this process is supposed to do, it is hard to say whether you should try to add the additional functionality into your existing tool, or look to implement a change on how the data is maintained.
backup on source and restore on destination might be faster and will preserve the attributes without the need to check.
DFS-replication to have it synchronize the data..
rsync/robocopy

I think you mentioned using caching, but figuring out how to maintain a cache and how to determine whether the cache is no longer valid and/or each directory will have a cache file or the entire repository will have a single file......
0
btanExec ConsultantCommented:
understand in C# code at .Net 4.0, it has inbuilt iterator block versions (EnumerateFiles, EnumerateFileSystemEntries) that may be faster (more direct access to the file system; less arrays), in case this comes handy..
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
onlinerackAuthor Commented:
Thank you guys for your help. Your comments were along the lines were I was heading but was not sure, so it helped determine getting to it. We ended up morphing your suggestions to fit it in our tool and it came out perfectly. Let's just say, it was 2.5 months worth of changes. :)
Thank you once again.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C#

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.