Good Practice to protect shared files on fileserver when write access is needed
It appears that we lost some file folders today in one of our Windows Server fileservers.
Heretofore there's been no attempt to protect the folders from the users.
Nothing seems to add up:
- If a file folder were deleted, wouldn't it be in the Recycle Bin? These weren't.
- If file folders were moved by drag and drop in the confines of the fileserver, surely we'd find it by Search, no?
I see that we can disable Delete and Delete Files and Folders for the user group. Does this also prevent MOVE or drag&drop? It seems like it would. This seems like a reasonable way to protect the presence of folders and files when widely shared.
Aside from human action, intentional or in error, what other things come to mind that might cause a contiguous set of file folders to disappear?
Heretofore there's been no attempt to protect the folders from the users.
Nothing seems to add up:
- If a file folder were deleted, wouldn't it be in the Recycle Bin? These weren't.
- If file folders were moved by drag and drop in the confines of the fileserver, surely we'd find it by Search, no?
I see that we can disable Delete and Delete Files and Folders for the user group. Does this also prevent MOVE or drag&drop? It seems like it would. This seems like a reasonable way to protect the presence of folders and files when widely shared.
Aside from human action, intentional or in error, what other things come to mind that might cause a contiguous set of file folders to disappear?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
First, when sharing folders, set NTFS permission only to a limited number of users, only those who need that access. Second, have a backup solution so in case that some files or folders are deleted, you are able to restore them.
ASKER
Hello There:
Thanks!
Each share is limited to an access group. About the only thing that might be done differently would be to use a finer-grained structure - but, in a small organization, I can't be sure that there would be much improvement re: access.
Also there's a backup solution which did help in this case.
About the only thing I can say about the recovery process is this:
The backup system is set up to store versions - which by itself is fine.
The backup system is set up to not delete anything - so if something is deleted, it remains at the top-level of the backup.
This makes restoration cumbersome because "old" files need to be culled in restoring. But, of course, it does eliminate accidental deletions.
It makes me wonder if keeping a "live mirror" added wouldn't be a good idea so that restoration would be easier.
serialband:
Thanks!
We've not used VSS and it appears that it might be used as the "live mirror" that I mentioned above.
Thanks!
Each share is limited to an access group. About the only thing that might be done differently would be to use a finer-grained structure - but, in a small organization, I can't be sure that there would be much improvement re: access.
Also there's a backup solution which did help in this case.
About the only thing I can say about the recovery process is this:
The backup system is set up to store versions - which by itself is fine.
The backup system is set up to not delete anything - so if something is deleted, it remains at the top-level of the backup.
This makes restoration cumbersome because "old" files need to be culled in restoring. But, of course, it does eliminate accidental deletions.
It makes me wonder if keeping a "live mirror" added wouldn't be a good idea so that restoration would be easier.
serialband:
Thanks!
We've not used VSS and it appears that it might be used as the "live mirror" that I mentioned above.
How are you doing your backups? You shouldn't have to cull old files. You should be able to have snapshots in your backups too.
If you want something free, you could try the linux rsnapshot for Windows https://blog.gingerlime.com/2007/rsnapshot-server-on-windows/.
If you want something simpler, and you formatted your backup disk with NTFS, you can try hardlinkbackup and manually backup your daily or weekly incremental backups. https://www.lupinho.net/en/hardlinkbackup/ If you like how it works, you can pay 39 € for a perpetual license to let you automate the backups to a single disk.
I would suggest either rsnapshot or hardlinkbackup as an addition to whatever backups you already do. Don't replace it with these.Since these both rely on NTFS hardlinks, you can only make 1023 links, so you'll only have 1023 incrementals.
If you want something free, you could try the linux rsnapshot for Windows https://blog.gingerlime.com/2007/rsnapshot-server-on-windows/.
If you want something simpler, and you formatted your backup disk with NTFS, you can try hardlinkbackup and manually backup your daily or weekly incremental backups. https://www.lupinho.net/en/hardlinkbackup/ If you like how it works, you can pay 39 € for a perpetual license to let you automate the backups to a single disk.
I would suggest either rsnapshot or hardlinkbackup as an addition to whatever backups you already do. Don't replace it with these.Since these both rely on NTFS hardlinks, you can only make 1023 links, so you'll only have 1023 incrementals.
ASKER
"How are you doing your backups?"
We are usiing a simple copy option that:
- copies anything new into the backup.
- if a filename in the source has a new date, there can be an archive copy made in the backup (and up to a selected number) and the new file is backed up.
- the backup deletes nothing in the backup unless it falls off the archive stack.
There is a different mode which will more or less mirror the files in the source - so deleted files will then be deleted. But that has never seemed the best idea for a backup system. So, overall, this approach mixes old (deleted) files with current files in the backup.
We are usiing a simple copy option that:
- copies anything new into the backup.
- if a filename in the source has a new date, there can be an archive copy made in the backup (and up to a selected number) and the new file is backed up.
- the backup deletes nothing in the backup unless it falls off the archive stack.
There is a different mode which will more or less mirror the files in the source - so deleted files will then be deleted. But that has never seemed the best idea for a backup system. So, overall, this approach mixes old (deleted) files with current files in the backup.
Then I suggest you use rsnapshot or pay for hardlinkbackup. They work like the Mac's Time Machine, but without the pretty slider graphics. The hard link does not take extra space you you can still find the older copies. Just open the folder to the correct backup time and files will appear or not appear based on the backup time/date. If you need to just find the existence of the file, then search the outermost folder and you can find every occurrence to every backup folder time.
ASKER
serialband: Any experience with SecondCopy? That's what we're using. I wonder how it might compare with hardlinkbackup? How does it provide space for older copies without taking extra space?
Since the fileserver is Windows Server 2019, I'm thinking about turning on VSS and continuing to use SecondCopy - as we have good experience with it.
Since the fileserver is Windows Server 2019, I'm thinking about turning on VSS and continuing to use SecondCopy - as we have good experience with it.
I've never used SecondCopy. If that's what you're familiar with, you can stick with it.
Hardlinkbackup uses deduplication to make multiple "copies" of the same file into different folders for each backup cycle.
The way Deduplication works is with hardlinks, which exists on unix file systems and NTFS. Hardlinks are not a copy of the file, but a file pointer that points to the same file that uses the same space on the same disk. You can't make a hardlink of the file on a different disk. As long as the file has not changed on each date you can create a new link with to the same file in a folder with a new date. On NTFS, you can have up to 1023 links to the original file, which do not use additional storage space for the file. They just use 1023 additional file name pointers to the same data.
If the file data has changed, you would have a new copy that takes up additional space. The majority of most people's files do not change, so you save space by using a hard link, as long as you don't run out of links or total file handles, you can save a lot of space when the file has not changed.
Hardlinkbackup uses deduplication to make multiple "copies" of the same file into different folders for each backup cycle.
The way Deduplication works is with hardlinks, which exists on unix file systems and NTFS. Hardlinks are not a copy of the file, but a file pointer that points to the same file that uses the same space on the same disk. You can't make a hardlink of the file on a different disk. As long as the file has not changed on each date you can create a new link with to the same file in a folder with a new date. On NTFS, you can have up to 1023 links to the original file, which do not use additional storage space for the file. They just use 1023 additional file name pointers to the same data.
If the file data has changed, you would have a new copy that takes up additional space. The majority of most people's files do not change, so you save space by using a hard link, as long as you don't run out of links or total file handles, you can save a lot of space when the file has not changed.
ASKER
serialband: Well I understand the words but I'm still grappling with the concepts.
Let's see ....
If nothing changes, then there is a set of 100% hard links for each backup cycle?
If a file is deleted, then there is no hardlink in the next backup cycle?
If a file is created, then there is a file backup?
So, if a file was deleted yesterday before the backup then how is the recovery done?
I guess one would use today's files with yesterday's hard links and/or files but don't envision the process making it easy.
Without trying to argue or sell - just to share:
At the begining of each backup cycle, SecondCopy does a comparison of the source and the destination then:
In its Simple Copy mode:
If nothing has changed, then nothing is copied.
If a file has been deleted, then nothing is done and it stays in the destination.
If a file has been changed, then (assuming that one has opted for at least one archive copy) the destination file is moved into the archive and marked accordingly AND the new file is copied from the source to the destination.
This way, the destination folder holds more than one might like and recovery can be tedious,.
In its Exact Copy mode (I've not used it):
If nothing has changed, then nothing is copied.
If a file is deleted, then (assuming that one has opted for at least one archive copy) the destination file is moved into the archive and marked accordingly AND it is deleted from the destination folder.
If a file has been changed, then (assuming that one has opted for at least one archive copy) the destination file is moved into the archive and marked accordingly AND the new file is copied from the source to the destination.
So, this way the destination folder always matches the source folder while deleted files are removed but kept in the archive.
I rather imagine that the archive contains files deleted over time - so I'm not sure how easy inadvertently-deleted files from yesterday would be to extract and recover. I'll have to ask them or test it.
Let's see ....
If nothing changes, then there is a set of 100% hard links for each backup cycle?
If a file is deleted, then there is no hardlink in the next backup cycle?
If a file is created, then there is a file backup?
So, if a file was deleted yesterday before the backup then how is the recovery done?
I guess one would use today's files with yesterday's hard links and/or files but don't envision the process making it easy.
Without trying to argue or sell - just to share:
At the begining of each backup cycle, SecondCopy does a comparison of the source and the destination then:
In its Simple Copy mode:
If nothing has changed, then nothing is copied.
If a file has been deleted, then nothing is done and it stays in the destination.
If a file has been changed, then (assuming that one has opted for at least one archive copy) the destination file is moved into the archive and marked accordingly AND the new file is copied from the source to the destination.
This way, the destination folder holds more than one might like and recovery can be tedious,.
In its Exact Copy mode (I've not used it):
If nothing has changed, then nothing is copied.
If a file is deleted, then (assuming that one has opted for at least one archive copy) the destination file is moved into the archive and marked accordingly AND it is deleted from the destination folder.
If a file has been changed, then (assuming that one has opted for at least one archive copy) the destination file is moved into the archive and marked accordingly AND the new file is copied from the source to the destination.
So, this way the destination folder always matches the source folder while deleted files are removed but kept in the archive.
I rather imagine that the archive contains files deleted over time - so I'm not sure how easy inadvertently-deleted files from yesterday would be to extract and recover. I'll have to ask them or test it.
As I've said,
I've never used SecondCopy. If that's what you're familiar with, you can stick with it.
ASKER
serialband: Sure. Can do. But I did have questions about hardlink... ?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
serialband: OK. Thanks!
So I'm pondering the management of such an implementation. Here's the scenario:
We have a folder being backed up that's full of files and folders.
Let's say no changes. So now we have a hardlink "backup" each day.
Then we delete folders. Now there's no hardlink for today's backup for those folders - but there is for yesterday.
To recover them, we have to search the other "backups".
Now, let's assume that someone deleted some folders inadvertently a few days ago and we just discovered the loss.
How do we find them?
How do we find them if we don't know how many were deleted?
So I'm pondering the management of such an implementation. Here's the scenario:
We have a folder being backed up that's full of files and folders.
Let's say no changes. So now we have a hardlink "backup" each day.
Then we delete folders. Now there's no hardlink for today's backup for those folders - but there is for yesterday.
To recover them, we have to search the other "backups".
Now, let's assume that someone deleted some folders inadvertently a few days ago and we just discovered the loss.
How do we find them?
How do we find them if we don't know how many were deleted?
I suggest a real backup solution that can show you the contents of a system as it existed at a point in time...which means that it won't show you stuff that was deleted before the backup.
A proper backup solution also needs to support having 3 copies of your data (1 original and 2 full copies), on 2 different types of media, and 1 is off site. Another requirement for ransomware is that a copy needs to be air gapped or otherwise immutable to your administrators over the network.
Recycle bin only protects against local delete, not via SMB.
VSS has been around since Windows 2003, and while it is not a backup (stored on same media as the original data), it is fantastic and should be on every Windows file server IMHO.
A proper backup solution also needs to support having 3 copies of your data (1 original and 2 full copies), on 2 different types of media, and 1 is off site. Another requirement for ransomware is that a copy needs to be air gapped or otherwise immutable to your administrators over the network.
Recycle bin only protects against local delete, not via SMB.
VSS has been around since Windows 2003, and while it is not a backup (stored on same media as the original data), it is fantastic and should be on every Windows file server IMHO.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks!
comprobsolv: What you describe is much like our setup.
kevinhsieh: I think this is a *real* backup system. But perhaps our selection of backup mode is somewhat unfortunate. I'm considering a different mode.
We've been using what's called "Simple Copy" which copies everything that's new into the backup and deletes nothing. It also creates archive copies of a selected number. So, if a folder is misplaced or deleted and it existed at the last backup, it will be there.
There's another mode called "Exact Copy" which mirrors the source in the backup. So, it copies anything new and deletes anything deleted. I'm considering this with archiving. Presumably it will archive anything that's been deleted since the last backup.
Why "on different media" necessarily? Locally, we're using hard drives - triply redundant and protected. That and a cloud-based service.
comprobsolv: What you describe is much like our setup.
kevinhsieh: I think this is a *real* backup system. But perhaps our selection of backup mode is somewhat unfortunate. I'm considering a different mode.
We've been using what's called "Simple Copy" which copies everything that's new into the backup and deletes nothing. It also creates archive copies of a selected number. So, if a folder is misplaced or deleted and it existed at the last backup, it will be there.
There's another mode called "Exact Copy" which mirrors the source in the backup. So, it copies anything new and deletes anything deleted. I'm considering this with archiving. Presumably it will archive anything that's been deleted since the last backup.
Why "on different media" necessarily? Locally, we're using hard drives - triply redundant and protected. That and a cloud-based service.
3-2-1 rule explained
https://www.networkworld.com/article/3527303/for-secure-data-backup-here-s-how-to-do-the-3-2-1-rule-right.html
Your triple redundant hard drives are in a single enclosure, right? That counts as a single piece of media that can be destroyed from a single act.
Having a cloud based service is good.
What you are currently doing with simple copy is more of an archive than a backup. Remember that archives and backups solve different problems, and most products can't do both properly.
https://www.networkworld.com/article/3527303/for-secure-data-backup-here-s-how-to-do-the-3-2-1-rule-right.html
Your triple redundant hard drives are in a single enclosure, right? That counts as a single piece of media that can be destroyed from a single act.
Having a cloud based service is good.
What you are currently doing with simple copy is more of an archive than a backup. Remember that archives and backups solve different problems, and most products can't do both properly.
ASKER
kevinhsieh: The triply-redundant backups (on hard drives) are geographically separated. So, one type of media. Sorry if I misunderstood. Thanks for the clarification and the article!
So we have 5-3 (each with 5)-3 (including the cloud service).
So, we're not using a shiny commercial app but a lesser-known one and some things are home-made.
The current challenge is as I posted originally. And, the solution may be at hand.
I have yet to figure out and/or implement your VSS comment entirely!!
So we have 5-3 (each with 5)-3 (including the cloud service).
So, we're not using a shiny commercial app but a lesser-known one and some things are home-made.
The current challenge is as I posted originally. And, the solution may be at hand.
I have yet to figure out and/or implement your VSS comment entirely!!
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Hi,
No, not unless it was deleted on the server itself. If deleted via a shared drive from a workstation, the recycle bin is not used and there is no safeguard to restore the files.
Correct. The exception to the rule is if a nest of folders is dragged to a secondary drive on the server that is also available to network users. For that reason, all networked (mapped) drives to the server on the user's machines should be searched for the missing files and folders.
I struck this exact problem at a company I administrated for over 18 years. Files and entire nests of folders disappeared periodically and it took me a long time to figure out what was happening. It was drag and drop.
Given there is no native way to disable drag and drop on networked drives to the server, the only solution I was able to come up with was to replace Explorer on every workstation with a more powerful and configurable tool. I chose Directory Opus which is totally configurable in this respect and it turned out to be a great decision. There are other similar tools you could also consider. Doing that and configuring DirOpus to only allow drag and drop on the local workstation drives alleviated the problem completely for me, so something you might want to consider.
In so far as recovering folders that are deleted by your users, then a backup is the only solution. To find out which user(s) is/are doing drag and drop mistakes or deleting, then turning on auditing is the only way I found to be able to track it. It proved to be such a time-consuming task though, I abandoned it at the time and just prevented drag and drop on networked drives, as well as the ability to delete more than one file or folder at once.
To be able to recover files when they were deleted regardless, I configured a nightly backup that backed up the entire mapped drives to a NAS drive and kept six months worth of incremental backups I could go back to. This backup ran at the end of each business day after all users had logged out (shut down) their workstations. That solved the problem entirely for me.
Hope that's helpful.
Regards, Andrew
- If a file folder were deleted, wouldn't it be in the Recycle Bin? These weren't.
No, not unless it was deleted on the server itself. If deleted via a shared drive from a workstation, the recycle bin is not used and there is no safeguard to restore the files.
- If file folders were moved by drag and drop in the confines of the fileserver, surely we'd find it by Search, no?
Correct. The exception to the rule is if a nest of folders is dragged to a secondary drive on the server that is also available to network users. For that reason, all networked (mapped) drives to the server on the user's machines should be searched for the missing files and folders.
I struck this exact problem at a company I administrated for over 18 years. Files and entire nests of folders disappeared periodically and it took me a long time to figure out what was happening. It was drag and drop.
Given there is no native way to disable drag and drop on networked drives to the server, the only solution I was able to come up with was to replace Explorer on every workstation with a more powerful and configurable tool. I chose Directory Opus which is totally configurable in this respect and it turned out to be a great decision. There are other similar tools you could also consider. Doing that and configuring DirOpus to only allow drag and drop on the local workstation drives alleviated the problem completely for me, so something you might want to consider.
In so far as recovering folders that are deleted by your users, then a backup is the only solution. To find out which user(s) is/are doing drag and drop mistakes or deleting, then turning on auditing is the only way I found to be able to track it. It proved to be such a time-consuming task though, I abandoned it at the time and just prevented drag and drop on networked drives, as well as the ability to delete more than one file or folder at once.
To be able to recover files when they were deleted regardless, I configured a nightly backup that backed up the entire mapped drives to a NAS drive and kept six months worth of incremental backups I could go back to. This backup ran at the end of each business day after all users had logged out (shut down) their workstations. That solved the problem entirely for me.
Hope that's helpful.
Regards, Andrew
Sorry, just realised I neglected to address this question you posed.
Scripts that have deleting capabilities based on certain criteria could be a reason, but as the administrator, you would likely already know that. Another is false positives by an AV that automatically quarantines or deletes the false positive, or a corrupted server drive. All of those scenarios are unlikely though. In the many years I administrated servers, deleted files and folders were always user related.
Regards, Andrew
Aside from human action, intentional or in error, what other things come to mind that might cause a contiguous set of file folders to disappear?
Scripts that have deleting capabilities based on certain criteria could be a reason, but as the administrator, you would likely already know that. Another is false positives by an AV that automatically quarantines or deletes the false positive, or a corrupted server drive. All of those scenarios are unlikely though. In the many years I administrated servers, deleted files and folders were always user related.
Regards, Andrew
Hmmm. I never saw/noticed a message indicating you responded back in march.
You will never completely solve user issues with file deletion no matter how well you manage backups. You can only reduce the problems.
Hardlinkbackup is a backup solution. It makes a copy on a disk. You can also back up that disk.
If something is deleted, a few days ago and you have a copy from before that, you can search the entire disk and find it. It puts the data into date stamped folders and creates links to every file that hasn't changed. You just need to do the search at the lower level. If you know your command line, then you can list files from multiple folders at the same time.
You will never completely solve user issues with file deletion no matter how well you manage backups. You can only reduce the problems.
Hardlinkbackup is a backup solution. It makes a copy on a disk. You can also back up that disk.
If something is deleted, a few days ago and you have a copy from before that, you can search the entire disk and find it. It puts the data into date stamped folders and creates links to every file that hasn't changed. You just need to do the search at the lower level. If you know your command line, then you can list files from multiple folders at the same time.
ASKER
Good info and insights!! Thanks all.
ASKER
Thaks all!