shadow protect images - at some point some files may not be recoverable? Or how do you do it right?
It dawned on me recently. Is this correct? under some situations, a file can't be recovered if
a) it wasn't on the server long
b) it's a long time after the file was deleted / corrupted?
Specifically with shadow protect, but I would think it would apply with other apps. Is there a way to avoid the problem? Do you point this out to the client? How? It seems like an obscure situation they may not grasp and get more afraid that you are saying things might not be recoverable.
Say a file is created on the server on the 2nd of the month. You are doing 15 min continuous incrementals. It winds up getting deleted / corrupted within a couple weeks - before the end of the month.
It IS on the 15 min. incrementals, the daily consolidated, the weekly consolidated... but not on the monthly consolidated.....
then over time, the retension settings for Shadow Protect throw out the unneeded dailys and weeklys that have the file in them. yes, using the defaults of SP, that's weeks / months later. but then the client says - 'we just noticed this file we made a while ago was deleted....
it could be unrecoverable, right? Even if you 'archive'. If you have an archive of the 1st or 31st of the month, the file will not be there. yes, maybe an obscure situation. but a possible situation?
how do you deal with that? Avoid that? etc.
a) it wasn't on the server long
b) it's a long time after the file was deleted / corrupted?
Specifically with shadow protect, but I would think it would apply with other apps. Is there a way to avoid the problem? Do you point this out to the client? How? It seems like an obscure situation they may not grasp and get more afraid that you are saying things might not be recoverable.
Say a file is created on the server on the 2nd of the month. You are doing 15 min continuous incrementals. It winds up getting deleted / corrupted within a couple weeks - before the end of the month.
It IS on the 15 min. incrementals, the daily consolidated, the weekly consolidated... but not on the monthly consolidated.....
then over time, the retension settings for Shadow Protect throw out the unneeded dailys and weeklys that have the file in them. yes, using the defaults of SP, that's weeks / months later. but then the client says - 'we just noticed this file we made a while ago was deleted....
it could be unrecoverable, right? Even if you 'archive'. If you have an archive of the 1st or 31st of the month, the file will not be there. yes, maybe an obscure situation. but a possible situation?
how do you deal with that? Avoid that? etc.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Sorry if I am being stubborn...
Hyper - I like how you make a point to mention the shorter (14 day) window. As much as we all talk of backups for months.... some weird situations might not have the files months later.
Philip - I understand what you are saying but do you agree with Hyper - there's situations where an admittedly likely tiny amount of files might not be there when you look at the backups at some point in the future?
Like Hyper's created and deleted between snapshots, but also my thinking that even file that exists for weeks, might not be recoverable?
Hyper - I like how you make a point to mention the shorter (14 day) window. As much as we all talk of backups for months.... some weird situations might not have the files months later.
Philip - I understand what you are saying but do you agree with Hyper - there's situations where an admittedly likely tiny amount of files might not be there when you look at the backups at some point in the future?
Like Hyper's created and deleted between snapshots, but also my thinking that even file that exists for weeks, might not be recoverable?
The only time we have experienced file loss was in the case of an IDE RAID 5 setup where some of the disks started to experience wobbly bits (bad sectors).
Those wobbly bits lead to a garbage in garbage out situation with the backups as they did not manifest themselves until the server went full-stop.
After a hard-reset the server came up and all was seemingly happy. Then the NTFS 55 errors came along. The she went full-stop again with no recovery.
A perfect storm of events lead to the backups being relatively useless (BackupExec to DAT libraries).
That was the last time we experienced file loss. Out of 650GB of data we recovered everything but one partner's 24 files.
To date we have gone through some very spectacular recoveries in part due to ShadowProtect and in part due to the skills learned via the SwingIT migration techniques (www.sbsmigration.com - best $400 I've ever spent). No data was lost.
Since switching to SAS only disks and hardware RAID we have not experienced any data loss. We have had a few lost drives over the years but no data lost.
So, to answer your question straight up: I am 100% confident that our backups are good all the way through.
But again, we _test_ those backups with full bare-metal restores (to Hyper-V or physical box).
Oh, and our primary vertical is accounting offices. They touch _everything_ stored on their systems year after year.
Philip
Those wobbly bits lead to a garbage in garbage out situation with the backups as they did not manifest themselves until the server went full-stop.
After a hard-reset the server came up and all was seemingly happy. Then the NTFS 55 errors came along. The she went full-stop again with no recovery.
A perfect storm of events lead to the backups being relatively useless (BackupExec to DAT libraries).
That was the last time we experienced file loss. Out of 650GB of data we recovered everything but one partner's 24 files.
To date we have gone through some very spectacular recoveries in part due to ShadowProtect and in part due to the skills learned via the SwingIT migration techniques (www.sbsmigration.com - best $400 I've ever spent). No data was lost.
Since switching to SAS only disks and hardware RAID we have not experienced any data loss. We have had a few lost drives over the years but no data lost.
So, to answer your question straight up: I am 100% confident that our backups are good all the way through.
But again, we _test_ those backups with full bare-metal restores (to Hyper-V or physical box).
Oh, and our primary vertical is accounting offices. They touch _everything_ stored on their systems year after year.
Philip
great to hear things are working! And yes, I love shadow Protect also. THe examples you give are for restores of the whole hard drive.
I am not saying there's a flaw in SP! I'm just throwing out this idea asking if I am mistaken in thinking there's a conceivable situation, as tiny as it may be, but doesn't it exist that in some situations, you may not be able to recover a file if months after it was deleted people realize it's missing / corrupt. again, not because of a flaw in SP or anything like that. just that by the nature of the daily, weekly, monthly rollups / consolidations, you lose - I guess the word to use here is granularity? within a week or 2 of a problem, you can restore down to a 15 minute window (assuming that's how often you take images). But weeks later, you can only restore down to a certain day. farther out you can only restore a file from the weekly rollups. and then after a longer time, you can only restore down to a monthly image, UNLESS you are saving all the individual snapshots, which over time takes up lots of space.
If you don't mind, just a simple yes or no to the idea that while I readily admit it's (very) unlikely, it is in the realm of possibility? Or am I wrong?
And if it is possible, how, do you minimize that possibility or just acknowledge the limits of the idea of backups.
I am not saying there's a flaw in SP! I'm just throwing out this idea asking if I am mistaken in thinking there's a conceivable situation, as tiny as it may be, but doesn't it exist that in some situations, you may not be able to recover a file if months after it was deleted people realize it's missing / corrupt. again, not because of a flaw in SP or anything like that. just that by the nature of the daily, weekly, monthly rollups / consolidations, you lose - I guess the word to use here is granularity? within a week or 2 of a problem, you can restore down to a 15 minute window (assuming that's how often you take images). But weeks later, you can only restore down to a certain day. farther out you can only restore a file from the weekly rollups. and then after a longer time, you can only restore down to a monthly image, UNLESS you are saving all the individual snapshots, which over time takes up lots of space.
If you don't mind, just a simple yes or no to the idea that while I readily admit it's (very) unlikely, it is in the realm of possibility? Or am I wrong?
And if it is possible, how, do you minimize that possibility or just acknowledge the limits of the idea of backups.
Stick with SAS and hardware RAID then chances are virtually nil.
Bits are bits. They are either there or not.
Zeroes and Ones. ;)
Our hosting partners deal with Petabytes and more. No bits lost there in the years we've been dealing with them.
The longest business relationship we have is about 14 years now. Other than the 24 files listed above not one bit has been missed. :)
That spans IDE, SATA, and now SAS on hardware RAID (was 5 now 6).
Philip
Bits are bits. They are either there or not.
Zeroes and Ones. ;)
Our hosting partners deal with Petabytes and more. No bits lost there in the years we've been dealing with them.
The longest business relationship we have is about 14 years now. Other than the 24 files listed above not one bit has been missed. :)
That spans IDE, SATA, and now SAS on hardware RAID (was 5 now 6).
Philip
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
This is how we structure the backup regimen:
+ Volume Shadow Copy snapshots at 0737, 1037, 1237, 1537, 1737, and 1937
+ ShadowProtect snapshots at 0837, 1137, 1337, 1637, 1837, and 2037
Between those two layers we will catch most everything.
There is no accounting for the user element. Yet, we have managed to keep everything that requires keeping.
And this after users get click happy and delete client files, nest folders elsewhere, and so on.
The Previous Versions tab is a user's best friend. :)
Philip
+ Volume Shadow Copy snapshots at 0737, 1037, 1237, 1537, 1737, and 1937
+ ShadowProtect snapshots at 0837, 1137, 1337, 1637, 1837, and 2037
Between those two layers we will catch most everything.
There is no accounting for the user element. Yet, we have managed to keep everything that requires keeping.
And this after users get click happy and delete client files, nest folders elsewhere, and so on.
The Previous Versions tab is a user's best friend. :)
Philip
Philip - maybe it's me.
You are a legend in my mind - 3rd tier, your lengthy blogs, MVP, etc. I really look to how you do things. I really just wanted to get acknowledgement from you that my feeble mind figured something out. Or that I am wrong - that's fine too.
But I'm not feeling I'm getting a yea or nea (sp?) from you on this.
I'm not questioning how often you are doing the snapshots
(by the way - any significance for you for the number 37 : ) ??
I'm not questioning RAID vs. JBOD, SATA vs. SAS, etc.
That last reply 'we will catch MOST everything'. Most isn't all. I'm NOT trying to rag on a company, procedure, process, etc... Just wanted to know if this thing that dawned on me is accurate.
As the other guys said - yeah, make a file at x:39 and delete it by x+1:33 and it won't be recoverable. But 'worse' - even with the S/F/G or all that other stuff, if you toss the incremental snapshots, there's a chance down the road that a file that existed for days or weeks WILL NOT be on the consolidated files and won't be recoverable. And if you acknowledge that, is there anything you do to try to prevent that (seems to me that keeping all the incrementals forever is the only way) or try to explain that to the client.
Not sure if I wasn't clear or why you didn't give a 'you are right' or 'you are wrong' in my thinking.
You are a legend in my mind - 3rd tier, your lengthy blogs, MVP, etc. I really look to how you do things. I really just wanted to get acknowledgement from you that my feeble mind figured something out. Or that I am wrong - that's fine too.
But I'm not feeling I'm getting a yea or nea (sp?) from you on this.
I'm not questioning how often you are doing the snapshots
(by the way - any significance for you for the number 37 : ) ??
I'm not questioning RAID vs. JBOD, SATA vs. SAS, etc.
That last reply 'we will catch MOST everything'. Most isn't all. I'm NOT trying to rag on a company, procedure, process, etc... Just wanted to know if this thing that dawned on me is accurate.
As the other guys said - yeah, make a file at x:39 and delete it by x+1:33 and it won't be recoverable. But 'worse' - even with the S/F/G or all that other stuff, if you toss the incremental snapshots, there's a chance down the road that a file that existed for days or weeks WILL NOT be on the consolidated files and won't be recoverable. And if you acknowledge that, is there anything you do to try to prevent that (seems to me that keeping all the incrementals forever is the only way) or try to explain that to the client.
Not sure if I wasn't clear or why you didn't give a 'you are right' or 'you are wrong' in my thinking.
Okay, straight up:
No. No data loss. That's 15 years experience speaking.
The 37 is due to folks tending to do things on the hour in groups. The above was an example. In some cases we run SP at :17 and VSC at :37.
Focus on the forest not the tree's branches.
Philip
No. No data loss. That's 15 years experience speaking.
The 37 is due to folks tending to do things on the hour in groups. The above was an example. In some cases we run SP at :17 and VSC at :37.
Focus on the forest not the tree's branches.
Philip
So, we grandfather, grandmother, and in some cases father the backup drives (6, 12, 24 months).
This means that our rotations must include enough drives to cover the regular rotations (2 sometimes 3 sets) and then the archive sets.
The data is encrypted so we don't worry too much about the drives sitting in our vault. :)
Philip