Solved

Windows Server Backup Issues

Posted on 2016-11-05
23
144 Views
Last Modified: 2016-11-14
I have a customer who is backing up almost 4 TB of data through Windows Server Backup (Server 2012) onto a 12 TB external hard drive.  Yes, I know there are better methods they should use for that amount of data, but I haven't talked them into that, and technically this method should work even if it isn't the best choice.  So recently the backups have not been going well.  If the server has just been rebooted, the backup will go ok, but will take 2 1/2 days - as a new full backup.  Then when it's time for the next backup, the System event viewer will fill up with VHDmp errors (129), and either the server will completely freeze, or at least the backup will stall.  And sometimes on reboot the server won't come all the way back up. although if you then hard reboot it , it will boot normally.  So it' s my understanding that the VHDmp errors are related to virtual drives.  But this is a physical server, not a virtual one.  However, I have found that while the backup is running, if you go into disk management, it shows a virtual drive that is a replica of the drive that is being backed up.  It shows up while the backup is running and goes away when it is done.  So I think that is all related.  But I am at a total loss how to fix this.  Once a full backup is done, Windows Server Backup should just automatically do incremental backups, at least for 14 days.  And I've never seen a system create a virtual replica of the volume being backed up (although I must admit I don't often go into disk management while a backup is going, so maybe it happens...).  The system has just recently been reloaded on this server, so it should be pretty clean.  Has anyone ever run into this?   Any ideas would be appreciated!  Thanks!
0
Comment
Question by:landiiiks2
  • 10
  • 7
  • 3
  • +1
23 Comments
 
LVL 58

Expert Comment

by:Cliff Galiher
ID: 41875816
If your backup includes the system state then incrementals are disabled. Yes the backup engine has always used a virtual disk and shadow copies for backups.

So two suggestions. Use USB3, thunderbolt, or eSATA drives with an appropriate server interface. USB2 just won't cut it. The drive can't keep up and merging shadow copies will usually fail.

Second, split up your backup goals. For example, you can back up the system drive and system state far less often than data partitions. Archive data can also he backed up far less often. Multiple backup jobs with less data is almost always better.
0
 

Author Comment

by:landiiiks2
ID: 41875819
Thank you and I agree with your suggestions.  However, I haven't been able to talk them into changing the hardware or basic methodology.  And the point is that this actually was working in the past.  Not the best method, we all agree, but it was working.  And right now I am not including the system state for the reasons you mention.
0
 
LVL 58

Expert Comment

by:Cliff Galiher
ID: 41875821
Sounds like your data set just grew too large for your hardware. That's why it used to work and now doesn't.  You outgrew it.  I used to fit into thinner slacks too....
0
Ransomware: The New Cyber Threat & How to Stop It

This infographic explains ransomware, type of malware that blocks access to your files or your systems and holds them hostage until a ransom is paid. It also examines the different types of ransomware and explains what you can do to thwart this sinister online threat.  

 

Author Comment

by:landiiiks2
ID: 41875823
Haha, Cliff!  Me too!  But it was working a week ago and the amount of data hasn't changed in that time....   Like I said, I know this isn't the best solution, but I think there has to be something more to it.  And I've gone into disk management on a couple other servers while backup was running and I didn't see this second, virtual copy of the disk.  So I think that has to be a clue.
0
 
LVL 58

Expert Comment

by:Cliff Galiher
ID: 41875826
It really really isn't. The drive only exists while the backup copies the shadow copy. A backup first *creates* the shadow copy. It doesn't mount the drive until that part is done. In most backups', that means the window of time that the drive is mounted is small compared to the shadow copy portion. But with your huge dataset, you have kicu more time to "see" behind the curtain.

Windows does cleanup processes regularly. When a backup takes too long, they collide. Anything longer that 24 hours is actually bad. I'm surprised you haven't had problems before now. But a 2.5 day backup? Yeah, the errors you report are exactly how I'd expect a backup taking that long to fail.

But hey, if you like punching brick walls, you can keep trying to "fix" something that isn't fixable (without actually making change...)

Good luck.
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 41876135
2.5 days to backup 4TB, that's really really slow! I would have thought max 24 hours.
0
 

Author Comment

by:landiiiks2
ID: 41876248
I appreciate all the advice.  I misspoke a bit - it's actually less than two days, but regardless, I know it is slow and I know it isn't the best way to do this.  I'm just trying to get by until I can convince the customer to spend the money for a better solution.  And my point is that this was working a week ago.  The amount of data has not changed since then.  I was not getting the vhdmp errors then, the backups were completing, and now I'm getting the errors and the server is freezing and/or not completing the backups.  I simply want to have the server do the best it can to complete backups until I have an alternate solution in place.  I was just hoping for someone to tell me exactly what might be causing the errors to occur (not just that it's too much data - it was too much data two weeks ago).  And maybe point me in the direction of a solution (one that will make it go back to working the way it was two weeks ago) until I can get something else in place.
0
 
LVL 47

Expert Comment

by:noxcho
ID: 41876282
Hi landiiiks2,

The fact that you see a vhd container mounted in Window Disk Management is absolutely normal. This is how Windows Server Backup works. It creates an empty vhd container, mounts it and then starts copying the data from source drive to it. Once backup is finished the vhd container is unmounted.
Now to the problem.
Did you customer recently install or activate any third party backup tool on this machine? Backup Exec, Acronis or something else? Go to the services and check it.
In the Update History check what exact update was installed at the day or day before when this problem occurred first time.
It can be caused by driver update or even by antivirus drivers.
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 41876486
@noxcho - re software installation/driver updates etc, that was the first thing that came to mind after reading the update from the author.
0
 

Author Comment

by:landiiiks2
ID: 41876526
The antivirus is McAfee and has been on there all along.  No other backup software is on there (although I've been thinking about trying Acronis to see if it works better).  The last Windows updates were almost two weeks before the problem started.  Sigh.  Good ideas, but unfortunately none seem to be the culprit.
0
 
LVL 47

Expert Comment

by:noxcho
ID: 41876633
So when exactly did this problem occur first time? You know that problems do not happen on their own.
0
 

Author Comment

by:landiiiks2
ID: 41877209
It started a week and a half ago.  Nothing new installed (this server is a domain controller and a file server, so there isn't a lot of software on it in the first place).  No updates installed since October 11.  No detectable viruses or malware.  One other note, for whatever it's worth.  The external hard drive is 12 TB and the total amount of data backed up is a little less than 4 TB.  When I run a backup after a reboot it runs fine, it takes a long time as I said, but completes successfully.  Then any additional backups stall out with the vhdmp error.  If I reboot, as I did this weekend, and run another backup it completes successfully but does it as a full backup and apparently overwrites the previous successful one.  To me this is not expected behavior.  The subsequent backups should be automatically incremental (at least for 14 days).  And even if another full one is performed, there should be plenty of space on the hard drive to hold both of them.  So in other words, although I had a successful backup on the 4th and another one ending last night, the backup software only shows that the hard drive is holding one copy of the backup.
0
 
LVL 47

Expert Comment

by:noxcho
ID: 41877960
Can you format the external drive on another system, connect it back to this server then delete all events from MS Server Backup and create new backup policy?
Check then if the problem reproduces.
BTW, does this server see this drive as internal or just like a share?
0
 

Author Comment

by:landiiiks2
ID: 41878054
Another good idea - but I tried that early on.  It didn't make a difference.  I tried a different external drive also....
0
 
LVL 47

Assisted Solution

by:noxcho
noxcho earned 500 total points
ID: 41878803
Ok, then let's try to narrow the problem.
Can you try to backup partitions one by one? I mean one partition per run. Does it fail on each partition?
0
 

Author Comment

by:landiiiks2
ID: 41878807
It's only backing up one partition....  :(
0
 
LVL 47

Expert Comment

by:noxcho
ID: 41878863
And if you try other partitions?
0
 

Author Comment

by:landiiiks2
ID: 41882979
I deleted the backup job and created a new one that included only one folder with a total of 27.55 GB.  I ran the backup.  It deleted my only full backup I had on there (which I was afraid of) and successfully backed up the new, small backup.  But every time the first backup goes fine - it's the subsequent ones that fail.  So I started a second backup.  This time it actually did start out ok and got to 25%.  Then the VHDMP errors started again and it is sitting there spinning and not progressing.  So now there is no question that the problem is NOT the size of the backup.  This failed after backing up 7 GB of 27 GB.  I really need an answer if anyone has any ideas.
0
 
LVL 47

Accepted Solution

by:
noxcho earned 500 total points
ID: 41883060
Can you confirm that this problem persists also when you take the backup to an internal drive? So that we exclude the USB drive from suspect list.
0
 

Author Comment

by:landiiiks2
ID: 41885810
I did run a backup  (the small 27 GB one) to the internal drive, and it completed successfully (and fast) and so did subsequent incremental backups - they properly acted as incrementals and didn't replace the previous backup.  I had alternately been using the two USB ports in the front of the server.  I moved the external drive to one of the USB ports in the back, and tried the small backup.  It completed successfully, and so did subsequent backups.  So over the weekend I ran the big backup using the rear USB port and it completed in about 30 hours.  And then the subsequent ones completed fine as well - no VHDMP errors and no hangs!  They completed in about 10 minutes, and incremented the number of backups instead of replacing the existing one.  So all is acting normally and I am very relieved!  The problem all along was two bad USB ports in the front.  I might have caught on quicker if it had only been one, but after I switched ports and had the same issue I moved on to other potential causes.  Thank you all for your help.  I guess the real moral of the story is to not necessarily get hung up and focused on what seems to be the most obvious answer (the size of the backup).
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 41885833
We have had discussions on EE before about connecter wear. Most connectors are rated for n insertion events plugging-in and removing counts as two events and its down to the thickness of the gold plated on the connector.

Disk connectors are usually rated at a few 10's of events, whereas USB should be rated at a few 100's to 1000.

Obviously if the motherboard gets excessive wear its an issue, and i have always recommended that a short extension tail be used so that the motherboard has a few insertion events, and all that is needed is a replacement cable (tail) if the connection becomes unreliable.

It may be that a brand new cable would have fixed your original problem?
0
 

Author Comment

by:landiiiks2
ID: 41885841
That's a good point, but the external hard drive normally just stays plugged in so there are very few insertion events.  I did change external hard drives in the process of troubleshooting, and did switch USB ports once, but otherwise everything stayed put.  But also, since there were two external hard drives involved, one brand new, there were also two cables used at different points during this whole process.
0
 
LVL 47

Expert Comment

by:noxcho
ID: 41886072
Cool, glad to hear you were able to get it fixed finally. Very often the problem can have simple cause than we think. That's why I use this "narrow-to-bottleneck" before getting onto a complicated track of problem solving.

Take care
Nox
1

Featured Post

Are your AD admin tools letting you down?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
A procedure for exporting installed hotfix details of remote computers using powershell
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This tutorial will walk an individual through the process of configuring their Windows Server 2012 domain controller to synchronize its time with a trusted, external resource. Use Google, Bing, or other preferred search engine to locate trusted NTP …

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question