Solved

Windows Server Backup Issues

Posted on 2016-11-05
23
95 Views
Last Modified: 2016-11-14
I have a customer who is backing up almost 4 TB of data through Windows Server Backup (Server 2012) onto a 12 TB external hard drive.  Yes, I know there are better methods they should use for that amount of data, but I haven't talked them into that, and technically this method should work even if it isn't the best choice.  So recently the backups have not been going well.  If the server has just been rebooted, the backup will go ok, but will take 2 1/2 days - as a new full backup.  Then when it's time for the next backup, the System event viewer will fill up with VHDmp errors (129), and either the server will completely freeze, or at least the backup will stall.  And sometimes on reboot the server won't come all the way back up. although if you then hard reboot it , it will boot normally.  So it' s my understanding that the VHDmp errors are related to virtual drives.  But this is a physical server, not a virtual one.  However, I have found that while the backup is running, if you go into disk management, it shows a virtual drive that is a replica of the drive that is being backed up.  It shows up while the backup is running and goes away when it is done.  So I think that is all related.  But I am at a total loss how to fix this.  Once a full backup is done, Windows Server Backup should just automatically do incremental backups, at least for 14 days.  And I've never seen a system create a virtual replica of the volume being backed up (although I must admit I don't often go into disk management while a backup is going, so maybe it happens...).  The system has just recently been reloaded on this server, so it should be pretty clean.  Has anyone ever run into this?   Any ideas would be appreciated!  Thanks!
0
Comment
Question by:landiiiks2
  • 10
  • 7
  • 3
  • +1
23 Comments
 
LVL 56

Expert Comment

by:Cliff Galiher
Comment Utility
If your backup includes the system state then incrementals are disabled. Yes the backup engine has always used a virtual disk and shadow copies for backups.

So two suggestions. Use USB3, thunderbolt, or eSATA drives with an appropriate server interface. USB2 just won't cut it. The drive can't keep up and merging shadow copies will usually fail.

Second, split up your backup goals. For example, you can back up the system drive and system state far less often than data partitions. Archive data can also he backed up far less often. Multiple backup jobs with less data is almost always better.
0
 

Author Comment

by:landiiiks2
Comment Utility
Thank you and I agree with your suggestions.  However, I haven't been able to talk them into changing the hardware or basic methodology.  And the point is that this actually was working in the past.  Not the best method, we all agree, but it was working.  And right now I am not including the system state for the reasons you mention.
0
 
LVL 56

Expert Comment

by:Cliff Galiher
Comment Utility
Sounds like your data set just grew too large for your hardware. That's why it used to work and now doesn't.  You outgrew it.  I used to fit into thinner slacks too....
0
 

Author Comment

by:landiiiks2
Comment Utility
Haha, Cliff!  Me too!  But it was working a week ago and the amount of data hasn't changed in that time....   Like I said, I know this isn't the best solution, but I think there has to be something more to it.  And I've gone into disk management on a couple other servers while backup was running and I didn't see this second, virtual copy of the disk.  So I think that has to be a clue.
0
 
LVL 56

Expert Comment

by:Cliff Galiher
Comment Utility
It really really isn't. The drive only exists while the backup copies the shadow copy. A backup first *creates* the shadow copy. It doesn't mount the drive until that part is done. In most backups', that means the window of time that the drive is mounted is small compared to the shadow copy portion. But with your huge dataset, you have kicu more time to "see" behind the curtain.

Windows does cleanup processes regularly. When a backup takes too long, they collide. Anything longer that 24 hours is actually bad. I'm surprised you haven't had problems before now. But a 2.5 day backup? Yeah, the errors you report are exactly how I'd expect a backup taking that long to fail.

But hey, if you like punching brick walls, you can keep trying to "fix" something that isn't fixable (without actually making change...)

Good luck.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
Comment Utility
2.5 days to backup 4TB, that's really really slow! I would have thought max 24 hours.
0
 

Author Comment

by:landiiiks2
Comment Utility
I appreciate all the advice.  I misspoke a bit - it's actually less than two days, but regardless, I know it is slow and I know it isn't the best way to do this.  I'm just trying to get by until I can convince the customer to spend the money for a better solution.  And my point is that this was working a week ago.  The amount of data has not changed since then.  I was not getting the vhdmp errors then, the backups were completing, and now I'm getting the errors and the server is freezing and/or not completing the backups.  I simply want to have the server do the best it can to complete backups until I have an alternate solution in place.  I was just hoping for someone to tell me exactly what might be causing the errors to occur (not just that it's too much data - it was too much data two weeks ago).  And maybe point me in the direction of a solution (one that will make it go back to working the way it was two weeks ago) until I can get something else in place.
0
 
LVL 46

Expert Comment

by:noxcho
Comment Utility
Hi landiiiks2,

The fact that you see a vhd container mounted in Window Disk Management is absolutely normal. This is how Windows Server Backup works. It creates an empty vhd container, mounts it and then starts copying the data from source drive to it. Once backup is finished the vhd container is unmounted.
Now to the problem.
Did you customer recently install or activate any third party backup tool on this machine? Backup Exec, Acronis or something else? Go to the services and check it.
In the Update History check what exact update was installed at the day or day before when this problem occurred first time.
It can be caused by driver update or even by antivirus drivers.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
Comment Utility
@noxcho - re software installation/driver updates etc, that was the first thing that came to mind after reading the update from the author.
0
 

Author Comment

by:landiiiks2
Comment Utility
The antivirus is McAfee and has been on there all along.  No other backup software is on there (although I've been thinking about trying Acronis to see if it works better).  The last Windows updates were almost two weeks before the problem started.  Sigh.  Good ideas, but unfortunately none seem to be the culprit.
0
 
LVL 46

Expert Comment

by:noxcho
Comment Utility
So when exactly did this problem occur first time? You know that problems do not happen on their own.
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 

Author Comment

by:landiiiks2
Comment Utility
It started a week and a half ago.  Nothing new installed (this server is a domain controller and a file server, so there isn't a lot of software on it in the first place).  No updates installed since October 11.  No detectable viruses or malware.  One other note, for whatever it's worth.  The external hard drive is 12 TB and the total amount of data backed up is a little less than 4 TB.  When I run a backup after a reboot it runs fine, it takes a long time as I said, but completes successfully.  Then any additional backups stall out with the vhdmp error.  If I reboot, as I did this weekend, and run another backup it completes successfully but does it as a full backup and apparently overwrites the previous successful one.  To me this is not expected behavior.  The subsequent backups should be automatically incremental (at least for 14 days).  And even if another full one is performed, there should be plenty of space on the hard drive to hold both of them.  So in other words, although I had a successful backup on the 4th and another one ending last night, the backup software only shows that the hard drive is holding one copy of the backup.
0
 
LVL 46

Expert Comment

by:noxcho
Comment Utility
Can you format the external drive on another system, connect it back to this server then delete all events from MS Server Backup and create new backup policy?
Check then if the problem reproduces.
BTW, does this server see this drive as internal or just like a share?
0
 

Author Comment

by:landiiiks2
Comment Utility
Another good idea - but I tried that early on.  It didn't make a difference.  I tried a different external drive also....
0
 
LVL 46

Assisted Solution

by:noxcho
noxcho earned 500 total points
Comment Utility
Ok, then let's try to narrow the problem.
Can you try to backup partitions one by one? I mean one partition per run. Does it fail on each partition?
0
 

Author Comment

by:landiiiks2
Comment Utility
It's only backing up one partition....  :(
0
 
LVL 46

Expert Comment

by:noxcho
Comment Utility
And if you try other partitions?
0
 

Author Comment

by:landiiiks2
Comment Utility
I deleted the backup job and created a new one that included only one folder with a total of 27.55 GB.  I ran the backup.  It deleted my only full backup I had on there (which I was afraid of) and successfully backed up the new, small backup.  But every time the first backup goes fine - it's the subsequent ones that fail.  So I started a second backup.  This time it actually did start out ok and got to 25%.  Then the VHDMP errors started again and it is sitting there spinning and not progressing.  So now there is no question that the problem is NOT the size of the backup.  This failed after backing up 7 GB of 27 GB.  I really need an answer if anyone has any ideas.
0
 
LVL 46

Accepted Solution

by:
noxcho earned 500 total points
Comment Utility
Can you confirm that this problem persists also when you take the backup to an internal drive? So that we exclude the USB drive from suspect list.
0
 

Author Comment

by:landiiiks2
Comment Utility
I did run a backup  (the small 27 GB one) to the internal drive, and it completed successfully (and fast) and so did subsequent incremental backups - they properly acted as incrementals and didn't replace the previous backup.  I had alternately been using the two USB ports in the front of the server.  I moved the external drive to one of the USB ports in the back, and tried the small backup.  It completed successfully, and so did subsequent backups.  So over the weekend I ran the big backup using the rear USB port and it completed in about 30 hours.  And then the subsequent ones completed fine as well - no VHDMP errors and no hangs!  They completed in about 10 minutes, and incremented the number of backups instead of replacing the existing one.  So all is acting normally and I am very relieved!  The problem all along was two bad USB ports in the front.  I might have caught on quicker if it had only been one, but after I switched ports and had the same issue I moved on to other potential causes.  Thank you all for your help.  I guess the real moral of the story is to not necessarily get hung up and focused on what seems to be the most obvious answer (the size of the backup).
0
 
LVL 16

Expert Comment

by:Gerald Connolly
Comment Utility
We have had discussions on EE before about connecter wear. Most connectors are rated for n insertion events plugging-in and removing counts as two events and its down to the thickness of the gold plated on the connector.

Disk connectors are usually rated at a few 10's of events, whereas USB should be rated at a few 100's to 1000.

Obviously if the motherboard gets excessive wear its an issue, and i have always recommended that a short extension tail be used so that the motherboard has a few insertion events, and all that is needed is a replacement cable (tail) if the connection becomes unreliable.

It may be that a brand new cable would have fixed your original problem?
0
 

Author Comment

by:landiiiks2
Comment Utility
That's a good point, but the external hard drive normally just stays plugged in so there are very few insertion events.  I did change external hard drives in the process of troubleshooting, and did switch USB ports once, but otherwise everything stayed put.  But also, since there were two external hard drives involved, one brand new, there were also two cables used at different points during this whole process.
0
 
LVL 46

Expert Comment

by:noxcho
Comment Utility
Cool, glad to hear you were able to get it fixed finally. Very often the problem can have simple cause than we think. That's why I use this "narrow-to-bottleneck" before getting onto a complicated track of problem solving.

Take care
Nox
1

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

VM backups can be lost due to a number of reasons: accidental backup deletion, backup file corruption, disk failure, lost or stolen hardware, malicious attack, or due to some other undesired and unpredicted event. Thus, having more than one copy of …
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This tutorial will walk an individual through the process of configuring their Windows Server 2012 domain controller to synchronize its time with a trusted, external resource. Use Google, Bing, or other preferred search engine to locate trusted NTP …

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now