Windows Server Backup Issues

I have a customer who is backing up almost 4 TB of data through Windows Server Backup (Server 2012) onto a 12 TB external hard drive.  Yes, I know there are better methods they should use for that amount of data, but I haven't talked them into that, and technically this method should work even if it isn't the best choice.  So recently the backups have not been going well.  If the server has just been rebooted, the backup will go ok, but will take 2 1/2 days - as a new full backup.  Then when it's time for the next backup, the System event viewer will fill up with VHDmp errors (129), and either the server will completely freeze, or at least the backup will stall.  And sometimes on reboot the server won't come all the way back up. although if you then hard reboot it , it will boot normally.  So it' s my understanding that the VHDmp errors are related to virtual drives.  But this is a physical server, not a virtual one.  However, I have found that while the backup is running, if you go into disk management, it shows a virtual drive that is a replica of the drive that is being backed up.  It shows up while the backup is running and goes away when it is done.  So I think that is all related.  But I am at a total loss how to fix this.  Once a full backup is done, Windows Server Backup should just automatically do incremental backups, at least for 14 days.  And I've never seen a system create a virtual replica of the volume being backed up (although I must admit I don't often go into disk management while a backup is going, so maybe it happens...).  The system has just recently been reloaded on this server, so it should be pretty clean.  Has anyone ever run into this?   Any ideas would be appreciated!  Thanks!
landiiiks2Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Cliff GaliherCommented:
If your backup includes the system state then incrementals are disabled. Yes the backup engine has always used a virtual disk and shadow copies for backups.

So two suggestions. Use USB3, thunderbolt, or eSATA drives with an appropriate server interface. USB2 just won't cut it. The drive can't keep up and merging shadow copies will usually fail.

Second, split up your backup goals. For example, you can back up the system drive and system state far less often than data partitions. Archive data can also he backed up far less often. Multiple backup jobs with less data is almost always better.
0
landiiiks2Author Commented:
Thank you and I agree with your suggestions.  However, I haven't been able to talk them into changing the hardware or basic methodology.  And the point is that this actually was working in the past.  Not the best method, we all agree, but it was working.  And right now I am not including the system state for the reasons you mention.
0
Cliff GaliherCommented:
Sounds like your data set just grew too large for your hardware. That's why it used to work and now doesn't.  You outgrew it.  I used to fit into thinner slacks too....
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

landiiiks2Author Commented:
Haha, Cliff!  Me too!  But it was working a week ago and the amount of data hasn't changed in that time....   Like I said, I know this isn't the best solution, but I think there has to be something more to it.  And I've gone into disk management on a couple other servers while backup was running and I didn't see this second, virtual copy of the disk.  So I think that has to be a clue.
0
Cliff GaliherCommented:
It really really isn't. The drive only exists while the backup copies the shadow copy. A backup first *creates* the shadow copy. It doesn't mount the drive until that part is done. In most backups', that means the window of time that the drive is mounted is small compared to the shadow copy portion. But with your huge dataset, you have kicu more time to "see" behind the curtain.

Windows does cleanup processes regularly. When a backup takes too long, they collide. Anything longer that 24 hours is actually bad. I'm surprised you haven't had problems before now. But a 2.5 day backup? Yeah, the errors you report are exactly how I'd expect a backup taking that long to fail.

But hey, if you like punching brick walls, you can keep trying to "fix" something that isn't fixable (without actually making change...)

Good luck.
0
Gerald ConnollyCommented:
2.5 days to backup 4TB, that's really really slow! I would have thought max 24 hours.
0
landiiiks2Author Commented:
I appreciate all the advice.  I misspoke a bit - it's actually less than two days, but regardless, I know it is slow and I know it isn't the best way to do this.  I'm just trying to get by until I can convince the customer to spend the money for a better solution.  And my point is that this was working a week ago.  The amount of data has not changed since then.  I was not getting the vhdmp errors then, the backups were completing, and now I'm getting the errors and the server is freezing and/or not completing the backups.  I simply want to have the server do the best it can to complete backups until I have an alternate solution in place.  I was just hoping for someone to tell me exactly what might be causing the errors to occur (not just that it's too much data - it was too much data two weeks ago).  And maybe point me in the direction of a solution (one that will make it go back to working the way it was two weeks ago) until I can get something else in place.
0
noxchoGlobal Support CoordinatorCommented:
Hi landiiiks2,

The fact that you see a vhd container mounted in Window Disk Management is absolutely normal. This is how Windows Server Backup works. It creates an empty vhd container, mounts it and then starts copying the data from source drive to it. Once backup is finished the vhd container is unmounted.
Now to the problem.
Did you customer recently install or activate any third party backup tool on this machine? Backup Exec, Acronis or something else? Go to the services and check it.
In the Update History check what exact update was installed at the day or day before when this problem occurred first time.
It can be caused by driver update or even by antivirus drivers.
0
Gerald ConnollyCommented:
@noxcho - re software installation/driver updates etc, that was the first thing that came to mind after reading the update from the author.
0
landiiiks2Author Commented:
The antivirus is McAfee and has been on there all along.  No other backup software is on there (although I've been thinking about trying Acronis to see if it works better).  The last Windows updates were almost two weeks before the problem started.  Sigh.  Good ideas, but unfortunately none seem to be the culprit.
0
noxchoGlobal Support CoordinatorCommented:
So when exactly did this problem occur first time? You know that problems do not happen on their own.
0
landiiiks2Author Commented:
It started a week and a half ago.  Nothing new installed (this server is a domain controller and a file server, so there isn't a lot of software on it in the first place).  No updates installed since October 11.  No detectable viruses or malware.  One other note, for whatever it's worth.  The external hard drive is 12 TB and the total amount of data backed up is a little less than 4 TB.  When I run a backup after a reboot it runs fine, it takes a long time as I said, but completes successfully.  Then any additional backups stall out with the vhdmp error.  If I reboot, as I did this weekend, and run another backup it completes successfully but does it as a full backup and apparently overwrites the previous successful one.  To me this is not expected behavior.  The subsequent backups should be automatically incremental (at least for 14 days).  And even if another full one is performed, there should be plenty of space on the hard drive to hold both of them.  So in other words, although I had a successful backup on the 4th and another one ending last night, the backup software only shows that the hard drive is holding one copy of the backup.
0
noxchoGlobal Support CoordinatorCommented:
Can you format the external drive on another system, connect it back to this server then delete all events from MS Server Backup and create new backup policy?
Check then if the problem reproduces.
BTW, does this server see this drive as internal or just like a share?
0
landiiiks2Author Commented:
Another good idea - but I tried that early on.  It didn't make a difference.  I tried a different external drive also....
0
noxchoGlobal Support CoordinatorCommented:
Ok, then let's try to narrow the problem.
Can you try to backup partitions one by one? I mean one partition per run. Does it fail on each partition?
0
landiiiks2Author Commented:
It's only backing up one partition....  :(
0
noxchoGlobal Support CoordinatorCommented:
And if you try other partitions?
0
landiiiks2Author Commented:
I deleted the backup job and created a new one that included only one folder with a total of 27.55 GB.  I ran the backup.  It deleted my only full backup I had on there (which I was afraid of) and successfully backed up the new, small backup.  But every time the first backup goes fine - it's the subsequent ones that fail.  So I started a second backup.  This time it actually did start out ok and got to 25%.  Then the VHDMP errors started again and it is sitting there spinning and not progressing.  So now there is no question that the problem is NOT the size of the backup.  This failed after backing up 7 GB of 27 GB.  I really need an answer if anyone has any ideas.
0
noxchoGlobal Support CoordinatorCommented:
Can you confirm that this problem persists also when you take the backup to an internal drive? So that we exclude the USB drive from suspect list.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
landiiiks2Author Commented:
I did run a backup  (the small 27 GB one) to the internal drive, and it completed successfully (and fast) and so did subsequent incremental backups - they properly acted as incrementals and didn't replace the previous backup.  I had alternately been using the two USB ports in the front of the server.  I moved the external drive to one of the USB ports in the back, and tried the small backup.  It completed successfully, and so did subsequent backups.  So over the weekend I ran the big backup using the rear USB port and it completed in about 30 hours.  And then the subsequent ones completed fine as well - no VHDMP errors and no hangs!  They completed in about 10 minutes, and incremented the number of backups instead of replacing the existing one.  So all is acting normally and I am very relieved!  The problem all along was two bad USB ports in the front.  I might have caught on quicker if it had only been one, but after I switched ports and had the same issue I moved on to other potential causes.  Thank you all for your help.  I guess the real moral of the story is to not necessarily get hung up and focused on what seems to be the most obvious answer (the size of the backup).
0
Gerald ConnollyCommented:
We have had discussions on EE before about connecter wear. Most connectors are rated for n insertion events plugging-in and removing counts as two events and its down to the thickness of the gold plated on the connector.

Disk connectors are usually rated at a few 10's of events, whereas USB should be rated at a few 100's to 1000.

Obviously if the motherboard gets excessive wear its an issue, and i have always recommended that a short extension tail be used so that the motherboard has a few insertion events, and all that is needed is a replacement cable (tail) if the connection becomes unreliable.

It may be that a brand new cable would have fixed your original problem?
0
landiiiks2Author Commented:
That's a good point, but the external hard drive normally just stays plugged in so there are very few insertion events.  I did change external hard drives in the process of troubleshooting, and did switch USB ports once, but otherwise everything stayed put.  But also, since there were two external hard drives involved, one brand new, there were also two cables used at different points during this whole process.
0
noxchoGlobal Support CoordinatorCommented:
Cool, glad to hear you were able to get it fixed finally. Very often the problem can have simple cause than we think. That's why I use this "narrow-to-bottleneck" before getting onto a complicated track of problem solving.

Take care
Nox
1
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2012

From novice to tech pro — start learning today.