ms 2016 server stuck in merge

we have an ongoing issue with cloudcare where we cannot log on to our vm servers.
we have 2 vms, 1 our 2016 dc and the other our 2016 exchange server.

once we get into this “lock out” situation we try closing down the servers but they just hang, so the only way we have been able to resolve this is to hard reset the server.

we have just done this again and now both vms are stuck starting up, where the status shows "merge in progress", but the progress is just sat at 26% for our dc and 7% for our exchange server.

we really need these servers back online, so does anyone know how we can tidy this up and get them operational?

any help much appreciated - desperate!!

thanks
LVL 1
nigelbeatsonAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

nigelbeatsonAuthor Commented:
i forgot to mention we do have a checkpoint for both vm's and wondered if i applied them, we could get out of this startup / merge situation?

they are both 3 weeks old, but having never carried this out before, wanted to check whether this would help or not and make sure it only changed the os and not the data?

how about exchange? would we still keep the current data in that too?

any help appreciated.

thanks
Patrick BogersDatacenter platform engineer LindowsCommented:
Hi,

It sounds like the checkpoint is being merged and since you say it is three weeks old this can take serious time if it is a high load server. IMHO you have to wait.
Advise, never keep checkpoints this long, they are no means as a backup.

Cheers.
nigelbeatsonAuthor Commented:
we took this when we started experiencing the problem.

our problem is that the servers are NOT starting. they are in startup mode, ie we get the windows logo and the progress bar, but the servers just stay that way.

we need these back now!

help!
5 Ways Acronis Skyrockets Your Data Protection

Risks to data security are risks to business continuity. Businesses need to know what these risks look like – and where they can turn for help.
Check our newest E-Book and learn how you can differentiate your data protection business with advanced cloud solutions Acronis delivers

nigelbeatsonAuthor Commented:
why would the servers just decide to merge an ancient checkpoint? i never asked them to??
Patrick BogersDatacenter platform engineer LindowsCommented:
Hi,

Best practive is to let hyper-v deal with it but if you are sure it cannot you can shutndown the vms and try powershell
Get-VMSnapshot -VMName | Remove-VMSnapshot

Cheers

Edit: keeping snapshots that long is a very good recipy for data corruption as you are finding out now. This automerge, as it seems, sound like Hyper-V has decided it has no use for the snapshot so it merges it. Merge does not mean it is rolling back the vm, it just means the current instance is becoming the soul file for the VM.
nigelbeatsonAuthor Commented:
unfortunately, im not sure of anything anymore! all i can say is that both servers are just sat starting up and the status in the hyperv manager shows the merge taking place. not sure why they are stuck at the same progress point.

i have tried running powershell on the hyperv host but although the cmd window opens, we just get a blinking cursor!

should i call microsoft?
Patrick BogersDatacenter platform engineer LindowsCommented:
If you have support that would be a good option yes.
nigelbeatsonAuthor Commented:
we have to pay! so if you think they would be the best people to resolve this, i will have to do it.

i cant (for some reason) get powershell to start all the way, we just get a blinking cursor!
Patrick BogersDatacenter platform engineer LindowsCommented:
Thats very strange, sounds like your host is in distress for resources. I cannot imagine but let me ask anyway, there are no AV or antimalware apps on the hyperv host right? Those could lock up the vm files.

Btw, Have you tried rebooting the host completely?

I would call cloudcare and ask if they know why this server is behaving poorly.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Click start on the host and type: ResMon.exe [CTRL+SHFT+ENTER] --> UAC --> Disk tab --> Disk Activity dropdown --> Sort largest Total (B/Sec)

Snip that and post please. The disk subsystem is probably maxed while the merges are running.

Please be patient.

And, it's better to use backup as a fallback than snapshots/checkpoints as the latter soak up free disk space as they grow!
nigelbeatsonAuthor Commented:
Thanks Patrick, yes we do have anti virus and malware on our host, but we did exclude key folders and files. Are you suggesting that we should not run this on the base host?

I did reboot the host, but it ended up being the big hard reset. The problem was that when we get locked out, when we try close down, it hangs when closing down, when we select reset, it hangs on resetting. The only option I was aware of was to power off the server. This is when we end up with the merging problems.

I did eventually reset (power off the server) as you suggested and it did then restart and the merge completed in a timely fashion. I have now deleted the checkpoints, and was frustrated to see it set about yet another merge!! Aaaggh! This is now completed and we have restarted all fully, and all seems to be well.

We do run a full system backup for each vm, but due to the problems with the server going down constantly, we only have one that is 5 days old. We have reported this to cloudcare nearly a month ago, and I am STILL waiting for any feedback from their development team. In the meantime, I have the choice of running the server without protection, or run protection and have to reset the server constantly! So frustrating!!

Many thanks to all.
nigelbeatsonAuthor Commented:
Many thanks to all who tried to assist.
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
What's "CloudCare" please? There are _lots_ of results that come up.
Patrick BogersDatacenter platform engineer LindowsCommented:
Hi again,

Glad to hear you are back up and running and yes, if the hyperV host is not serving frontend requests so basically is not reachable from the outside i would leave AV off.
Protection on the vms themselfes is another story.

Cheers!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Hyper-V

From novice to tech pro — start learning today.