VMware Emergency - Domain Controller issue

OUR DC has been running perfectly for months and months with no issues.
Today we found the server to be very sluggish and attempted to reboot it via RDP

After about 45mins I managed to get to restart - many services are failing and still sluggish.
I fear I may be recovering from backup
I dont know if the issue is OS related or VM or both.

I see that there are snapshots now in the datastore?   I dont fully understand how the vm manages its disks etc,  I just know that more files are in the store now with .000001 etc for both our C and D disks.
What is happening here?
I copied the original vdmk to another esxi server we have to rule out hardware etc
It booted up but was not current!!
Please help!
LVL 1
j-teksolutionsAsked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
Okay, proceed as follows:-

Hit the DELETE ALL Button, and WAIT and Be Patient, whilst the Snapshots are deleted.

this could take days, hours, minutes or seconds, depending upon the speed of your datastore.

Do not be tempted to do anything to the VM, shutdown, restart, play, fiddle with it of anyrthing, just wait......
0
 
j-teksolutionsAuthor Commented:
Worst case scenario:  

1. I had Backup Exec r3 running in the vm and its current.
2.  Current backup of AD,  SQL,  Exchange
3.  I have a clean full backup of the vms from OCT 2012

Never done an AD restore etc with a vm hoping not to.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
It's possible you have a snapshot on your VM, because when VMs run slow, due to high CPU it's because they are running on a snapshot disk.

Check my EE Article and confirm you are running on a snapshot:-

HOW TO: VMware Snapshots :- Be Patient

PLease upload a screenshot of your datastore, and I'll work with you to get out of this snapshot mess.

Please be patient, and calm, do not fiddle, and follow my instructions.


doing the following may have caused MORE ISSUES!
I copied the original vdmk to another esxi server we have to rule out hardware etc
It booted up but was not current!!
0
The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

 
ITguy565Commented:
Before you recover the server from backup media this can usually be caused by a low physical diskspace condition. Please provide more information as to what exactly the event is saying in regards to the failed services.
0
 
Aaron TomoskySD-WAN SimplifiedCommented:
1. Listen to hanccocka, he knows his stuff, be patient

2. Do you have a BDC running? Is dhcp running on something besides the dc vm we are working with?
0
 
j-teksolutionsAuthor Commented:
Ok Hanccocka stress meter is high thanks so much
Getting a screen shot of vm store for you
The server is crawling
0
 
j-teksolutionsAuthor Commented:
146G is original C the OS      the 312G is the original D (data)
0
 
j-teksolutionsAuthor Commented:
Also when i EDIT the vm just to peek - both HD1 and HD2 path to the .00000 files! not the original vdmks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, your DC is running on a snapshot, probably caused by a backup application e.g. Veeam etc

Just to summarise, looks like your DC (VM) has two disks, both which have a snapshot attached.

Can you check Snapshot Manager on the VM, any snapshots listed?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
also is this VM currently OFF?

what is the freespace on the current datastore?
0
 
ITguy565Commented:
I will withdraw from this question you are in good hands..
0
 
j-teksolutionsAuthor Commented:
I have 2 esxi servers - prepared esxi2 as I planned to move copies of my vms to it for hardware and esxi redundancy

The screenshot you see above is from  esxi1  - I had never had the chance to "copy" it yet to esxi2

The screenshot above the vm is stopped

The datastore on that esxi server still has 917G free tons of space

I dont understand why using veeam to copy of the vms would create a snapshot?  I would rather manage the original vdmk files?
0
 
j-teksolutionsAuthor Commented:
FYI - just prior to writing this - as mentioned I copied all the files using veeam to esxi 2 - "except" the massive 300+G data vdmk for time sake - I just wanted to see if the OS would boot up properly.
It did hanccocka , but stale dated.   I did not boot the snapshot - I boot the orginal 146G vdmk which took like 3hrs! to copy=(
So - I thought I would start the vm once more on esxi1 - (nothing was touched here only copying)  and it took some time but its trying to start and stuck on stage 3 of 3 updates for over half hr=(   havent touched - its sitting on that screen at this point
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, next question, if you select the VM which is off (good), Right Click the VM, Select Snapshot, Snapshot Manager

anything listed in here?
0
 
j-teksolutionsAuthor Commented:
Yes!  Temporary snapshot for veeam quick migration=(
I always stopped the vms though never hot!?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
well Veeam Quick MIgrations is rather "luke warm"

okay, any snapshots listed in the Snapshot Manager?
0
 
j-teksolutionsAuthor Commented:
1hr stuck here
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
this is a different VM from the one we are working on?

(same VM, but on a different server?)

Windows 2008 R2 Updates are always very CPU and Disk intensive, and if you are also running on a snapshot, will be much slower.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, any snapshots listed in the Snapshot Manager? and then we can continue....with the following procedure.....
0
 
j-teksolutionsAuthor Commented:
sorry this is not R2 this 2008 Server
as mentioned above I had started it before even writing this entire POST
its stuck at that screen still  - Do i power down cold?
sending screenshot of snap man
0
 
j-teksolutionsAuthor Commented:
snap man
0
 
j-teksolutionsAuthor Commented:
UPDATE!
finally got to the logon over 1.4hrs
Server is now logged on standing by
I turned off nics in case
0
 
j-teksolutionsAuthor Commented:
UPDATE:   very very sluggish which we expected Hanccocka
Do we merge a snapshot ?  I wont even ask:) I know you have a plan.
It appears the OS is slowly trying to start services etc just crawling
0
 
j-teksolutionsAuthor Commented:
OK just read your instruction.... its in between my 2 posts as you can see.  So proceed to soft shutdown this vm and then proceed with your last instruction?
0
 
j-teksolutionsAuthor Commented:
Can I cold shutdown the OS for this snapshot deletion procedure?  I can barely get control over it - resources somewhere are swamped
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, you can shutdown the VM, but once the task is started it cannot be stopped or cancelled, and you will not be able to power on until task ends
0
 
j-teksolutionsAuthor Commented:
in progress -!
0
 
j-teksolutionsAuthor Commented:
UPDATE:   Deletion of snapshots has been on 99% for a bit
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
yes, it could sit there for hours......be patient
0
 
j-teksolutionsAuthor Commented:
thank you absolutely - just read similar cases in the meantime - i am going to check in a couple of hrs .
Can I ask something?
I assume snapshots are not meant to boot from?   Why would my vm boot from a snapshot on its own I did not specify that target vdmk?
Lastly - should I make it habit to shutdown the vms from time to time and delete all snapshots?
just learning from this
thanks
0
 
j-teksolutionsAuthor Commented:
oh one more thing.   when copying vms to my esxi2 box.  Is it ok to use "quick migration" with the delete source vm box unchecked or use copy feature for basic flat copy of entire vm folder from esxi1 datastore to esxi2 datastore?

thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
you should check regularly, every day as part of your VMware Admin daily routines for snapshots.

Backup utilities are often responsible for leaving VMs on snapshots, in error. Its not normal, they should be create and then deleted.

its okay to leave the source checkbox unchecked gives you a backup.
0
 
Aaron TomoskySD-WAN SimplifiedCommented:
Snapshots are nice for before applying an update or something else with the potential to break stuff. Backup software uses them. But they are not backups and should be removed as soon as possible.
0
 
RickEpnetCommented:
Note: You do not need to shut down a VM to delete snapshots it can delete them hot.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
@RickEpnet but it can take longer, use more storage space, and cause the VM to unaccessible! So if the VM is going to unaccessible, due to a large snapshot, you may as well shut it down!

and on the subject of updates, never revert a DC that has snapshots, so Updating and using Snapshots to reverse the issue - do not do it!
0
 
j-teksolutionsAuthor Commented:
Hanccocoka you are a life saver!   My DC is running perfectly now I really appreciate all of your help!
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Thanks for your kind comments.

Remember to add Snapshot Checks to your Daily Admin Tasks!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.