Solved

VMware Emergency - Domain Controller issue

Posted on 2013-01-19
37
451 Views
Last Modified: 2013-01-20
OUR DC has been running perfectly for months and months with no issues.
Today we found the server to be very sluggish and attempted to reboot it via RDP

After about 45mins I managed to get to restart - many services are failing and still sluggish.
I fear I may be recovering from backup
I dont know if the issue is OS related or VM or both.

I see that there are snapshots now in the datastore?   I dont fully understand how the vm manages its disks etc,  I just know that more files are in the store now with .000001 etc for both our C and D disks.
What is happening here?
I copied the original vdmk to another esxi server we have to rule out hardware etc
It booted up but was not current!!
Please help!
0
Comment
Question by:j-teksolutions
  • 19
  • 13
  • 2
  • +2
37 Comments
 

Author Comment

by:j-teksolutions
Comment Utility
Worst case scenario:  

1. I had Backup Exec r3 running in the vm and its current.
2.  Current backup of AD,  SQL,  Exchange
3.  I have a clean full backup of the vms from OCT 2012

Never done an AD restore etc with a vm hoping not to.
0
 
LVL 117

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
Comment Utility
It's possible you have a snapshot on your VM, because when VMs run slow, due to high CPU it's because they are running on a snapshot disk.

Check my EE Article and confirm you are running on a snapshot:-

HOW TO: VMware Snapshots :- Be Patient

PLease upload a screenshot of your datastore, and I'll work with you to get out of this snapshot mess.

Please be patient, and calm, do not fiddle, and follow my instructions.


doing the following may have caused MORE ISSUES!
I copied the original vdmk to another esxi server we have to rule out hardware etc
It booted up but was not current!!
0
 
LVL 11

Expert Comment

by:itguy565
Comment Utility
Before you recover the server from backup media this can usually be caused by a low physical diskspace condition. Please provide more information as to what exactly the event is saying in regards to the failed services.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
1. Listen to hanccocka, he knows his stuff, be patient

2. Do you have a BDC running? Is dhcp running on something besides the dc vm we are working with?
0
 

Author Comment

by:j-teksolutions
Comment Utility
Ok Hanccocka stress meter is high thanks so much
Getting a screen shot of vm store for you
The server is crawling
0
 

Author Comment

by:j-teksolutions
Comment Utility
146G is original C the OS      the 312G is the original D (data)
0
 

Author Comment

by:j-teksolutions
Comment Utility
Also when i EDIT the vm just to peek - both HD1 and HD2 path to the .00000 files! not the original vdmks
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Yes, your DC is running on a snapshot, probably caused by a backup application e.g. Veeam etc

Just to summarise, looks like your DC (VM) has two disks, both which have a snapshot attached.

Can you check Snapshot Manager on the VM, any snapshots listed?
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
also is this VM currently OFF?

what is the freespace on the current datastore?
0
 
LVL 11

Expert Comment

by:itguy565
Comment Utility
I will withdraw from this question you are in good hands..
0
 

Author Comment

by:j-teksolutions
Comment Utility
I have 2 esxi servers - prepared esxi2 as I planned to move copies of my vms to it for hardware and esxi redundancy

The screenshot you see above is from  esxi1  - I had never had the chance to "copy" it yet to esxi2

The screenshot above the vm is stopped

The datastore on that esxi server still has 917G free tons of space

I dont understand why using veeam to copy of the vms would create a snapshot?  I would rather manage the original vdmk files?
0
 

Author Comment

by:j-teksolutions
Comment Utility
FYI - just prior to writing this - as mentioned I copied all the files using veeam to esxi 2 - "except" the massive 300+G data vdmk for time sake - I just wanted to see if the OS would boot up properly.
It did hanccocka , but stale dated.   I did not boot the snapshot - I boot the orginal 146G vdmk which took like 3hrs! to copy=(
So - I thought I would start the vm once more on esxi1 - (nothing was touched here only copying)  and it took some time but its trying to start and stuck on stage 3 of 3 updates for over half hr=(   havent touched - its sitting on that screen at this point
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
okay, next question, if you select the VM which is off (good), Right Click the VM, Select Snapshot, Snapshot Manager

anything listed in here?
0
 

Author Comment

by:j-teksolutions
Comment Utility
Yes!  Temporary snapshot for veeam quick migration=(
I always stopped the vms though never hot!?
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
well Veeam Quick MIgrations is rather "luke warm"

okay, any snapshots listed in the Snapshot Manager?
0
 

Author Comment

by:j-teksolutions
Comment Utility
1hr stuck here
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
this is a different VM from the one we are working on?

(same VM, but on a different server?)

Windows 2008 R2 Updates are always very CPU and Disk intensive, and if you are also running on a snapshot, will be much slower.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
okay, any snapshots listed in the Snapshot Manager? and then we can continue....with the following procedure.....
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 

Author Comment

by:j-teksolutions
Comment Utility
sorry this is not R2 this 2008 Server
as mentioned above I had started it before even writing this entire POST
its stuck at that screen still  - Do i power down cold?
sending screenshot of snap man
0
 

Author Comment

by:j-teksolutions
Comment Utility
snap man
0
 

Author Comment

by:j-teksolutions
Comment Utility
UPDATE!
finally got to the logon over 1.4hrs
Server is now logged on standing by
I turned off nics in case
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
Comment Utility
Okay, proceed as follows:-

Hit the DELETE ALL Button, and WAIT and Be Patient, whilst the Snapshots are deleted.

this could take days, hours, minutes or seconds, depending upon the speed of your datastore.

Do not be tempted to do anything to the VM, shutdown, restart, play, fiddle with it of anyrthing, just wait......
0
 

Author Comment

by:j-teksolutions
Comment Utility
UPDATE:   very very sluggish which we expected Hanccocka
Do we merge a snapshot ?  I wont even ask:) I know you have a plan.
It appears the OS is slowly trying to start services etc just crawling
0
 

Author Comment

by:j-teksolutions
Comment Utility
OK just read your instruction.... its in between my 2 posts as you can see.  So proceed to soft shutdown this vm and then proceed with your last instruction?
0
 

Author Comment

by:j-teksolutions
Comment Utility
Can I cold shutdown the OS for this snapshot deletion procedure?  I can barely get control over it - resources somewhere are swamped
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Yes, you can shutdown the VM, but once the task is started it cannot be stopped or cancelled, and you will not be able to power on until task ends
0
 

Author Comment

by:j-teksolutions
Comment Utility
in progress -!
0
 

Author Comment

by:j-teksolutions
Comment Utility
UPDATE:   Deletion of snapshots has been on 99% for a bit
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
yes, it could sit there for hours......be patient
0
 

Author Comment

by:j-teksolutions
Comment Utility
thank you absolutely - just read similar cases in the meantime - i am going to check in a couple of hrs .
Can I ask something?
I assume snapshots are not meant to boot from?   Why would my vm boot from a snapshot on its own I did not specify that target vdmk?
Lastly - should I make it habit to shutdown the vms from time to time and delete all snapshots?
just learning from this
thanks
0
 

Author Comment

by:j-teksolutions
Comment Utility
oh one more thing.   when copying vms to my esxi2 box.  Is it ok to use "quick migration" with the delete source vm box unchecked or use copy feature for basic flat copy of entire vm folder from esxi1 datastore to esxi2 datastore?

thanks
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
you should check regularly, every day as part of your VMware Admin daily routines for snapshots.

Backup utilities are often responsible for leaving VMs on snapshots, in error. Its not normal, they should be create and then deleted.

its okay to leave the source checkbox unchecked gives you a backup.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
Snapshots are nice for before applying an update or something else with the potential to break stuff. Backup software uses them. But they are not backups and should be removed as soon as possible.
0
 
LVL 14

Expert Comment

by:RickEpnet
Comment Utility
Note: You do not need to shut down a VM to delete snapshots it can delete them hot.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
@RickEpnet but it can take longer, use more storage space, and cause the VM to unaccessible! So if the VM is going to unaccessible, due to a large snapshot, you may as well shut it down!

and on the subject of updates, never revert a DC that has snapshots, so Updating and using Snapshots to reverse the issue - do not do it!
0
 

Author Closing Comment

by:j-teksolutions
Comment Utility
Hanccocoka you are a life saver!   My DC is running perfectly now I really appreciate all of your help!
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Thanks for your kind comments.

Remember to add Snapshot Checks to your Daily Admin Tasks!
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
Teach the user how to install vSphere Update Manager  Console to Windows system:  Install vSphere Update Manager: Configure vSphere Update Manager plug-in in vSphere Client: Verify vSphere Update Manager settings in vSphere Client:
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now