Virtual machine file server SLLLLOW after trying to back it up with Symantec Backup Exec

Posted on 2011-10-21
Last Modified: 2012-05-12
I'm not sure where to begin with this problem but I'm losing my mind!!!!

So we have vSphere 4.  I have a Windows 2003 Enterprise VM that serves as a file server for My Documents / Application Data redirection through active directory.  That partition is around 1TB and has nearly 1 million files in it.  Up until the other day it worked great...

All started like this...   I installed the Symantec Backup Exec Remote Windows Agent on this machine.  We have Symantec Vmware Agent on our Backup Exec Server.  For GRT to work, the agent must be installed on the client.  This is all Symantec Backup Exec 2010 R3 stuff.

So... I set up a job in backup exec to backup this server.  It ran for 48 hours before failing.  I noticed in vcenter that it had created a snapshot / removed a snapshot.. etc. etc.  The error code in BUE was "Unable to copy the virtual machine disk using the VMware VixDiskLib".  

So now... this server is HORRIBLY SLOW.  The CPU stays pegged out all the time inside vcenter and inside the VM.  Rebooting does nothing to help.  I've uninstalled the remote backup agent.  I've uninstalled endpoint protection.  Inside vcenter if I right click the VM, migrate is grayed out.  I can't drag and drop to another server.  All other servers are running fine alongside this one.

After hours when this server is idle.. CPU is still high.  Ping responses are slow.

I'm convinced Symantec Backup Exec has done somethign to the VM inside vcenter.   But I just don't know where to look.  

Does anyone have ANY ideas?

Question by:ZenQuest
    LVL 37

    Expert Comment

    Are all snapshots now commited?
    It sounds like you have created a snapshot, left it there for a long period of time and now your VM is deleting that snapshot still. Just a possible thought??
    LVL 116

    Expert Comment

    by:Andrew Hancock (VMware vExpert / EE MVE)
    Check VM Properties Right Click the VM, Select Snapshot, Snapshot Manager, and check if there is a Snapshot Present?

    Author Comment

    That's exactly what I thought.. in fact, this happened before and deleting the snapshot fixed the issue.  Snapshot manager shows no snapshots though.

    Is it possible vcenter isn't aware that its in a snapshot state?

    LVL 28

    Expert Comment

    It is possible. You can use vCenter to browse the data stores. Select Datastores from the View _> Inventory menu and then locate the data tore that the server is using and browse it and see if there are snapshot files in the folder holding the VMDK and other files associated with that VM.  If there are, sometimes you can create another snapshot in the snapshot manager and then choose to delete all snaps and it will roll back through them all. You will have to have enough disk space on that LUN to allow the snapshot process to roll back and if there are a lot of them or a really large one, it could take quite a while.If you find this is the case and have any support from vmWare or a third party you may also want to engage them.

    Author Comment

    I've embedded a screenshot of the VM's datastore.  It has a 10gig and 600gig partition...  I was a little high when I said close to a TB.  I've grown that 600gig partition from 250 to 350 to 600 over the last two years so Windows show alls the different partitions making up the volume.  I did notice that on ANOTHER datastore I have a UserData folder with a 100Gig VMDK file in it (only file in directory).  I'm thinking I created that drive a long time ago and deleted it but it still remains on the datastore.  The date on the file is May, 2010.    For some reason, today, the migrate option is accessible but I haven't tried doing anything as this is a production server.  I'm tempted to  'clone' this server and see if the clone runs better...  or maybe provision a new server and copy all the data from the old to the new... I'm running out of thigns to check.
     Screenshot of Datastore
    LVL 116

    Expert Comment

    by:Andrew Hancock (VMware vExpert / EE MVE)
    No snapshot present in that folder. But it is possible snapshots can exist but not listed, but not this time.

    What services are running in VM, and what does task manager show?
    LVL 12

    Accepted Solution

    how about registering the vm again in vcenter or s-vmotion to a different datastore.

    Featured Post

    IT, Stop Being Called Into Every Meeting

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Join & Write a Comment

    This Tutorial covers a very basic and common question asked on Experts Exchange, "How Do I Clone or Copy a virtual machine in VMware vSphere Hypervisor ESX/ESXi 4.x, ESXi 5.0?" Using the following method, no third party tools are required or need…
    A quick step-by-step overview of installing and configuring Carbonite Server Backup.
    In this Micro Tutorial viewers will learn how to use Boot Corrector from Paragon Rescue Kit Free to identify and fix the boot problems of Windows 7/8/2012R2 etc. As an example is used Windows 2012R2 which lost its active partition flag (often happen…
    To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    15 Experts available now in Live!

    Get 1:1 Help Now