Defrag or not to Defrag...that is the question

Hello,
I have a bit of a problem.  I've got an Exchange 2010 sp2 rollup 8 server that everybody seemed to have forgotten about for a couple years and the performance is awful.  The worst part is that it really only hosts one mailbox (aside from admin and discovery/search) that people use as their CRM and share the calendar with a bunch of complicated tasks and appointments.  The existing mailbox is now 18GB in size and the database has mutated into this 109GB disaster that has about 80GB of whitespace in it.  The server itself is Server 2008R2 on an ESXi 5.0 host.  The host is using NFS shares coming off a NetApp.  The configuration is as follows:
Host:  Esxi5
Datastores:  Storage1(Local)--84GB free/272GB
VMWare1(NFS)--148GB free/970GB
VMWare2(NFS)--121GB free/690GB
VM:  Ex01
HDD1(C:): 55GB vmdk in VMWare2.  22GB free
HDD2(D:):  350GB vmdk in VMWare1.  241GBfree  and houses the exchange DB.
HDD3(E:):   140GB vmdk in VMWare1.  138GBfree and houses the exchange logs
HDD4:  32GB vmdk in VMWare1. Page file
HDD5:  200GB vmdk in VMware1.  50GB free and has some old databases in there that are not mounted.
All of the vmdk files are in an EX01.domain.com folder in their respective datastore.  The vm's actual vmx file and working location is in a different EX01_1 folder within Vmware1.  VMware1 not only has VMDK1,2,3, and 4 in it but it also has a -000003.vmdk for each drive that is a snapshot that hasn't been modified in over a week.  Problem 1.  The 0000001 and 0000002 snapshots are gone now and vsphere says they need consolidation but "Consolidate" is grayed out and not an option.  Somebody said, at one time, they tried to consolidate the disks and the vm locked up and said the datastore was full and nothing would boot up anymore so they deleted them and only kept the 000003 vmdks because they had the most recent modified date.  
Problem 2:  I try moving the 18GB mailbox to a new database on drive D: so I can remove the fragmented DB and I get:
Error: MapiExceptionNotFound: Unable to synchronize manifest. (hr=0x8004010f, ec=-2147221233)
Diagnostic context:
    Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=239]
    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=719][latency=546]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropIncrCfg [112]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropFXSrcGetBufferEx [156]
    Lid: 17082   ROP Error: 0x8004010F
    Lid: 23137  
    Lid: 21921   StoreEc: 0x8004010F
    Lid: 27962   ROP: ropExtendedError [250]
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 1238    Remote Context Overflow
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 46077   dwParam: 0xA52224
    Lid: 46121   StoreEc: 0x8004010F
    Lid: 22864  
    Lid: 3742    StoreEc: 0x8004010F
    Lid: 23296  
    Lid: 2526    StoreEc: 0x8004010F
    Lid: 20912  
    Lid: 24504   StoreEc: 0x8004010F
    Lid: 24148   StoreEc: 0x8004010F
    Lid: 23796  
    Lid: 2478    StoreEc: 0x8004010F
    Lid: 1750    ---- Remote Context End ----
    Lid: 26849  
    Lid: 21817   ROP Failure: 0x8004010F
    Lid: 32758  
    Lid: 16586   StoreEc: 0x8004010F
    Lid: 22518  
    Lid: 28874   StoreEc: 0x8004010F
    Lid: 29516  
    Lid: 31820   StoreEc: 0x8004010F

Can anybody help out?

Thanks!
JasonAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Performance will be poor, if you have VMs which are currently writing to a snapshot disk.

Lets resolve the snapshot issue....

Have a read of my EE Article

HOW TO: VMware Snapshots :- Be Patient

and then let me know, when you want to continue resolving the snapshot issues.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Miguel Angel Perez MuñozCommented:
hopefully this wrong, but looks like this discovery mailbox is corrupted. Can you recover from backup?
0
Seth SimmonsSr. Systems AdministratorCommented:
in addition to your snapshot issue, seems in the process of trying to fix things there is data corruption in your database.

suggest using isinteg on it but one thing at a time; try to get your vmware issue straightened out first

hope you have a good backup...
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Download your Guide
JasonAuthor Commented:
Thanks for all the replies!

So I can just manually delete the vmdk files that end in 000003.vmdk, right? ex.  priv_1-000003.vmdk. The settings of the vm specify that the hard drives are priv_1.vmdk, etc.
Just want to make sure.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
No you cannot just manually delete the snapshots files.

Can you upload a screenshot of the VM folder on the datastore, and I can work with you to tell you how to proceed.

Also can you let me know, how much free space is available on the datastore.
0
JasonAuthor Commented:
The file on the very bottom is vmxrebuild.sh.
As I stated earlier, this is the servername.ad.domain.priv folder within the datastore that has the actual vmdk files.
Capture.PNG
0
JasonAuthor Commented:
This is the EX1_1 folder at the same level as the previous one that is the working location of the vm.  I chopped it in half with snipping tool to remove the full server name.
Capture1.PNG
Capture3.PNG
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
can you check your disk properties in the VM settings, is the disk on a snapshot, e.g. the disk name is -00003.vmdk ?
0
JasonAuthor Commented:
No, the hard disks 1-4 are using Priv_#.vmdks.
I think somebody deleted all the -000001, -000002 delta files and just "rebuilt" the vm to use the priv_1 vmdks and restored the exchange database from a backup and left the -0000003 delta files hanging around because, at the time, the vm was running off of that.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Well, I'm sorry to tell you the damage is already done, you are not  running on a snapshot disk, they could be disposed of.

but information is probably lost of corrupted.

the person which just deleted the files, is an idiot, and needs shooting, you could have recovered the VM correctly.

The -0000003 delta files are useless.

You need to have the entire chain, of snapshot files, because they are linked.

e.g. 3 linked to 2, linked to 1 linked to parent disk.

It states consolidate because vCenter is not that clever, and just detects a file called -00000x.vmdk, and therefore displays the message Consolidate.

Move the files to archive, if required move them back, but delete in a few weeks.

You will now have to continue with an Exchange DB restore, or Exchange DB repair.
0
JasonAuthor Commented:
Yea, they're not here anymore but I think the datastore got full and nothing would work so they saw the old delta files "not being written to" and freed up space that way.  
Sooo....now do you think the database is jacked because some of it's data was spread across the actual vmdks and the 3 delta disks?  Will a DB repair do anything?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You will need to recover or restore the Exchange database, BEFORE the issue with the snapshot disk.

It's a shame, you didn't seek EE help sooner.
0
JasonAuthor Commented:
Yea, tell me about it.  So is there a way (and I'm reaching here) that I can restore the database back from say August 27th or so and merge the data together that they've entered since then?
Can I restore the database from the 27th and then just restore the mailbox from the backup tonight?
0
JasonAuthor Commented:
Hey it says that I can't move the vmdk delta disks because there is a lock on it.  Do you know how to see what has the lock on it when it's an NFS share?  We're using Commvault for a backup solution and I've already tried powering down the backup server to see if somehow that had a lock on it.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
checking for locks, can be chasing your tail...

see here

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051

and sometimes, it's easier to restart the host.

Unless the VM is actually writing to the snapshot disk!

You can check this, easily by looking at the virtual machine disks, in the disk settings of the VM.
0
JasonAuthor Commented:
I restarted the host this morning as well and verified that it is not using the delta disks in the settings of the vm.
0
JasonAuthor Commented:
I read that article already but got hung up at the NFS volume part.
Start by identifying the server whose VMkernel may be locking the file. To identify the server:
Report the MAC address of the lock holder by running this command (except on an NFS volume):
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
if no VMs are using the snapshot, what do you want to do with the snapshot ?

how many hosts do you have ?

if all the hosts have been powered off, the lock should be gone, on the snapshot.
0
JasonAuthor Commented:
I just wanted to move those deltas like you said.  There are two hosts.  Why would the second host be locking any of those files?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, it could be locking the files, if you have vMotion or DRS.
0
JasonAuthor Commented:
It magically unlocked when I tried it again.  
They don't have vMotion.
Anyway, we're using the standalone converter to move a copy to another host so we can bring it back into our test environment.  I'm only getting 1.5MBps on the transfer and it will say like 4 days estimate.  I set the SSL to false and that brought it to 3.5 days.  You know of any other way to speed it up?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Be patient! It's a data copy operation, these can be slow, it depends on source and destination datastore speeds and network.

I'm just reminding you politely, that the last number of posts, have been really off topic, from the original question posted, and should have been split into separate questions, this is for the benefit of EE members/readers and Experts.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.