Avatar of Jason
Jason
 asked on

Defrag or not to Defrag...that is the question

Hello,
I have a bit of a problem.  I've got an Exchange 2010 sp2 rollup 8 server that everybody seemed to have forgotten about for a couple years and the performance is awful.  The worst part is that it really only hosts one mailbox (aside from admin and discovery/search) that people use as their CRM and share the calendar with a bunch of complicated tasks and appointments.  The existing mailbox is now 18GB in size and the database has mutated into this 109GB disaster that has about 80GB of whitespace in it.  The server itself is Server 2008R2 on an ESXi 5.0 host.  The host is using NFS shares coming off a NetApp.  The configuration is as follows:
Host:  Esxi5
Datastores:  Storage1(Local)--84GB free/272GB
VMWare1(NFS)--148GB free/970GB
VMWare2(NFS)--121GB free/690GB
VM:  Ex01
HDD1(C:): 55GB vmdk in VMWare2.  22GB free
HDD2(D:):  350GB vmdk in VMWare1.  241GBfree  and houses the exchange DB.
HDD3(E:):   140GB vmdk in VMWare1.  138GBfree and houses the exchange logs
HDD4:  32GB vmdk in VMWare1. Page file
HDD5:  200GB vmdk in VMware1.  50GB free and has some old databases in there that are not mounted.
All of the vmdk files are in an EX01.domain.com folder in their respective datastore.  The vm's actual vmx file and working location is in a different EX01_1 folder within Vmware1.  VMware1 not only has VMDK1,2,3, and 4 in it but it also has a -000003.vmdk for each drive that is a snapshot that hasn't been modified in over a week.  Problem 1.  The 0000001 and 0000002 snapshots are gone now and vsphere says they need consolidation but "Consolidate" is grayed out and not an option.  Somebody said, at one time, they tried to consolidate the disks and the vm locked up and said the datastore was full and nothing would boot up anymore so they deleted them and only kept the 000003 vmdks because they had the most recent modified date.  
Problem 2:  I try moving the 18GB mailbox to a new database on drive D: so I can remove the fragmented DB and I get:
Error: MapiExceptionNotFound: Unable to synchronize manifest. (hr=0x8004010f, ec=-2147221233)
Diagnostic context:
    Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=239]
    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=719][latency=546]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropIncrCfg [112]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropUpldStStrmBegin [117]
    Lid: 27962   ROP: ropUpldStStrmEnd [119]
    Lid: 27962   ROP: ropFXSrcGetBufferEx [156]
    Lid: 17082   ROP Error: 0x8004010F
    Lid: 23137  
    Lid: 21921   StoreEc: 0x8004010F
    Lid: 27962   ROP: ropExtendedError [250]
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 1238    Remote Context Overflow
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 21970   StoreEc: 0x8004010F PropTag: 0x668F0040
    Lid: 46077   dwParam: 0xA52224
    Lid: 46121   StoreEc: 0x8004010F
    Lid: 22864  
    Lid: 3742    StoreEc: 0x8004010F
    Lid: 23296  
    Lid: 2526    StoreEc: 0x8004010F
    Lid: 20912  
    Lid: 24504   StoreEc: 0x8004010F
    Lid: 24148   StoreEc: 0x8004010F
    Lid: 23796  
    Lid: 2478    StoreEc: 0x8004010F
    Lid: 1750    ---- Remote Context End ----
    Lid: 26849  
    Lid: 21817   ROP Failure: 0x8004010F
    Lid: 32758  
    Lid: 16586   StoreEc: 0x8004010F
    Lid: 22518  
    Lid: 28874   StoreEc: 0x8004010F
    Lid: 29516  
    Lid: 31820   StoreEc: 0x8004010F

Can anybody help out?

Thanks!
VMwareExchangeWindows Server 2008

Avatar of undefined
Last Comment
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Miguel Angel Perez Muñoz

hopefully this wrong, but looks like this discovery mailbox is corrupted. Can you recover from backup?
Seth Simmons

in addition to your snapshot issue, seems in the process of trying to fix things there is data corruption in your database.

suggest using isinteg on it but one thing at a time; try to get your vmware issue straightened out first

hope you have a good backup...
Jason

ASKER
Thanks for all the replies!

So I can just manually delete the vmdk files that end in 000003.vmdk, right? ex.  priv_1-000003.vmdk. The settings of the vm specify that the hard drives are priv_1.vmdk, etc.
Just want to make sure.
Your help has saved me hundreds of hours of internet surfing.
fblack61
SOLUTION
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Jason

ASKER
The file on the very bottom is vmxrebuild.sh.
As I stated earlier, this is the servername.ad.domain.priv folder within the datastore that has the actual vmdk files.
Capture.PNG
Jason

ASKER
This is the EX1_1 folder at the same level as the previous one that is the working location of the vm.  I chopped it in half with snipping tool to remove the full server name.
Capture1.PNG
Capture3.PNG
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

can you check your disk properties in the VM settings, is the disk on a snapshot, e.g. the disk name is -00003.vmdk ?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Jason

ASKER
No, the hard disks 1-4 are using Priv_#.vmdks.
I think somebody deleted all the -000001, -000002 delta files and just "rebuilt" the vm to use the priv_1 vmdks and restored the exchange database from a backup and left the -0000003 delta files hanging around because, at the time, the vm was running off of that.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Well, I'm sorry to tell you the damage is already done, you are not  running on a snapshot disk, they could be disposed of.

but information is probably lost of corrupted.

the person which just deleted the files, is an idiot, and needs shooting, you could have recovered the VM correctly.

The -0000003 delta files are useless.

You need to have the entire chain, of snapshot files, because they are linked.

e.g. 3 linked to 2, linked to 1 linked to parent disk.

It states consolidate because vCenter is not that clever, and just detects a file called -00000x.vmdk, and therefore displays the message Consolidate.

Move the files to archive, if required move them back, but delete in a few weeks.

You will now have to continue with an Exchange DB restore, or Exchange DB repair.
Jason

ASKER
Yea, they're not here anymore but I think the datastore got full and nothing would work so they saw the old delta files "not being written to" and freed up space that way.  
Sooo....now do you think the database is jacked because some of it's data was spread across the actual vmdks and the 3 delta disks?  Will a DB repair do anything?
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
SOLUTION
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Jason

ASKER
Yea, tell me about it.  So is there a way (and I'm reaching here) that I can restore the database back from say August 27th or so and merge the data together that they've entered since then?
Can I restore the database from the 27th and then just restore the mailbox from the backup tonight?
Jason

ASKER
Hey it says that I can't move the vmdk delta disks because there is a lock on it.  Do you know how to see what has the lock on it when it's an NFS share?  We're using Commvault for a backup solution and I've already tried powering down the backup server to see if somehow that had a lock on it.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

checking for locks, can be chasing your tail...

see here

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051

and sometimes, it's easier to restart the host.

Unless the VM is actually writing to the snapshot disk!

You can check this, easily by looking at the virtual machine disks, in the disk settings of the VM.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Jason

ASKER
I restarted the host this morning as well and verified that it is not using the delta disks in the settings of the vm.
Jason

ASKER
I read that article already but got hung up at the NFS volume part.
Start by identifying the server whose VMkernel may be locking the file. To identify the server:
Report the MAC address of the lock holder by running this command (except on an NFS volume):
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

if no VMs are using the snapshot, what do you want to do with the snapshot ?

how many hosts do you have ?

if all the hosts have been powered off, the lock should be gone, on the snapshot.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Jason

ASKER
I just wanted to move those deltas like you said.  There are two hosts.  Why would the second host be locking any of those files?
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Yes, it could be locking the files, if you have vMotion or DRS.
Jason

ASKER
It magically unlocked when I tried it again.  
They don't have vMotion.
Anyway, we're using the standalone converter to move a copy to another host so we can bring it back into our test environment.  I'm only getting 1.5MBps on the transfer and it will say like 4 days estimate.  I set the SSL to false and that brought it to 3.5 days.  You know of any other way to speed it up?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Be patient! It's a data copy operation, these can be slow, it depends on source and destination datastore speeds and network.

I'm just reminding you politely, that the last number of posts, have been really off topic, from the original question posted, and should have been split into separate questions, this is for the benefit of EE members/readers and Experts.