asked on
Defrag or not to Defrag...that is the question
Hello,
I have a bit of a problem. I've got an Exchange 2010 sp2 rollup 8 server that everybody seemed to have forgotten about for a couple years and the performance is awful. The worst part is that it really only hosts one mailbox (aside from admin and discovery/search) that people use as their CRM and share the calendar with a bunch of complicated tasks and appointments. The existing mailbox is now 18GB in size and the database has mutated into this 109GB disaster that has about 80GB of whitespace in it. The server itself is Server 2008R2 on an ESXi 5.0 host. The host is using NFS shares coming off a NetApp. The configuration is as follows:
Host: Esxi5
Datastores: Storage1(Local)--84GB free/272GB
VMWare1(NFS)--148GB free/970GB
VMWare2(NFS)--121GB free/690GB
VM: Ex01
HDD1(C:): 55GB vmdk in VMWare2. 22GB free
HDD2(D:): 350GB vmdk in VMWare1. 241GBfree and houses the exchange DB.
HDD3(E:): 140GB vmdk in VMWare1. 138GBfree and houses the exchange logs
HDD4: 32GB vmdk in VMWare1. Page file
HDD5: 200GB vmdk in VMware1. 50GB free and has some old databases in there that are not mounted.
All of the vmdk files are in an EX01.domain.com folder in their respective datastore. The vm's actual vmx file and working location is in a different EX01_1 folder within Vmware1. VMware1 not only has VMDK1,2,3, and 4 in it but it also has a -000003.vmdk for each drive that is a snapshot that hasn't been modified in over a week. Problem 1. The 0000001 and 0000002 snapshots are gone now and vsphere says they need consolidation but "Consolidate" is grayed out and not an option. Somebody said, at one time, they tried to consolidate the disks and the vm locked up and said the datastore was full and nothing would boot up anymore so they deleted them and only kept the 000003 vmdks because they had the most recent modified date.
Problem 2: I try moving the 18GB mailbox to a new database on drive D: so I can remove the fragmented DB and I get:
Error: MapiExceptionNotFound: Unable to synchronize manifest. (hr=0x8004010f, ec=-2147221233)
Diagnostic context:
Lid: 55847 EMSMDBPOOL.EcPoolSessionDo
Lid: 43559 EMSMDBPOOL.EcPoolSessionDo
Lid: 23226 --- ROP Parse Start ---
Lid: 27962 ROP: ropIncrCfg [112]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropFXSrcGetBufferEx [156]
Lid: 17082 ROP Error: 0x8004010F
Lid: 23137
Lid: 21921 StoreEc: 0x8004010F
Lid: 27962 ROP: ropExtendedError [250]
Lid: 1494 ---- Remote Context Beg ----
Lid: 1238 Remote Context Overflow
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 46077 dwParam: 0xA52224
Lid: 46121 StoreEc: 0x8004010F
Lid: 22864
Lid: 3742 StoreEc: 0x8004010F
Lid: 23296
Lid: 2526 StoreEc: 0x8004010F
Lid: 20912
Lid: 24504 StoreEc: 0x8004010F
Lid: 24148 StoreEc: 0x8004010F
Lid: 23796
Lid: 2478 StoreEc: 0x8004010F
Lid: 1750 ---- Remote Context End ----
Lid: 26849
Lid: 21817 ROP Failure: 0x8004010F
Lid: 32758
Lid: 16586 StoreEc: 0x8004010F
Lid: 22518
Lid: 28874 StoreEc: 0x8004010F
Lid: 29516
Lid: 31820 StoreEc: 0x8004010F
Can anybody help out?
Thanks!
I have a bit of a problem. I've got an Exchange 2010 sp2 rollup 8 server that everybody seemed to have forgotten about for a couple years and the performance is awful. The worst part is that it really only hosts one mailbox (aside from admin and discovery/search) that people use as their CRM and share the calendar with a bunch of complicated tasks and appointments. The existing mailbox is now 18GB in size and the database has mutated into this 109GB disaster that has about 80GB of whitespace in it. The server itself is Server 2008R2 on an ESXi 5.0 host. The host is using NFS shares coming off a NetApp. The configuration is as follows:
Host: Esxi5
Datastores: Storage1(Local)--84GB free/272GB
VMWare1(NFS)--148GB free/970GB
VMWare2(NFS)--121GB free/690GB
VM: Ex01
HDD1(C:): 55GB vmdk in VMWare2. 22GB free
HDD2(D:): 350GB vmdk in VMWare1. 241GBfree and houses the exchange DB.
HDD3(E:): 140GB vmdk in VMWare1. 138GBfree and houses the exchange logs
HDD4: 32GB vmdk in VMWare1. Page file
HDD5: 200GB vmdk in VMware1. 50GB free and has some old databases in there that are not mounted.
All of the vmdk files are in an EX01.domain.com folder in their respective datastore. The vm's actual vmx file and working location is in a different EX01_1 folder within Vmware1. VMware1 not only has VMDK1,2,3, and 4 in it but it also has a -000003.vmdk for each drive that is a snapshot that hasn't been modified in over a week. Problem 1. The 0000001 and 0000002 snapshots are gone now and vsphere says they need consolidation but "Consolidate" is grayed out and not an option. Somebody said, at one time, they tried to consolidate the disks and the vm locked up and said the datastore was full and nothing would boot up anymore so they deleted them and only kept the 000003 vmdks because they had the most recent modified date.
Problem 2: I try moving the 18GB mailbox to a new database on drive D: so I can remove the fragmented DB and I get:
Error: MapiExceptionNotFound: Unable to synchronize manifest. (hr=0x8004010f, ec=-2147221233)
Diagnostic context:
Lid: 55847 EMSMDBPOOL.EcPoolSessionDo
Lid: 43559 EMSMDBPOOL.EcPoolSessionDo
Lid: 23226 --- ROP Parse Start ---
Lid: 27962 ROP: ropIncrCfg [112]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropUpldStStrmBegin [117]
Lid: 27962 ROP: ropUpldStStrmEnd [119]
Lid: 27962 ROP: ropFXSrcGetBufferEx [156]
Lid: 17082 ROP Error: 0x8004010F
Lid: 23137
Lid: 21921 StoreEc: 0x8004010F
Lid: 27962 ROP: ropExtendedError [250]
Lid: 1494 ---- Remote Context Beg ----
Lid: 1238 Remote Context Overflow
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 46077 dwParam: 0xA52224
Lid: 46121 StoreEc: 0x8004010F
Lid: 22864
Lid: 3742 StoreEc: 0x8004010F
Lid: 23296
Lid: 2526 StoreEc: 0x8004010F
Lid: 20912
Lid: 24504 StoreEc: 0x8004010F
Lid: 24148 StoreEc: 0x8004010F
Lid: 23796
Lid: 2478 StoreEc: 0x8004010F
Lid: 1750 ---- Remote Context End ----
Lid: 26849
Lid: 21817 ROP Failure: 0x8004010F
Lid: 32758
Lid: 16586 StoreEc: 0x8004010F
Lid: 22518
Lid: 28874 StoreEc: 0x8004010F
Lid: 29516
Lid: 31820 StoreEc: 0x8004010F
Can anybody help out?
Thanks!
VMwareExchangeWindows Server 2008
Last Comment
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
hopefully this wrong, but looks like this discovery mailbox is corrupted. Can you recover from backup?
in addition to your snapshot issue, seems in the process of trying to fix things there is data corruption in your database.
suggest using isinteg on it but one thing at a time; try to get your vmware issue straightened out first
hope you have a good backup...
suggest using isinteg on it but one thing at a time; try to get your vmware issue straightened out first
hope you have a good backup...
ASKER
Thanks for all the replies!
So I can just manually delete the vmdk files that end in 000003.vmdk, right? ex. priv_1-000003.vmdk. The settings of the vm specify that the hard drives are priv_1.vmdk, etc.
Just want to make sure.
So I can just manually delete the vmdk files that end in 000003.vmdk, right? ex. priv_1-000003.vmdk. The settings of the vm specify that the hard drives are priv_1.vmdk, etc.
Just want to make sure.
SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
The file on the very bottom is vmxrebuild.sh.
As I stated earlier, this is the servername.ad.domain.priv folder within the datastore that has the actual vmdk files.
Capture.PNG
As I stated earlier, this is the servername.ad.domain.priv folder within the datastore that has the actual vmdk files.
Capture.PNG
ASKER
This is the EX1_1 folder at the same level as the previous one that is the working location of the vm. I chopped it in half with snipping tool to remove the full server name.
Capture1.PNG
Capture3.PNG
Capture1.PNG
Capture3.PNG
can you check your disk properties in the VM settings, is the disk on a snapshot, e.g. the disk name is -00003.vmdk ?
ASKER
No, the hard disks 1-4 are using Priv_#.vmdks.
I think somebody deleted all the -000001, -000002 delta files and just "rebuilt" the vm to use the priv_1 vmdks and restored the exchange database from a backup and left the -0000003 delta files hanging around because, at the time, the vm was running off of that.
I think somebody deleted all the -000001, -000002 delta files and just "rebuilt" the vm to use the priv_1 vmdks and restored the exchange database from a backup and left the -0000003 delta files hanging around because, at the time, the vm was running off of that.
Well, I'm sorry to tell you the damage is already done, you are not running on a snapshot disk, they could be disposed of.
but information is probably lost of corrupted.
the person which just deleted the files, is an idiot, and needs shooting, you could have recovered the VM correctly.
The -0000003 delta files are useless.
You need to have the entire chain, of snapshot files, because they are linked.
e.g. 3 linked to 2, linked to 1 linked to parent disk.
It states consolidate because vCenter is not that clever, and just detects a file called -00000x.vmdk, and therefore displays the message Consolidate.
Move the files to archive, if required move them back, but delete in a few weeks.
You will now have to continue with an Exchange DB restore, or Exchange DB repair.
but information is probably lost of corrupted.
the person which just deleted the files, is an idiot, and needs shooting, you could have recovered the VM correctly.
The -0000003 delta files are useless.
You need to have the entire chain, of snapshot files, because they are linked.
e.g. 3 linked to 2, linked to 1 linked to parent disk.
It states consolidate because vCenter is not that clever, and just detects a file called -00000x.vmdk, and therefore displays the message Consolidate.
Move the files to archive, if required move them back, but delete in a few weeks.
You will now have to continue with an Exchange DB restore, or Exchange DB repair.
ASKER
Yea, they're not here anymore but I think the datastore got full and nothing would work so they saw the old delta files "not being written to" and freed up space that way.
Sooo....now do you think the database is jacked because some of it's data was spread across the actual vmdks and the 3 delta disks? Will a DB repair do anything?
Sooo....now do you think the database is jacked because some of it's data was spread across the actual vmdks and the 3 delta disks? Will a DB repair do anything?
SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Yea, tell me about it. So is there a way (and I'm reaching here) that I can restore the database back from say August 27th or so and merge the data together that they've entered since then?
Can I restore the database from the 27th and then just restore the mailbox from the backup tonight?
Can I restore the database from the 27th and then just restore the mailbox from the backup tonight?
ASKER
Hey it says that I can't move the vmdk delta disks because there is a lock on it. Do you know how to see what has the lock on it when it's an NFS share? We're using Commvault for a backup solution and I've already tried powering down the backup server to see if somehow that had a lock on it.
checking for locks, can be chasing your tail...
see here
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051
and sometimes, it's easier to restart the host.
Unless the VM is actually writing to the snapshot disk!
You can check this, easily by looking at the virtual machine disks, in the disk settings of the VM.
see here
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051
and sometimes, it's easier to restart the host.
Unless the VM is actually writing to the snapshot disk!
You can check this, easily by looking at the virtual machine disks, in the disk settings of the VM.
ASKER
I restarted the host this morning as well and verified that it is not using the delta disks in the settings of the vm.
ASKER
I read that article already but got hung up at the NFS volume part.
Start by identifying the server whose VMkernel may be locking the file. To identify the server:
Report the MAC address of the lock holder by running this command (except on an NFS volume):
Start by identifying the server whose VMkernel may be locking the file. To identify the server:
Report the MAC address of the lock holder by running this command (except on an NFS volume):
if no VMs are using the snapshot, what do you want to do with the snapshot ?
how many hosts do you have ?
if all the hosts have been powered off, the lock should be gone, on the snapshot.
how many hosts do you have ?
if all the hosts have been powered off, the lock should be gone, on the snapshot.
ASKER
I just wanted to move those deltas like you said. There are two hosts. Why would the second host be locking any of those files?
Yes, it could be locking the files, if you have vMotion or DRS.
ASKER
It magically unlocked when I tried it again.
They don't have vMotion.
Anyway, we're using the standalone converter to move a copy to another host so we can bring it back into our test environment. I'm only getting 1.5MBps on the transfer and it will say like 4 days estimate. I set the SSL to false and that brought it to 3.5 days. You know of any other way to speed it up?
They don't have vMotion.
Anyway, we're using the standalone converter to move a copy to another host so we can bring it back into our test environment. I'm only getting 1.5MBps on the transfer and it will say like 4 days estimate. I set the SSL to false and that brought it to 3.5 days. You know of any other way to speed it up?
Be patient! It's a data copy operation, these can be slow, it depends on source and destination datastore speeds and network.
I'm just reminding you politely, that the last number of posts, have been really off topic, from the original question posted, and should have been split into separate questions, this is for the benefit of EE members/readers and Experts.
I'm just reminding you politely, that the last number of posts, have been really off topic, from the original question posted, and should have been split into separate questions, this is for the benefit of EE members/readers and Experts.