markfurey
asked on
ESXI 5.1 Thin provisioning
Hello Experts,
I'm hoping you can clear up some issues I'm having with VMware/ESXI. We have an Exchange VM running on an ISCSI SAN using ESXI 5.1 and VMFS 5 using block sizes of 1MB. The Datastore where the volume is located is 4TB in size, the Exchange VMDK file is in thin provisioned mode a provisioned size of 2.02 TB with 216 GB in use and the data store has 3.67TB free.
The issue is occurring when we try to take a snap-shot using either VCentre Server or using the Veem backup tool, we get the following error; "File <unspecified filename> is larger than the maximum size supported by datastore <unspecified datastore>". I have looked at various internet posts and they indicate various reasons why this might occur. And it looks like the issue is either one of two things; because the provisioned size of the VMDK file which is just over 2TB, although 2TB was selected when it was setup, I have seen posts that suggest that the max size of the VMDK file needs to be slightly less than 2TB
The other reason might be because there isn’t enough disk space available to complete the snap-shot despite the datastore indicating that there is 3.67TB available, this might be because this reading is based on disk space used where the snap-shot calculation is based on the provisioned size, not the space physically used?
So in short, is the problem that there isn’t enough disk space because the "provisioned" size is too big for the snap shot to fit in the datastore, or is it because VMware allowed me to create a VMDK file just over the 2TB threshold.
At the moment we're not getting any snap-shots or backups which we need before pushing some Exchange 2013 updates through.
I'm hoping you can clear up some issues I'm having with VMware/ESXI. We have an Exchange VM running on an ISCSI SAN using ESXI 5.1 and VMFS 5 using block sizes of 1MB. The Datastore where the volume is located is 4TB in size, the Exchange VMDK file is in thin provisioned mode a provisioned size of 2.02 TB with 216 GB in use and the data store has 3.67TB free.
The issue is occurring when we try to take a snap-shot using either VCentre Server or using the Veem backup tool, we get the following error; "File <unspecified filename> is larger than the maximum size supported by datastore <unspecified datastore>". I have looked at various internet posts and they indicate various reasons why this might occur. And it looks like the issue is either one of two things; because the provisioned size of the VMDK file which is just over 2TB, although 2TB was selected when it was setup, I have seen posts that suggest that the max size of the VMDK file needs to be slightly less than 2TB
The other reason might be because there isn’t enough disk space available to complete the snap-shot despite the datastore indicating that there is 3.67TB available, this might be because this reading is based on disk space used where the snap-shot calculation is based on the provisioned size, not the space physically used?
So in short, is the problem that there isn’t enough disk space because the "provisioned" size is too big for the snap shot to fit in the datastore, or is it because VMware allowed me to create a VMDK file just over the 2TB threshold.
At the moment we're not getting any snap-shots or backups which we need before pushing some Exchange 2013 updates through.
ASKER
Hello Hanccocka,
Than you for your input but I have a couple of questions regarding your response. Regarding Exchange not supporting thin provisioned disks, I haven’t seen this anywhere and I'd assumed that the VMDK type would be transparent to the VM inside anyway?
Also, you say that the snapshot and the VM exceeds max size of the VMDK file, excuse my ignorance but I didn’t think a snapshot increased the size of the VMDK file, I'd assumed that the snapshot would generate a new file not increase the existing one? Does this mean that when creating a VM you must create a disk/VMDK file size that will always be smaller than 2TB with the additional size of a snapshot?
Also I'm not aware of any methods to "shrink" a thin provisioned disk, I'm guessing I would need to use a VMconverter and set the destination drive to the smaller size, but then I think the destination/cloned VMDK would then be "thick" provisioned which may then need yet another conversion to put it back to "thin".
Sorry about all the questions but alot of the VM support information is slightly ambiguous.
Than you for your input but I have a couple of questions regarding your response. Regarding Exchange not supporting thin provisioned disks, I haven’t seen this anywhere and I'd assumed that the VMDK type would be transparent to the VM inside anyway?
Also, you say that the snapshot and the VM exceeds max size of the VMDK file, excuse my ignorance but I didn’t think a snapshot increased the size of the VMDK file, I'd assumed that the snapshot would generate a new file not increase the existing one? Does this mean that when creating a VM you must create a disk/VMDK file size that will always be smaller than 2TB with the additional size of a snapshot?
Also I'm not aware of any methods to "shrink" a thin provisioned disk, I'm guessing I would need to use a VMconverter and set the destination drive to the smaller size, but then I think the destination/cloned VMDK would then be "thick" provisioned which may then need yet another conversion to put it back to "thin".
Sorry about all the questions but alot of the VM support information is slightly ambiguous.
It's a Microsoft support statement. We have many clients using thin with Exchange, but if you need to escalate a support issue to Microsoft with Exchange, they may decide not to support your environment. This could have changed with the recent release of 2013.
This is correct, snapshot does not exceed the size of the parent disk, but at a moment in time, size of snapshot + size of parent must be less than 2TB-512 bytes.
Yes, you will need to use VMware vCenter Converter Standalone, this is the only supported, method, VMware vCenter Converter Standalone can create thick or thin options in the Wizard.
see my EE Article, Step by Step Tutorial Instructions with Screenshots
HOW TO: P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5
HOW TO: Synchronize changes when completing a P2V or V2V with VMware vCenter Converter Standalone 5.1
HOW TO: Improve the transfer rate of a Physical to Virtual (P2V), Virtual to Virtual Conversion (V2V) using VMware vCenter Converter Standalone 5.0
Does this mean that when creating a VM you must create a disk/VMDK file size that will always be smaller than 2TB with the additional size of a snapshot?
This is correct, snapshot does not exceed the size of the parent disk, but at a moment in time, size of snapshot + size of parent must be less than 2TB-512 bytes.
Yes, you will need to use VMware vCenter Converter Standalone, this is the only supported, method, VMware vCenter Converter Standalone can create thick or thin options in the Wizard.
see my EE Article, Step by Step Tutorial Instructions with Screenshots
HOW TO: P2V, V2V for FREE - VMware vCenter Converter Standalone 5.5
HOW TO: Synchronize changes when completing a P2V or V2V with VMware vCenter Converter Standalone 5.1
HOW TO: Improve the transfer rate of a Physical to Virtual (P2V), Virtual to Virtual Conversion (V2V) using VMware vCenter Converter Standalone 5.0
ASKER
Hello Hanccocka,
Again thanks for your input, but I would just like to clarify a point, we were under the assumption that the snapshot was failing because the datastore was 4TB in size and it has other smaller VM's occupying 300Gb or so, the Exchange VM is using 2TB thin provisioned and the datastore is reporting over 3TB of free diskspace, but when trying to run a snapshot there wasn’t enough diskspace because it was provisioned to 2TB, so the provisioned size of 2TB and the additional 300 Gb of the other VM's means there's less than the required 2TB available to snapshot the Exchange VM.
I think you’re saying that it’s the size of the VMDK combined with the snapshot that's over 2TB that's the issue? In that case how do you calculate the max size of a VMDK that will be able to perform snapshots?
Also regarding the converters, should you use the latest, or the one that matches the version of ESXI?
Again thanks for your input, but I would just like to clarify a point, we were under the assumption that the snapshot was failing because the datastore was 4TB in size and it has other smaller VM's occupying 300Gb or so, the Exchange VM is using 2TB thin provisioned and the datastore is reporting over 3TB of free diskspace, but when trying to run a snapshot there wasn’t enough diskspace because it was provisioned to 2TB, so the provisioned size of 2TB and the additional 300 Gb of the other VM's means there's less than the required 2TB available to snapshot the Exchange VM.
I think you’re saying that it’s the size of the VMDK combined with the snapshot that's over 2TB that's the issue? In that case how do you calculate the max size of a VMDK that will be able to perform snapshots?
Also regarding the converters, should you use the latest, or the one that matches the version of ESXI?
I think you’re saying that it’s the size of the VMDK combined with the snapshot that's over 2TB that's the issue? In that case how do you calculate the max size of a VMDK that will be able to perform snapshots?
This is correct. You would need to work out the rate of change of data, in your Exchange Server, during the backup Window, or reduce the size of the VMDK. e.g. 1TB.
Also regarding the converters, should you use the latest, or the one that matches the version of ESXI?
I would use the latest version 5.5, as it fixes a lot of bugs. It's compatible with ESXi 5.1.
ASKER
Thanks for your help Hanccocka, it's odd that I haven't seen this referenced anywhere else, it effectivley means that you shousld never allocate a VM diskspace of above or around 2TB as you wont be able to get a snapshot or in many cases a backup as vendors like Veem use this technology to run the backup. I will try the conversion in a lab environment before attempting it on a live system. One of my main concerns is the time it takes to run the conversion.
Regards,
Mark.
Regards,
Mark.
It's quite common, see my EE Article
HOW TO: VMware Snapshots :- Be Patient
also there is a VMware KB here:-
For ESXi 5.0 and 5.1, this error occurs when any individual flat .vmdk file exceeds 2,181,972,430,848 bytes. Try to keep the size of the disk below 2,181,972,430,848 bytes for normal snapshot operations. This is the size generated when adding a virtual disk with size of 1.984492366201720 TB is specified.
source
http://kb.vmware.com/kb/10
This is no longer valid for 5.5, because virtual disks, can be up to 63TB. (of course, it would still not be able to snapshot, if the size was set to 63TB).
HOW TO: VMware Snapshots :- Be Patient
also there is a VMware KB here:-
For ESXi 5.0 and 5.1, this error occurs when any individual flat .vmdk file exceeds 2,181,972,430,848 bytes. Try to keep the size of the disk below 2,181,972,430,848 bytes for normal snapshot operations. This is the size generated when adding a virtual disk with size of 1.984492366201720 TB is specified.
source
http://kb.vmware.com/kb/10
This is no longer valid for 5.5, because virtual disks, can be up to 63TB. (of course, it would still not be able to snapshot, if the size was set to 63TB).
ASKER
Hello Hanncocka,
Tried using the VM converter over the weekend but it wouldn’t run, as soon as I start the process it shows this error; "unable to obtain hardware information for the selected machine". I thought it might be because the VMDK file was hosted on an ISCSI SAN, so I tried using VM converter 5.5 as it has improved support for SAN's but just got the same error. I have tried it in a lab environment where the VM is located on and a SAN and it just works, any ideas? One thing different is that the live site has a VM infrastructure managed with VCenter Server and the VM conversion is being initiated from this Server, I'm wondering if there's a access/credentials issue here?
Tried using the VM converter over the weekend but it wouldn’t run, as soon as I start the process it shows this error; "unable to obtain hardware information for the selected machine". I thought it might be because the VMDK file was hosted on an ISCSI SAN, so I tried using VM converter 5.5 as it has improved support for SAN's but just got the same error. I have tried it in a lab environment where the VM is located on and a SAN and it just works, any ideas? One thing different is that the live site has a VM infrastructure managed with VCenter Server and the VM conversion is being initiated from this Server, I'm wondering if there's a access/credentials issue here?
Have you installed Converter on the VM to be converted?
ASKER
No, I've installed VMconverter on the VCentrer Server as it already has access to the ESXI hosts and VM's within.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hello Hanncocka,
Sorry but I'm a little confused, when running a V2V VM conversion the VM is powered off.
Sorry but I'm a little confused, when running a V2V VM conversion the VM is powered off.
Clearly if you turn if off, you cannot do the conversion if the software is installed on the computer to be converted, but many issues can be overcome, by installing converter on the VM to be converted, and converted Live or Hot.
To overcome permission issues, it's often recommended to install on the computer to be converted.
iSCSI or FC SAN should not make any difference, the VM is not aware of where it's hosted.
To overcome permission issues, it's often recommended to install on the computer to be converted.
iSCSI or FC SAN should not make any difference, the VM is not aware of where it's hosted.
ASKER
Hello Hanncocka,
I'm guessing this would effectively mimic a P2V conversion from within the VM shell/OS? I have added another Windows 2012 VM to the same ESXI host and SAN volume and it looks like the Conversion tool can connect and extract it's hardware details without any issues so it looks like its something specific to this VM, as the other identical VM on the same SAN volume doesn’t appear to have any issues.
I'm guessing this would effectively mimic a P2V conversion from within the VM shell/OS? I have added another Windows 2012 VM to the same ESXI host and SAN volume and it looks like the Conversion tool can connect and extract it's hardware details without any issues so it looks like its something specific to this VM, as the other identical VM on the same SAN volume doesn’t appear to have any issues.
Converter, cannot tell if the computer is physical or virtual!
ASKER
OK, but I'm guessing it will take a longer time to convert a live machine that one that's powered off? Also, as previously mentioned, a test VM created in the same location/volume using the same Hw, ESXI host and OS is communicating with the converter so I think I'll pursue this line to get it working with the Exchange VM. I'm assuming there isnt any issues with drive size and the converter? Kinda ironic if there is!
No issues with drive size.
In our experience, it's quicker to convert Live than powered off!
In our experience, it's quicker to convert Live than powered off!
ASKER
Hello Hanccocka,
Thank you for your assistance with this matter, I went with your suggestion and ran the conversion from within the VM, it completed successfully after about 7.5hrs and the new VM has been running fine on a live system for over two days now at its reduced, thin-provisioned size. Never got to the bottom of why the V2V conversion though, but its all sorted now anyway
Thank you for your assistance with this matter, I went with your suggestion and ran the conversion from within the VM, it completed successfully after about 7.5hrs and the new VM has been running fine on a live system for over two days now at its reduced, thin-provisioned size. Never got to the bottom of why the V2V conversion though, but its all sorted now anyway
Thanks for your feedback.
The issue that you are experiencing is your virtual machine disk plus the snapshot size, exceeds the maximum size of virtual disk (vmdk), which is 2TB-512bytes.
So, a fix, shrink the size of the exchange virtual disk to 500MB-1TB, and then the snapshot plus main virtual disk, will be supported.
Upgrade to ESXi 5.5, 63TB virtual machine disks are supported.
Add a new disk, and move the mailbox store to this new smaller disk.