Solved

How can I create a VMware datastore larger than 2TB with this config?

Posted on 2010-08-17
32
6,605 Views
Last Modified: 2012-05-10
Hi Everyone,

I'm using ESXi 4.1 and my hardware config a single physical box with:
2 x 250GB drives in RAID1 with ESXi installed here
7 x 2TB drive in RAID5 w/Hot Spare so ~10TB useable

I would like to be able to setup 4 virtual servers on this box, each with 2 drives as shown

Virtual Server 1 & 2:
    Drive C:  100GB
    Drive D:  150GB

Virtual Server 3 & 4
   Drive C:  250GB
   Drive D:  4.5TB

Any ideas would be greatly appreciated.



0
Comment
Question by:RBrogen
  • 8
  • 5
  • 4
  • +8
32 Comments
 
LVL 6

Expert Comment

by:JRaster
Comment Utility
Just create your virtual machine, and set the size of the hard disk the size you want the c: drive. Then at the end check the box to edit the VM.
Under the hardware tab, click Add, then hard disk.
Create new virutal disk.  Set your size of the D drive, specify the datastore if needed or differant than where you put the VM.
Hit next, next, finish.  

There are ways to copy that config, but i think its just as easy to create another one just like it following those steps.  

0
 

Author Comment

by:RBrogen
Comment Utility
Thanks JRaster,

The issue I'm running into is the actual physical configuration is 7 x 2TB drives in a RAID 5 w/Hot Spare.  VMware has a 2TB limit on datastore size but you can use extent to add additional 2TB space.  However, to do this I would have to break the Raid5 and configure the drives on the raid hardware as individual drives and thus loosing any benefit of RAID5 and my hot spare.
0
 
LVL 6

Expert Comment

by:JRaster
Comment Utility
Oh i missed your question in the title.  You have to create your volume with larger block sizes.  
My one terabyte array has a 8 MB block size.  You will see that when you create your volumnes.  
0
 

Author Comment

by:RBrogen
Comment Utility
The 8MB block size only gets you to a 2TB datastore.  Optimally what I wanted to have is a RAID 5 that had 10TB of useable and a Hot Spare that could be partitioned via VMWare into my 4 servers configuration.
0
 
LVL 4

Expert Comment

by:ericnils
Comment Utility
I'm not sure how to create a single disk that large, but when I ran into this problem in the past I attached several smaller disks to the virtual server.  I then used LVM in Linux to present the pool of smaller disks to the operating system as a single larger disk.  Its not ideal, but it worked and left me in a position to easily add more storage to the server in the future.
0
 
LVL 6

Expert Comment

by:JRaster
Comment Utility
0
 

Author Comment

by:RBrogen
Comment Utility
I had read that entry before and it is deailing with an external SAN configuration and not a server with internal drive array.
0
 
LVL 6

Expert Comment

by:JRaster
Comment Utility
Sorry i had forgotten about the 2 TB limit.  
I also have a ESXI 4 server with internal storage.  The 2TB limit didnt matter to me.  
What im reading is the only way around is using extents.
http://pubs.vmware.com/vi301/server_config/wwhelp/wwhimpl/common/html/wwhelp.htm?context=server_config&file=sc_storage_manage.11.8.html
0
 

Author Comment

by:RBrogen
Comment Utility
What I have done is take the 7 x 2TB drives and created 6 x 2TB Luns and 1 x 2TB Hot Spare in the RAID configuration.  I then created a new 2TB data store and added 2 more of the 2TB drives using extent and repeated the process giving me 2 datastores each at 5.46TB which I can now break up into the original configuration.  My only question now is what happens if one of the 2TB drives goes bad?  Does the hotspare just kick in and give me no down time?  I'm thinking I'd have some dataloss here without the RAID 5 in place.
0
 
LVL 6

Expert Comment

by:JRaster
Comment Utility
Your raid should be handled by the raid controller.  If one drive goes bad, the hot spare should come online and start the rebuild.  Remember, virutual machines are only files on a  hard disk.  The RAID doesnt care.  
So you should have no dataloss with one drive failure.
0
 
LVL 4

Expert Comment

by:ericnils
Comment Utility
Did you stop using RAID?  Then yes, you will have data loss during a failure.  If you need to break it up you should do this instead:

2x2TB RAID1 x3
1x2TB hot spare

That will lose a lot of disk space, but preserve redundancy while still providing you as much as you said you require.

0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
Only way to get a volume (not datastore) over 2TB is to:

1.  Use MS iSCSI to connect the volume to a SAN/NAS inside the volume OS
0
 

Expert Comment

by:mpstenson
Comment Utility
One other way to go about getting a larger volume is to create two 2tb volumes, convert them to dynamic partitions, then use the microsoft software raid tools to raid them together into one large partition.  A lot of people don't like using software raid but it is about your only option unless you use something like iscsi.  This link has some basic info http://www.petri.co.il/difference_between_basic_and_dynamic_disks_in_windows_xp_2000_2003.htm.
0
 
LVL 28

Expert Comment

by:bgoering
Comment Utility
Will your raid controller allow you to create multiple LUNs from your RAID5 setup - like you can often do with a SAN? If so do that with 2TB LUNs, you can use multiple LUNs for extents to create a large datastore. The limit for any extent is 2TB. The largest vmdk you can create is 2TB - 512KB. You can use OS facilites (LVM for Linux, Disk Manager in Windows) to concatenate the vmdks into a single logical drive for the OS.

Hope that helps
0
 
LVL 4

Expert Comment

by:EshuunDara
Comment Utility
See if you can let the san manage the disks.  On my equallogic SANs, I can create multiple luns that are spread across the entire array instead of specifiying individual disks assigned to each lun.  I've got a raid 50 setup with two hot spares.  When I have a disk failure, one of the hot spares kicks in immediately -- no data loss or downtime.  Once I replace the bad disk, the new disk becomes the hot spare and the array starts to rebuild the parity info onto the spare.

If you can't distribute your LUNs over the entire array, one thing you could do is to split your VM into two drives, one that has the OS, and then set up the other as an iSCSI connection that points to your larger RAID setup.  The primary boot drive would be a vmdk file on your vmware server, and the other drive would be a raw mapping to the allocated space.  
0
 
LVL 2

Expert Comment

by:jlprasadreddy
Comment Utility
i wood suggest the following
Inside your raid manager Create multiple volumes in the following way
VOL1     1 TB
VOL2      4.5 TB
VOL3      4.5 TB
in your ESX Storage Configuration create a DataStore for the VOL1 (1TB Volume) and leave the VOL2 & VOL3 as is.
Create your VM's 1 & 2 with both C & D drives in The above DataStore
Create your VM's 3 & 4 with C Drive in The above DataStore
once you have created edit your VM 3 and add additional disk and choose RAW disk Mappings and you should see the 2 4.5 TB Volumes, if you dont see them go to storage configuration and rescan the disks then you should be able to see them.
Let me know if this statisify's your requirement?
 
JLP
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
@jlprasadreddy: I don't believe the information you have is correct.  An RDM can't be over 2TB:

Per vmware KB

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

Item Maximum
VMFS-General
Raw device mapping (RDM) size 2TB minus 512B

The max you can do for Raw Data Mappings is 2TB. A disk volume cannot go over 2TB period.  This is a SCSI 2 limitation that is used by vSphere.

Unless you use extents your datastore cannot go over 2TB.  With extents the datastore over 2TB is possible but an individual VMDK cannot be over 2TB
0
 
LVL 22

Expert Comment

by:Luciano Patrão
Comment Utility
Hi

And using a 4Tb LUN on a VM you will have performance issues

Only use this is necessary, if not divide your LUN to have a good write/read performance

Jail
0
 

Author Comment

by:RBrogen
Comment Utility
I'll see if the RAID controller will allow for multiple LUNS to point to my array.
0
 

Author Comment

by:RBrogen
Comment Utility
I'll let you know what happens and see which solution/recommendation is most appropriate.  Thanks to all who responded.
0
 
LVL 2

Expert Comment

by:jlprasadreddy
Comment Utility
@paulsolov:  the 2 TB limit for RDM you are talking about is for vmfs FileSystem.  Infact the 2 TB Limitation in vmware is not actually VMWare limitation, its the Linux File system limitation.
In this case RBrogen is usign windows machines and windows OS does not have any volume size limitation hence he will be able to use as i have mentioned.
JLP
0
 

Author Comment

by:RBrogen
Comment Utility
Thanks jlprasadreddy...I will try this configuration.  My only other concern is loosing the RAID5 redundancy.
0
 
LVL 28

Expert Comment

by:bgoering
Comment Utility
You may be able to "get away with it" but should look at the following kb article...

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=3371739

"Raw Device Mapping Limit
In the case of physical mode RDMs, the SCSI traffic sent by the guest operating system to do reads or writes are passed through by the VMware ESX storage layer to the underlying storage. This allows some guests to break the 2 terabyte limit but this is an unsupported configuration because the storage stack of the ESX is not designed to handle such capacity.


Note: When migrating and powering on virtual machines, ESX 4.0 enforces the 2 TB - 512 bytes limit where as ESX 3.5 does not."
0
 
LVL 2

Expert Comment

by:jlprasadreddy
Comment Utility
@RBrogen: i dont think you will be loosing RAID5 redundancy because you will be creating the volumes after you have created the raid group, or to be specific you will be creating the partitions after creating the raid volume.
JLP
0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
@jlprasadreddy

I disagree the 2TB is a SCSI 2 limitation and has nothing to do with Linux or Windows.  If you take a look at the mix/max KB you will see this.


You state "map raw disk"  In vmware a raw disk is a RDM.  Unless I missed something in class or the vmware articles, or the many implementations I have done  I don't think you are correct. If it's a raw disk map than the 2TB-512MB still applies.


0
 
LVL 22

Expert Comment

by:Luciano Patrão
Comment Utility
Hi

@paulsolov

Yes you are correct regarding the SCSI limitation, and RDM

Jail
0
 
LVL 3

Expert Comment

by:jodix2002
Comment Utility
Hi..
I have read that a lot of people are using extent only when they absolutely have no other choice. People also suggest that extents should not be used for a long period of time.
I have some links to discussions about extents below. Sorry, these will not answer your question, but perhaps you may find it useful:

http://communities.vmware.com/thread/81494
http://communities.vmware.com/thread/65156
http://communities.vmware.com/message/61679
http://communities.vmware.com/message/323671
http://communities.vmware.com/message/604008
0
 
LVL 42

Expert Comment

by:paulsolov
Comment Utility
I just use NFS when it gets to that point.  I have seend datastores over 3TB since they are basically share volumes
0
 

Accepted Solution

by:
RBrogen earned 0 total points
Comment Utility
After much research, the configuration of local hard dirves will not work unless I break the 7 2TB drives into individual LUNs and then in VM create the initial data store of 2TB and add 2 extents to reach the size I want.  This doesn't work for me as I don't want to loose the RAID5/HS config so I am going with an SAN and just creating the luns I need and building on the extents to get the RAID5/HS.
0
 
LVL 4

Expert Comment

by:EshuunDara
Comment Utility
I don't understand where you're saying that you have to lose the RAID5 config because you're creating multiple LUNs.  Stop creating your LUNs and assigning individual drives to them.  It's completely possible to have a giant RAID5/10/50 array and then partition the space up into separate LUNs and then use extents to create a large datastore.  You should try an implement that.  I've got several systems here at work that do just that; as an example, I've got a 14 drive array with two drives configured as a hot spare in a RAID50 configuration (all 12 remaining drives) with a series of 500G LUNs.  Each LUN is spanned across 12 drives.  When I lose a drive, the hot spare kicks in and the array rebuilds immediately.

What are you using for your SAN?
0
 
LVL 28

Expert Comment

by:bgoering
Comment Utility
@EshuunDara - I believe he is using local storage and many internal raid controllers do not support partitioning of a raid group into multiple LUNs while almost all flavors of SAN do
0
 
LVL 1

Expert Comment

by:jjoz
Comment Utility
is there any performance penalty in using multiple extends to create > 2 TB datastore ?
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
Teach the user how to delpoy the vCenter Server Appliance and how to configure its network settings Deploy OVF: Open VM console and configure networking:
Teach the user how to install ESXi 5.5 and configure the management network System Requirements: ESXi Installation:  Management Network Configuration: Management Network Testing:

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now